[TYPO3-typo3org] Post mortem: Gerrit outage Jan 12th

Steffen Gebert steffen.gebert at typo3.org
Mon Jan 12 21:46:19 CET 2015


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey,

tl;dr this is a first public post mortem report. If you are not
administrating any MySQL servers, this might not be of interest for you

Today from ca. 18:20-21:00h CEST Gerrit showed error 500s when
accessing changes because the MySQL service stopped crashed (repeatedly).
Prior to that, the server received a hard reset two times today due to
data center power outages.

Reason for the problem now was that the "changes" mysql table was
crashed and InnoDB halts the server once it notices checksum mismatches.

How to fix:
http://www.percona.com/blog/2008/07/04/recovering-innodb-table-corruption/

- - start server with innodb_force_recovery=1 so that the server is
read-only but will *not* halt on read error
- - dump table
- - restart without recovery flag
- - drop original table
- - import dump

What also helped me to verify what table is crashed:
# for i in $(ls /var/lib/mysql/*/*.ibd); do innochecksum -v $i; done
via
http://dba.stackexchange.com/questions/6191/how-do-you-identify-innodb-table-corruption

Best regards
Steffen
- -- 
Steffen Gebert
TYPO3 Server Administration Team Member

TYPO3 .... inspiring people to share!
Get involved: http://typo3.org/teams/server-team/

My wish list:
https://www.amazon.de/registry/wishlist/922E3JYSQ7CV/ref=cm_wl_sb_v?sort=priority
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org

iQEcBAEBAgAGBQJUtDKXAAoJEIskG/rSlyw4qloH/09hpsV/CChcZSiCG9J/OLp1
0xxkEJKBjpv432nDZcwLFXRRkWPfoMQLciA0SjULC4jM1mycJUbm4K5aoex2qQLk
p9BBuWcL7uGYrYC23xi9ZN1z3lM8PKtp2kGVmKscpKcSQqyODaXs9wUKLHiOdUCu
tqSX0aaUqXUxx1+szkiDb/pO9ooaySBcExVL79rrxkAMMrtzpyhEkMp+fi/WbVsp
7smyUHlp9k8PhHTHrT6on6Ls1WrlKCDburr0NbdjUBAHzQ/wYnu+JyoWPuXRDRNE
vhKNmIbcPLYyouieR2skuYxN9qsq8Gc7aRs0Eb9HKlxlUQeXgDgUasAPQ7MhZ2U=
=wv9e
-----END PGP SIGNATURE-----


More information about the TYPO3-team-typo3org mailing list