[TYPO3-typo3org] Post mortem: Gerrit outage Jan 12th
Steffen Gebert
steffen.gebert at typo3.org
Mon Jan 12 21:46:19 CET 2015
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hey,
tl;dr this is a first public post mortem report. If you are not
administrating any MySQL servers, this might not be of interest for you
Today from ca. 18:20-21:00h CEST Gerrit showed error 500s when
accessing changes because the MySQL service stopped crashed (repeatedly).
Prior to that, the server received a hard reset two times today due to
data center power outages.
Reason for the problem now was that the "changes" mysql table was
crashed and InnoDB halts the server once it notices checksum mismatches.
How to fix:
http://www.percona.com/blog/2008/07/04/recovering-innodb-table-corruption/
- - start server with innodb_force_recovery=1 so that the server is
read-only but will *not* halt on read error
- - dump table
- - restart without recovery flag
- - drop original table
- - import dump
What also helped me to verify what table is crashed:
# for i in $(ls /var/lib/mysql/*/*.ibd); do innochecksum -v $i; done
via
http://dba.stackexchange.com/questions/6191/how-do-you-identify-innodb-table-corruption
Best regards
Steffen
- --
Steffen Gebert
TYPO3 Server Administration Team Member
TYPO3 .... inspiring people to share!
Get involved: http://typo3.org/teams/server-team/
My wish list:
https://www.amazon.de/registry/wishlist/922E3JYSQ7CV/ref=cm_wl_sb_v?sort=priority
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJUtDKXAAoJEIskG/rSlyw4qloH/09hpsV/CChcZSiCG9J/OLp1
0xxkEJKBjpv432nDZcwLFXRRkWPfoMQLciA0SjULC4jM1mycJUbm4K5aoex2qQLk
p9BBuWcL7uGYrYC23xi9ZN1z3lM8PKtp2kGVmKscpKcSQqyODaXs9wUKLHiOdUCu
tqSX0aaUqXUxx1+szkiDb/pO9ooaySBcExVL79rrxkAMMrtzpyhEkMp+fi/WbVsp
7smyUHlp9k8PhHTHrT6on6Ls1WrlKCDburr0NbdjUBAHzQ/wYnu+JyoWPuXRDRNE
vhKNmIbcPLYyouieR2skuYxN9qsq8Gc7aRs0Eb9HKlxlUQeXgDgUasAPQ7MhZ2U=
=wv9e
-----END PGP SIGNATURE-----
More information about the TYPO3-team-typo3org
mailing list