You are confronted with various database errors in the /var/log/vmware/vpx/vpxd.log and you also have various errors in /storage/db/vpostgres/pg_log
Similar in theme to:
error ‘Default’ opID=HB-host-8637@82541-3778684f] [VdbStatement] SQLError was thrown: “ODBC error: (00000) – ” is returned when executing SQL statement
error ‘Default’ opID=HB-host-8637@82541-3778684f] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) “ODBC error: (00000) – ” is returned when executing SQL statement
PANIC: checksum mismatch: disk has 0x4addc132, should be 0x7cb3b932
filename base/16385/24461, BlockNum 37, block specifier 1663/16385/24461/0/37
Cause:
The root cause is you likely lost power, but the larger problem is obviously that the VMware vCSA is built using one of the most un-user friendly databases in this galaxy (Postgres). With that said, let’s get this hunk of garbage beautiful piece of modern infrastructure management back online.
Solutions:
Repair vPostgres Indexes:
If you have Index errors in the log file (anything mentioning issues with *idx), you will need to regenerate the particular index in the Postgres database. To accomplish this:
- Get your postgres password: cat /etc/vmware-vpx/embedded_db.cfg | grep PASSWORD
- You will need to add a shell to the postgres user: type: ‘vipw’, locate the postgres user, change the shell from /bin/false to /bin/bash, save the file (:wq!) It will look this when you are done: postgres:x:1000:100::/var/vmware/vpostgres/9.0:/bin/bash
- su postgres
- /opt/vmware/vpostgres/9.0/bin/psql VCDB
- REINDEX DATABASE;
- Optional and recommended: VACUUM FULL; (this will recover disk space and will recreate all of your tables)
Repair vPostgres Data:
If you have errors in your logs that complain about invalid checksums, you will need to force Postgres to fix these. There is no MySQL type command to automatically repair all databases (glorious!). You will need repair each issue manually. There will likely only be one that is preventing vCenter from fully starting.
Your error message will look like this:
PANIC: checksum mismatch: disk has 0x4addc132, should be 0x7cb3b932
filename base/16385/24461, BlockNum 37, block specifier 1663/16385/24461/0/37
filename base/16385/54431780, BlockNum 102, block specifier 1663/16385/54431780/0/102
To fix these errors (there might be more than one):
- service vmware-vpxd stop ; service vmware-vpostgres stop
- Get your postgres password: cat /etc/vmware-vpx/embedded_db.cfg | grep PASSWORD
- You will need to add a shell to the postgres user: type: ‘vipw’, locate the postgres user, change the shell from /bin/false to /bin/bash, save the file (:wq!)
- su postgres
- /opt/vmware/vpostgres/9.0/bin/postgres -D /storage/db/vpostgres -c fix_block_checksum=”1663/16385/54431780/0/102″ [Note: this last section is just a copy and paste of the “block specifier from the log file”.]
