vSphere Replication Appliance won’t reconnect to vCenter

~ admin

For a multitude of reasons your VR appliance is not connected to vCenter, such as:

  1. You updated the SSL certificate
  2. You disconnected the ESXi host that had the vSphere Replication appliance
  3. You rolled back vCenter from a backup or snapshot

Login to the VMware vSphere Replication appliance.

Identify the password of your keystore. To do this, type this command:

/opt/vmware/hms/bin/hms-configtool -cmd list | grep keystore

Note the keystore password.

Now execute this command:

java -jar va-util.jar -cmd certauth -host vCenter_Server_IP-address -port 80 -user vCenter_Server_USERNAME -pass vCenter_Server_PASSWORD -extkey com.vmware.vcHms -keystore /opt/vmware/hms/security/hms-keystore.jks -keystorealias jetty -keystorepass KEYSTOREPASSWORD

FSCK LVM on CentOS Won’t Boot

~ admin

Your VM won’t boot. FSCK needed on an LVM. You don’t have root and are screwed.

Boot the rescue disk.

Scan all disks for partiitions: 

Scan all disks for volume groups and build /etc/lvmtab and /etc/lvmtab.d/* which are the database for all other lvm commands:

Change attributes of a logical volume

Scan all disks for logical volumes

Then I was able to run fsck as follows

fsck -f /dev/VolGroup00/LogVol00

zfs: cannot import : I/O error Destroy and re-create the pool from a backup source

~ admin

Your FreeNAS, or FreeBSD, ZFS zpool lost power and can’t be accessed.

Boot the machine and enter the boot loader menu and set these:

set vfs.zfs.debug=1
set vfs.zfs.recover=1
set debug.bootverbose=1

boot -s <enter> to start the machine in single user mode.

Once booted:

zpool import -fFX -o readonly=on -R /mnt tank (or whatever the name of your zpool is).

This should work in most all cases and allow you to rsync/scp/copy your data off of the zpool.

Don’t deviate from the above. I know you are super smart and believe you don’t need to set the flags, don’t need single user mode, or don’t need to mount in read only. You do. Don’t waste time and risk screwing up the pool permanently.

Update vCenter VAMI 5480 SSL Certificate

~ admin

Note: The below is for vCenter 6.0. VMware has made it easier in 6.7. You just need to restart VAMI using:

/sbin/service vami-lighttp restart

You have updated the SSL certificate on your Platform Service Controller (PSC) and you probably went through a further nightmare of update the SSL certificates on a bunch of other services.

After all of the above was completed, you realized that none of this tediosity updated the VAMI SSL certificate on port 5480.

Login to the vCenter shell and run this command:

/usr/lib/applmgmt/support/scripts/postinstallscripts/lighttpd-vecs-integration.sh

It will update the SSL certificate used on the VAMI to the SSL certificate used on the PSC, vCenter webclient, etc:

Process Sendmail clientmqueue

~ admin

You have emails stuck in clientmqueue that are stuck because of a prior sendmail issue.

To process all e-mails in the clientmqueue:

 

sendmail -Ac -q -v

Manually flush clientmqueue:

sendmail -Ac -q -v

00:0c:29:4c:6f:70 is not an allowed static Ethernet address. It conflicts with VMware reserved MACs.

~ admin

You are attempting to manually assign a MAC to a vSphere VM in the 00:0c:29:xxx range.

You receive this error:

Module DevicePowerOn power on failed.
Could not set up “macAddress” for ethernet0.
Invalid MAC address specified.
00:0c:29:4c:6f:70 is not an allowed static Ethernet address. It conflicts with VMware reserved MACs.

Solution:  Edit the .vmx file for the VM and add this line:

ethernet0.checkMACAddress = "false"

vCSA Postgres Database Repair – PANIC: checksum mismatch

~ admin

You are confronted with various database errors in the /var/log/vmware/vpx/vpxd.log and you also have various errors in /storage/db/vpostgres/pg_log

Similar in theme to:

error ‘Default’ opID=HB-host-8637@82541-3778684f] [VdbStatement] SQLError was thrown: “ODBC error: (00000) – ” is returned when executing SQL statement

error ‘Default’ opID=HB-host-8637@82541-3778684f] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) “ODBC error: (00000) – ” is returned when executing SQL statement

PANIC: checksum mismatch: disk has 0x4addc132, should be 0x7cb3b932
filename base/16385/24461, BlockNum 37, block specifier 1663/16385/24461/0/37

Cause:

The root cause is you likely lost power, but the larger problem is obviously that the VMware vCSA is built using one of the most un-user friendly databases in this galaxy (Postgres).  With that said, let’s get this hunk of garbage beautiful piece of modern infrastructure management back online.

Solutions:

Repair vPostgres Indexes:

If you have Index errors in the log file (anything mentioning issues with *idx), you will need to regenerate the particular index in the Postgres database.    To accomplish this:

  1. Get your postgres password:  cat /etc/vmware-vpx/embedded_db.cfg | grep PASSWORD
  2. You will need to add a shell to the postgres user:  type:  ‘vipw’, locate the postgres user, change the shell from /bin/false to /bin/bash, save the file (:wq!)       It will look this when you are done:  postgres:x:1000:100::/var/vmware/vpostgres/9.0:/bin/bash
  3. su postgres
  4. /opt/vmware/vpostgres/9.0/bin/psql VCDB
  5. REINDEX DATABASE;
  6. Optional and recommended:  VACUUM FULL; (this will recover disk space and will recreate all of your tables)

Repair vPostgres Data:

If you have errors in your logs that complain about invalid checksums, you will need to force Postgres to fix these.   There is no MySQL type command to automatically repair all databases (glorious!).  You will need repair each issue manually.   There will likely only be one that is preventing vCenter from fully starting.

Your error message will look like this:

PANIC: checksum mismatch: disk has 0x4addc132, should be 0x7cb3b932
filename base/16385/24461, BlockNum 37, block specifier 1663/16385/24461/0/37

filename base/16385/54431780, BlockNum 102, block specifier 1663/16385/54431780/0/102

To fix these errors (there might be more than one):

  1. service vmware-vpxd stop ; service vmware-vpostgres stop
  2. Get your postgres password:  cat /etc/vmware-vpx/embedded_db.cfg | grep PASSWORD
  3. You will need to add a shell to the postgres user:  type:  ‘vipw’, locate the postgres user, change the shell from /bin/false to /bin/bash, save the file (:wq!)
  4. su postgres
  5. /opt/vmware/vpostgres/9.0/bin/postgres -D /storage/db/vpostgres -c fix_block_checksum=”1663/16385/54431780/0/102″   [Note: this last section is just a copy and paste of the “block specifier from the log file”.]

 

Acronis True Image TIB to VMware VM

~ admin

How to convert an Acronis True Image TIB backup file to a VMware VM.

  1. Create a VM with the same OS as the backup with a larger disk than the backup.  How much bigger does not matter.
  2. Boot the VM with the Acronis True Image recovery ISO.  If you do not have this, it is available for download on various websites and found via a Google search.
  3. Once the ISO is booted, choose recovery and connect to the remote location with the TIB file.  I find a local FTP connection to work fine.
  4. After the restore is done, reboot the VM and see if it is boots.  It likely won’t boot.  Don’t waste any time trying to troubleshoot.
  5. Load up VMware converter and convert the VM.   VMware converter will fix all drive label, boot, etc issues.  You will have a fully functioning VM once it is done.
  6. Host your Acronis True Image TIB at VMDK Hosting.   They will do the entire conversion listed above for you and host the exported VMware VM.

Japan Rail Pass – How to Buy – Should I buy

~ admin

You are planning a trip to Japan and wondering about the Japan Rail Pass.

Should I buy the Japan Rail Pass?   The quick and easy answer is likely:  NO

The rail pass is expensive and restrictive.   It is HIGHLY doubtful you would use its full value unless you plan on going to a bunch of cities in a short span of time.

If you are planning on going to Tokyo for a few days and then onto Osaka, Kyoto, or some other place…  YOU SHOULD NOT BUY THE RAIL PASS.  You will lose money and a lot of money.

Let’s do some math…  The Japan Rail Pass is good for 7 days.

You arrive in Tokyo on day #1 and decide to spend 3 nights in Tokyo and then 3 nights in Kyoto and then fly out of Tokyo on day #7.

Narita to Tokyo = 2200 Yen

Tokyo to Kyoto = 8500 Yen (each way) = 17000 Yen

If you are lucky, your pass has not expired and you get your final trip to Narita on the Japan Rail Pass = 2200 Yen   (This rarely happens)

There are instances where you can use the JR Rail Pass in the cities (instead of the metro) and this would save you 200 Yen on those trips…   Lets assume you used it 10 times = 2000 Yen.

Total:  17000 + 2200 + 2200 (questionable – last trip to Narita) + 2000 (questionable – usage instead of metro) = 23400

The Japan Rail Pass costs: 27000 Yen…  you just wasted 3600 Yen (~$36) in the best case and 8000 Yes (~$80) is the most common example.

Don’t buy the rail pass unless you absolutely are going to use the train A LOT in 7 (SEVEN) days…  otherwise you will realize you should never have bought it.   I wish I did not buy it and would never buy one again…

A lot of people on the internet are pushing them because they want you to click their affiliate link (so they get paid)…   Do the math for yourself.

vMotion Failing: Invalid Argument IPv6

~ admin

You are attempting to vMotion a VM, likely using the vSphere web client and likely trying to change host and storage.

/var/log/vmkernel.log on the source ESXi host is showing errors.    Specifically this error:

2016-09-24T14:39:57.638Z cpu5:34608)FSS: 5914: Failed to open file ‘vmware.log’; Requested flags 0x2, world: 34608 [vpxa-worker] (Existing flags 0x5, world: 76640135 [vmx]): Busy

/var/log/host.log one source ESXi host is showing that the source machine is an IPv6 IP and the destination IP is an IPv4 IP.  Like this:

 

40B70 info ‘Vmsvc.vm:/vmfs/volumes/562578d3-39000d7a-8011-blah/Analyzer/Analyzer.vmx’ opID=78d0fdec-95-54 user=vpxuser] VMotionPrepare: srcMgmtIp=2607:0000:6:8:225:9999:fe00
40B70 info ‘Vmsvc.vm:/vmfs/volumes/562578d3-39000d7a-8011-blahe/Analyzer/Analyzer.vmx’ opID=78d0fdec-95-54 user=vpxuser] VMotionPrepare: dstMgmtIp=192.212.10.10

Fix your IP issues.   Vcenter has added one of your host with its IPv4 address and the other host with its IPv6 address.   Temp fix is to disable IPv6…  Long term…  fix your IPv6 issue (remove re-add host).