vMotion failed (The VM failed to resume on the destination during early power on.)
I had to do some maintenance on a host but one of the Virtual Machines failed to move away, and came with a “General System Error”:
In the log file (vmware.log) of the Virtual Machine was the following lines:
2014-09-08T10:42:01.220Z| Worker#0| I120: DISKLIB-CTK : Could not open tracking file. File open returned IO error 4.
2014-09-08T10:42:01.225Z| Worker#0| I120: DISKLIB-CTK : Could not open change tracking file "/vmfs/volumes/datastore-UUID/<servername>/<servername>-ctk.vmdk": Could not open/create change tracking file.
2014-09-08T10:42:01.229Z| Worker#1| I120: DISKLIB-CTK : Could not open tracking file. File open returned IO error 4.
2014-09-08T10:42:01.229Z| Worker#1| I120: DISKLIB-CTK : Could not open change tracking file "/vmfs/volumes/datastore-UUID/<servername>/<servername>_1-ctk.vmdk": Could not open/create change tracking file.
2014-09-08T10:42:01.238Z| Worker#0| I120: DISKLIB-LIB : Could not open change tracker /vmfs/volumes/datastore-UUID/<servername>/<servername>-ctk.vmdk: Could not open/create change tracking file.
2014-09-08T10:42:01.241Z| Worker#0| I120: DISKLIB-VMFS : "/vmfs/volumes/datastore-UUID/<servername>/<servername>-flat.vmdk" : closed.
2014-09-08T10:42:01.241Z| Worker#0| I120: DISKLIB-LIB : Failed to open '/vmfs/volumes/datastore-UUID/<servername>/<servername>.vmdk' with flags 0xa Could not open/create change tracking file (2108).
2014-09-08T10:42:01.241Z| Worker#0| I120: DISK: Cannot open disk "/vmfs/volumes/datastore-UUID/<servername>/<servername>.vmdk": Could not open/create change tracking file (2108).
2014-09-08T10:42:01.241Z| Worker#1| I120: DISKLIB-LIB : Could not open change tracker /vmfs/volumes/datastore-UUID/<servername>/<servername>_1-ctk.vmdk: Could not open/create change tracking file.
2014-09-08T10:42:01.242Z| Worker#1| I120: DISKLIB-VMFS : "/vmfs/volumes/datastore-UUID/<servername>/<servername>_1-flat.vmdk" : closed.
2014-09-08T10:42:01.242Z| Worker#1| I120: DISKLIB-LIB : Failed to open '/vmfs/volumes/datastore-UUID/<servername>/<servername>_1.vmdk' with flags 0xa Could not open/create change tracking file (2108).
2014-09-08T10:42:01.242Z| Worker#1| I120: DISK: Cannot open disk "/vmfs/volumes/datastore-UUID/<servername>/<servername>_1.vmdk": Could not open/create change tracking file (2108).
2014-09-08T10:42:01.242Z| Worker#1| I120: Msg_Post: Error
2014-09-08T10:42:01.242Z| Worker#1| I120: [msg.literal] Could not open/create change tracking file
With indicates that there is something wrong with the Change Block Tracking (CBT) files. I know that CBT should not be enabled on this machine but some time ago somebody used this machine for some image backup testing, but forgot to disable CBT on it afterwards.
After some searching i found this VMware KB on the problem: KB2001004 and there was also some blog articles describing the same solution, but I could not power off the Virtual Machine, and I know that some image backup software is able to do this thru the API, so I thought that this was possible thru PowerCLI, and it turnout it is. Here is how:
connect-viserver <vcenter FQDN> $vm = get-vm -name <VM name> $vmview = $vm | get-view $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec $vmConfigSpec.changeTrackingEnabled = $false $vmview.reconfigVM($vmConfigSpec) $snap=$vm | New-Snapshot -Name 'Disable CBT' $snap | Remove-Snapshot -confirm:$false
This will disable CBT on a running machine.
very good