I've been troubleshooting this error on and off for weeks and haven't seemed to be able to get to the bottom of it. We have a relatively small ESX 3.5 infrastructure with two hosts and roughly 12 guest VMs connected to an iSCSI SAN. We're using Backup Exec along with the agent for VMWare to back up all of the virtual servers. Backups using the VMWare agent are successful 99% of the time for all hosts except one. This one host will back up successfully 5-7 times in a row, then will fail 10-15 times always with the same error displaying in Backup Exec:
V-79-57344-38220 - Backup of the virtual machine 'CFHCUtil' failed. VMware VCB framework reported the following error
Error: Other error encountered: Snapshot creation failed: Could not quiesce file system.
I believe I'm running the latest version of VMWare Tools which is 3.5.0 build 184236. I have the following VMWare device drivers enabled/installed within the client:
SCSI Driver
SVGA Driver
Mouse Driver
VMXNet NIC driver
Memory Control Driver
Filesystem Sync Driver
Volume Shadow Copy Services Support
I've reviewed the logs on the ESX host and have cut/pasted the following section which details the events leading up to the failure. It looks the the snapshot completes successfully, but there appears to be an issue when reporting back to Backup Exec. If anyone has had any luck resolving this once and for all, I'd love to hear the solution.
Task Created : haTask-16-vim.VirtualMachine.createSnapshot-38159
2010-04-09 05:56:25.426 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 info State Transition (VM_STATE_ON -> VM_STATE_CREATE_SNAPSHOT)
2010-04-09 05:56:25.427 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 info Trying quiesced snapshot via backup agent...
2010-04-09 05:58:00.185 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 info Starting to snapshot VM.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-000001-delta.vmdk" : open successful (21) size = 26843545600, hd = 0. Type 8
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-000001-delta.vmdk" : open successful (21) size = 5368709120, hd = 0. Type 8
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-000001-delta.vmdk" : open successful (21) size = 8589934592, hd = 0. Type 8
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-flat.vmdk" : open successful (21) size = 26843545600, hd = 0. Type 3
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-flat.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-flat.vmdk" : open successful (21) size = 5368709120, hd = 0. Type 3
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-flat.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-flat.vmdk" : open successful (21) size = 8589934592, hd = 0. Type 3
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-flat.vmdk" : closed.
2010-04-09 05:58:02.315 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 info Snapshot guest reported no errors.
2010-04-09 05:58:02.315 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 info Guest snapshot completed for quiesced snapshot via backup agent.
Task Created : haTask-ha-root-pool-vim.ResourcePool.updateConfig-38166
Task Completed : haTask-ha-root-pool-vim.ResourcePool.updateConfig-38166
Task Created : haTask-ha-root-pool-vim.ResourcePool.updateConfig-38167
Task Completed : haTask-ha-root-pool-vim.ResourcePool.updateConfig-38167
2010-04-09 05:58:10.392 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 info Notifying completion of quiesced snapshot via backup agent.
2010-04-09 05:58:10.647 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 error Quiesced snapshot backup agent error: Snapshot operation aborted (3)
2010-04-09 05:58:12.698 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 error Quiesced snapshot backup agent error: Asynchronous operation failed: Start
(5)
2010-04-09 05:58:14.699 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 error An error occurred while creating quiesced snapshot via backup agent.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-000001-delta.vmdk" : open successful (21) size = 26843545600, hd = 0. Type 8
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-000001-delta.vmdk" : open successful (21) size = 5368709120, hd = 0. Type 8
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-000001-delta.vmdk" : open successful (21) size = 8589934592, hd = 0. Type 8
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-flat.vmdk" : open successful (21) size = 26843545600, hd = 0. Type 3
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-flat.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-flat.vmdk" : open successful (21) size = 5368709120, hd = 0. Type 3
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-flat.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-flat.vmdk" : open successful (21) size = 8589934592, hd = 0. Type 3
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-flat.vmdk" : closed.
2010-04-09 05:58:15.329 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 info Removing timedout snapshot...
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-000001-delta.vmdk" : open successful (21) size = 26843545600, hd = 0. Type 8
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-000001-delta.vmdk" : open successful (21) size = 5368709120, hd = 0. Type 8
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-000001-delta.vmdk" : open successful (21) size = 8589934592, hd = 0. Type 8
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-000001-delta.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-flat.vmdk" : open successful (21) size = 26843545600, hd = 0. Type 3
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName-flat.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-flat.vmdk" : open successful (21) size = 5368709120, hd = 0. Type 3
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_1-flat.vmdk" : closed.
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-flat.vmdk" : open successful (21) size = 8589934592, hd = 0. Type 3
DISKLIB-VMFS : "/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName_2-flat.vmdk" : closed.
2010-04-09 05:58:51.952 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 warning Failed operation
2010-04-09 05:58:51.953 'vm:/vmfs/volumes/4b217231-245bf669-695f-0022640c9c72/ServerName/ServerName.vmx' 22137776 info State Transition (VM_STATE_CREATE_SNAPSHOT -> VM_STATE_ON)
Task Completed : haTask-16-vim.VirtualMachine.createSnapshot-38159