Hey All,

Experiencing an issue where my jobs for VMs running on local disk are failing due to what appears to be a timeout on the take snapshot action. So at first I thought it was a timing issue, so I looked at that. It takes about 1 minute for quiesed snapshots to actually happen.

I found the other articles that mention you should change the --subprocesstimeout=600 parameter to the /usr/local/avamarclient/var/avvcbimageAll.cmd file, so I did that with no luck. I also tried increasing the snapshot removal time, with no luck.

Still, after applying this and rebooting the appliance no dice, even though the snapshot is completing before the VDP task, it's still failing.

VMware KB: vSphere Data Protection backup jobs fail intermittently

I had another VM work just fine that was on a local SSD, but any disks that are otherwise located on slow local magnetic disk fail.

Looking at the logs, it would appear the most interesting entry is bolded and underlined below, which leads me to the following article, which says, your storage sucks, figure out why your snapshots are taking so long. Anyone have an idea how to increase the timeout when snapshots are being "TAKEN" as opposed to being removed? Next up will try removing queising from the config.

VMware KB: Backing up vCenter Server with vSphere Data Protection (VDP) fails with error: Soap fault due to Connecti…

2015-01-23T11:42:45.635+04:00 avvcbimage Info <19704>: DataStore Storage Info:ESX4_Local_ESXi_Installable capacity=1995012308992  free=1682109890560

2015-01-23T11:42:45.635+04:00 avvcbimage Info <19716>: DS Capacity=1995012308992  FreeSpace=1682109890560  / HD committed=57196542066    unCommitted=29917971642    unShared=57196542066

2015-01-23T11:42:46.584+04:00 avvcbimage Info <16001>: Found 2 disk(s), 0 snapshots, and 0 snapshot files, on the VMs datastore.

2015-01-23T11:42:46.584+04:00 avvcbimage Info <0000>: isExitOK()=0

2015-01-23T11:42:46.599+04:00 avvcbimage Info <19680>: vmAction runBackupScript: ()

2015-01-23T11:42:46.599+04:00 avvcbimage Info <19681>: vmAction runBackupScript: script is skipped because it is null

2015-01-23T11:42:46.600+04:00 avvcbimage Info <0000>: [IMG0009] Pre-snapshot script:  completed successfully

2015-01-23T11:42:46.600+04:00 avvcbimage Info <9692>: a VM snapshot has been requested

2015-01-23T11:42:46.600+04:00 avvcbimage Info <14627>: Creating snapshot 'VDP-1422031366f2a36877d42af6c1a25a994c67b393fa02286299', quieceFS=1

2015-01-23T11:42:46.635+04:00 avvcbimage Info <14631>: Snapshot 'VDP-1422031366f2a36877d42af6c1a25a994c67b393fa02286299' creation for VM '[ESX4_Local_ESXi_Installable] <VMNAME>/<VMNAME>.vmx' task still in progress, sleep for 2 sec

2015-01-23T11:42:48.662+04:00 avvcbimage Info <14631>: Snapshot 'VDP-1422031366f2a36877d42af6c1a25a994c67b393fa02286299' creation for VM '[ESX4_Local_ESXi_Installable] <VMNAME>/<VMNAME>.vmx' task still in progress, sleep for 2 sec

2015-01-23T11:42:50.689+04:00 avvcbimage Info <14631>: Snapshot 'VDP-1422031366f2a36877d42af6c1a25a994c67b393fa02286299' creation for VM '[ESX4_Local_ESXi_Installable] <VMNAME>/<VMNAME>.vmx' task still in progress, sleep for 2 sec

2015-01-23T11:43:12.796+04:00 avvcbimage Warning <16004>: Soap fault detected, Query problem, Msg:'SOAP 1.1 fault: SOAP-ENV:Client [no subcode]

"Name or service not known"

Detail: getaddrinfo failed in tcp_connect()

'

2015-01-23T11:43:12.796+04:00 avvcbimage Error <17773>: Snapshot 'VDP-1422031366f2a36877d42af6c1a25a994c67b393fa02286299' creation for VM '[ESX4_Local_ESXi_Installable] <VMNAME>/<VMNAME>.vmx' task failed to start

2015-01-23T11:43:12.796+04:00 avvcbimage Info <19680>: vmAction runBackupScript: ()

2015-01-23T11:43:12.796+04:00 avvcbimage Info <19681>: vmAction runBackupScript: script is skipped because it is null

2015-01-23T11:43:12.796+04:00 avvcbimage Info <0000>: [IMG0009] Post-snapshot script:  completed successfully

2015-01-23T11:43:12.796+04:00 avvcbimage FATAL <0000>: [IMG0003] The VMX '[ESX4_Local_ESXi_Installable] <VMNAME>/<VMNAME>.vmx' could not be snapshot.

2015-01-23T11:43:12.796+04:00 avvcbimage Info <9772>: Starting graceful (staged) termination, Create Snapshot failure. (wrap-up stage)

2015-01-23T11:43:12.796+04:00 avvcbimage Error <0000>: [IMG0009] createSnapshot: snapshot creation  or pre/post snapshot script failed

2015-01-23T11:43:12.796+04:00 avvcbimage Error <0000>: [IMG0009] createSnapshot: snapshot creation/pre-script/post-script failed

2015-01-23T11:43:12.796+04:00 avvcbimage Info <0000>: isExitOK()=202

2015-01-23T11:43:12.796+04:00 avvcbimage Info <40370>: snapshot created:false NOMC:false ChangeBlTrackingAvail:true UsingChBl:true, ExitOK:false, cancelled:false, fatal: true

2015-01-23T11:43:12.796+04:00 avvcbimage Info <0000>: vcbimage_progress::terminate

2015-01-23T11:43:12.796+04:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_EndAccess: Disk access completed.

Basically the article says, your storage sucks, deal with it and get better storage.

2015-01-23T11:43:12.796+04:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_Connect: Establish connection.

2015-01-23T11:43:12.796+04:00 avvcbimage Info <16041>: VDDK:VixDiskLibVim: VixDiskLibVim_AllowVMotion: Enable VMotion.

2015-01-23T11:43:12.798+04:00 avvcbimage Info <16038>: Final summary, cancelled/aborted 0, snapview 0, exitcode 202: plugin error 02

2015-01-23T11:43:14.907+04:00 avvcbimage Info <17819>: VixDiskLib vMotion reservation successfully released

--------------------------------------------------------------------------------------------------------

----- END avvcbimage log 2015-01-23 11:43:17 EST  (1 warning, 4 errors, 1 fatal error)

--------------------------------------------------------------------------------------------------------

I get this error message:

The most recent request has been rejected by the server

when I attempt to connect to VDP. I found this:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2040078

and tried running the command:

ntpq -d IP Address

but no offset values were presented, instead I was at the ntpq> prompt where I am clueless.

HELP me. How do I get my VDP back on track!

Thanks!

I am running a VDP Integrity Check on our VDP 5.5 instance. The system has been stuck on 99% for over 24hrs. I am not seeing an Memory issues as per some documentations I have seen. I have rebooted the system and tried doing it again but it still stays at 99%. Please let me know what I can do to troubleshoot this and what information will help you. Thanks for any help

When deploying VDP for a customer, i encountered the problem with disk attachment by VDP.

There is a knowledge base article written, which gives a workaround for the problem. The workaround states that setting the disk.enableUUID=true to "false" solves the problem for now.

But what is the impact of changing this value to false, when by default it's turned on.? How does this affect my VDP backups other that marking it is successful?

The KB article is published here: http://kb.vmware.com/kb/2035736

Hi All,

I recently deployed a VDP appliance to replace VDR in our 5.1 enviroment, backing up through a FC EMC CX3-40f SATA array.

As soon as I setup the appliance (which was a nightmare due to the DNS error message!), CPU usage shot up to 2-3Ghz and stayed this way throughout the day. Bear in mind this was with no backup job configured, and there was little disk IO during this period.

I logged in to the appliance using SSH, and using top, found that the Java processes are taking up most of the CPU - along with intermittant bursts from "gsan".

I then setup and kicked off a backup of our enviroment after hours, and saw that the CPU usage shot up to 10GHz - I had to stop it to stop it affecting other VMs.

I understand that this product will require heavy resources due to the dedup, but is this really normal behaviour? It's maxing out an HP G7 blade at the moment!

I have tried combinations of 1 - 4 vCPU, upping the memory to 16Gb and updating Tools.

Anyone else seeing similar behaviour, or have any advice?

Hi,

Has anyone tried to limit concurrent backups?

Default number (8) is just too much for our NAS and i'd like to have 4 or even 2 backups running at the same time.

Jeff Hunter mentions that it's possible:

Setting the record straight on VMware vSphere Data Protection | VMware vSphere Blog - VMware Blogs

"It is possible to change the number of proxies in use by running the registerproxy.sh script located in /usr/local/avamarclient/etc/. Please understand this is something that has not been tested and it is NOT SUPPORTED."

If someone has actually been able to do this I'd like to hear how it's done.

Thanks!

Using VDP to backup Windows 2008 R2 server - upgraded to 5.1.11 to fix the quiesced snapshot issue.

Some VMs are now backing up, but random others are getting this message:

Error code 30915: An attempt was made to backup a client in a group failed because the client does not have any data matching the data target specified in the dataset.

Google searches are proving fruitless....  any clues?

Scenario:

SBS 2011.  Exchange.

Attempting to use VDP advanced to backup.  Backup client loads successfully and points properly to VDP server.  MAPI CDO version 6.5.8224 successfully installed on server.    Image backup of the SBS server is successful.   Application database backup fails immediately.

Below is client log file from VDP:

I am trying to deploy vSphere Data Protection 5.5 ova appliance.

I am on my third attempt at installation; the previous two installed vm's I have deleted (or at least what I could find of them) after hacking them to death trying to get this thing to work.

I am the vdp-configure stage; but I am now stuck at the Create Storage page.

If I try to either "create" storage or "attach" storage I get the same error box: "cannot proceed because the appliance already has attached storage".

So now I am stuck.

Searching KB has come up with nothing.

I have checked the vm/ova settings (but not changed anything).

I have checked the datastore, nothing unusual there.

So: where do I look next?

Are there remnants of the first two installs hidden someplace in the guts of vcenter?

Can I ssh in to vm to check a setting/log file?

Any suggestions

Many of my customers have very simple installs with a single SAN.

I have a test environment right now where I have 2 esxi hosts and a san. One of the hosts has a local datastore. I am using the local datastore to house VDP and my backups. The idea being 1 this is a lab and 2 if i lose the SAN i probably wont lose the local datastore.

That being said i have 2 issues.

1. First i cannot add the vcenter server to the backup job. I suppose this could possibly be by design given my next question

2. Can you restore anything without the vcenter server? The file level recovery allows you to log directly into the VDP appliance from a protected host. My main purpose of a vm level backup is to restore everything if i lose the datastore. So if i have a virtualized vcenter server (i assume most do) and i lose my primary datastore i am going to lose both the protected virtual machines and the vcenter server. I have not going to lose the VDP data since it would reside on a seperate datastore. Can you initiate a restore directly from the VDP appliance without vcenter? If not this seems like a glaring oversite that should be changed. I would prefer to be able to backup vcenter with VDP and initiate a restore of vcenter directly from the VDP appliance if i lose everything.

Also are there any plans to make vdp managed from the vcenter client, the web client is not that great compared to the standard client.

What is the best way to make a full guest OS backup in the case that the hardware failed? I want to be able to simply re-install the hypervisor and restore the guest OS instead of reinstall everything from scratch. Can I do all this through vSphere Client or do I need a third-party software to do this?

Hi,

I have installed VDP 5.5.5 on 2 different subnet in a single VCenter. First VDP is in the same subnet (192.168.50.x) than the VCenter 5.5. No issues at all. It works like a charm. My other VDP 5.5.5 appliance is under another subnet (192.168.51.x) and both subnet have no issues to see each other. However, we I go to the Web Client and I try to start it I got this error message ...

Could not connect to the requested VDP Appliance.

The SSO Server could not be found. Please make sure the SSO configuration on the VDP Appliance is correct.

Would you like to be directed to the VDP Configuration utility to troubleshoot the issue?

How can I troubleshoot why it can't connect to SSO?

PS : The time is in sync. VCenter and VDP have the same time.

I have been getting this error on several VMs but have not been able to figure out why.  I found the following in the log file.  Can someone tell me why VDP cannot create the directories?  How can I fix?

2016-02-09T22:39:45.917+06:00 avvcbimage Info <11986>: Changed block tracking is engaged for this VM

2016-02-09T22:39:45.917+06:00 avvcbimage Info <11988>: A reference to a valid prior backup is not available so this will be a full level zero backup.

2016-02-09T22:39:45.959+06:00 avvcbimage Info <19549>: metadata tmp dir: /usr/local/avamarclient/var/vmware/2/temp

2016-02-09T22:39:45.959+06:00 avvcbimage Info <40640>: create tmpDir: /usr, failed (-1)

2016-02-09T22:39:45.959+06:00 avvcbimage Info <40640>: create tmpDir: /usr/local, failed (-1)

2016-02-09T22:39:45.959+06:00 avvcbimage Info <40640>: create tmpDir: /usr/local/avamarclient, failed (-1)

2016-02-09T22:39:45.959+06:00 avvcbimage Info <40640>: create tmpDir: /usr/local/avamarclient/var, failed (-1)

2016-02-09T22:39:45.959+06:00 avvcbimage Info <40640>: create tmpDir: /usr/local/avamarclient/var/vmware, failed (-1)

2016-02-09T22:39:45.959+06:00 avvcbimage Info <40640>: create tmpDir: /usr/local/avamarclient/var/vmware/2, failed (-1)

2016-02-09T22:39:45.960+06:00 avvcbimage Info <40640>: create tmpDir: /usr/local/avamarclient/var/vmware/2/temp, failed (-1)

2016-02-09T22:39:45.980+06:00 avvcbimage Info <40654>: isExitOK()=0

I'm using VDP 6.1.1

Hi,

I am unable to register VDP to vCenter, i am getting the following error "Unable to find this VDP in the vCenter inventory." or sometimes i am getting

"The connection test did not complete successfully. Modify your network settings or VDP password and then click "Test connection" to run the connection test again." I googled and followed many articles but still no success.

I rebooted my VPD appliance 1 hr ago and it has sat at this line now for as long.. What can I do. THe schedules replication and backup have started and all are failing.

Hi

I have a central site which is running vCentre 6 Standard.  There is a private MPLS network joining the 11 remote sites to the central site.  Each site will be added into the vCentre with its own site folder.

Each of the remote sites is running a single physical host with ESXi 6 hv and one application server vm.  Each remote site host has 2TB of usable storage capacity, the central site has 20TB of storage reserved for backup.

My plan is to install a VDP 6.1 appliance for each of the 11 sites (20 sites are supported in vCentre 6) on the central site vCentre, with each appliance pointing to the relevant remote sites host and using its local storage as back up capacity. 

Each site would have a back up job which would back up the application server on the remote host to the local appliance storage.  The remote site VDP appliances would be the 0.5TB versions and the central site offices VDP would bet the 8TB version.

The plan would be that each remote office would then replicate to the central site VDP appliance for offsite storage.

The VMware licensing for the hosts is ROBO Standard

Would this be possible in VDP 6.1?  The docs I have seen support it in principle but I haven't seen another design like it to confirm.

Hi,

I'm using the last version of VDP (6.1.1)

I succesfully backup VM but I have problem with other VM under Windows 2008R2

The backup start and stuck at 92%.

I've see on the other discussion but the bug concern older version.... I have no error messages and I don't know where to search.....

Have you an idea to adress this problem ?

Thanks

Hi,

Where I should look for problems when few of our VMs take very long time to backup.

We're using VDP appliance 6.1.1 and from the VDP SSH shell I can see current progress which looks like this:

root@vdp:~/#: /usr/local/avamar/bin/mccli activity show --active=true

0,23000,CLI command completed successfully.

ID               Status  Error Code Start Time           Elapsed     End Time             Type             Progress Bytes New Bytes

---------------- ------- ---------- -------------------- ----------- -------------------- ---------------- -------------- ---------

9145825200059909 Running 0          2016-03-18 09:13 EET 06h:33m:54s 2016-03-19 00:00 EET Scheduled Backup 49.1 GB        0.5%    

9145825200126109 Running 0          2016-03-18 07:30 EET 08h:16m:06s 2016-03-19 00:00 EET Scheduled Backup 35.8 GB        1.5%

9145825200058209 Running 0          2016-03-18 05:45 EET 10h:01m:08s 2016-03-19 00:00 EET Scheduled Backup 98.6 GB        6.6%

9145825200094609 Running 0          2016-03-18 00:42 EET 15h:04m:34s 2016-03-19 00:00 EET Scheduled Backup 139.7 GB       5.6%

9145825200073809 Running 0          2016-03-18 03:57 EET 11h:49m:00s 2016-03-19 00:00 EET Scheduled Backup 45.2 GB        1.9%

Even there is only 0,5 % of new bytes it takes 6+ hours to complete.

None of those VMs are larger than 200 GB and no other backups are running at the same time.

We have about 5 of these in every 500 VMs. Some are Linux, some are Windows and even different versions.  All of them have vmware-tools installed and virtual hardware is version 10.

I don't mind if backups are taking long but every now and then it takes so long that VM can not be backed up at all in VDPs backup window.

Network is 1 gbit and backup storage HP D2600 disk shelf with 12 x 3 TB NL-SAS disks in RAID6.

Any ideas?

Found many Warning in /usr/local/avamar/var/vdr/server_logs/vdr-server.log

2016-03-22 17:39:31,217 WARN  [Timer_general]-vi.VCenterServiceImpl: No VCenter found in domain 'esxihost.domain.com' which has 'VirtualMachines' subDomain

2016-03-22 17:40:01,246 WARN  [Timer_general]-vi.VCenterServiceImpl: No VCenter found in domain 'esxihost.domain.com' which has 'VirtualMachines' subDomain

2016-03-22 17:40:31,270 WARN  [Timer_general]-vi.VCenterServiceImpl: No VCenter found in domain 'esxihost.domain.com' which has 'VirtualMachines' subDomain

In 'esxihost.domain.com' was local esxi host FQDN where running that VDP instance.

Anyone have ideas what is reason for this warning?

All services running fine.

Only found what one of backup job stuck with 92% in vcenter gui.

Thru ssh show what no running jobs.

root@vpd:~/#: mccli activity show --active

0,23000,CLI command completed successfully.

ID Status Error Code Start Time Elapsed End Time Type Progress Bytes New Bytes Client Domain

-- ------ ---------- ---------- ------- -------- ---- -------------- --------- ------ ------

Message was edited by: Igor Rotbart

Hi everybody,

I have the vSphere Data Protection 6.1 (v. 6.1.0.173) in my environment and daily I do a backup of my database virtual machine. Normally it tooks about 2-3 hours to do that but with no reason in the last 2 weeks the duration has grown up till 10-12 hours.

I have downloaded the logsfile but I have some problems to analyze it.

What can I do and what I need to search to understand what's the problem with the duration of this backup?

Thx a lot in advance, greetings