Hello,

I am attempting to set up a Hortonworks sandbox on our VSphere 5.5 lab environment.  We have deployed many times before and have had various problems, but this is a new one to me.

After selecting the OVA file from my local disk within the File/Deploy OVF Template wizard, and selecting where to store the VM (we use SAN), it errors out after I click "Finish" with the error "Failed to Deploy OVF package: A general system error occurred: Failed to write to C:\ProgramData\VMware\VMware VirtualCenter\journal\1433426209.11: Error writing file.  There is not enough space on the disk."

I don't know why it is trying to write a journal file to my local workstation, but at any rate the path referenced in the error did not exist.  I tried creating those folders but the same error was returned.

Note that this is a 5.41 GB OVA file and the local disk on my workstation has 244 GB free.

I then tried adding additional permissions (Users did not have Write) but it still threw the error.  I noticed that the file properties page for the OVA showed it as blocked since it came from a different computer so I unblocked it, but again had no luck with the deploy...same error.

I assume that the path is written during the conversion of a VM at Hortonworks to an OVA (Export?) but I have no idea how, or even if I could change it.  Or even if changing the path would help.

Has anyone else run into this issue and is there a work-around that you can provide?

Thanks is advance for any help,

Rick

Hi,

I'm using the last version of VDP (6.1.1)

I succesfully backup VM but I have problem with other VM under Windows 2008R2

The backup start and stuck at 92%.

I've see on the other discussion but the bug concern older version.... I have no error messages and I don't know where to search.....

Have you an idea to adress this problem ?

Thanks

Hello,

Running vCenter 6.0 and VDP 6.1 and after launching a backup job ( through the vSphere web client) I can see the job start but only using the viclient's recent tasks ( it DOES NOT show any status or progress through the web client).  Now what makes this problem an issue is that one of the VM's I had in the backup job failed to create a shapshot.  The ONLY REASON I know this is because the viclient recent tasks showing the progress of the backup job alerted me to the issue.  Even after a refresh several times in the webclient there is not indication that the job was ever even run, completed, or FAILED under the Recent Tasks section.  Looking in the Alarms section of the web client also fails to show any indication that there was a failed snapshot on one of the backup jobs.  The only way to view the status was to look at the Tasks Console in the web client and there I could also see the job and failed backup but unless you are already looking for the failure you would never have known.

It is very concerning when VMware is pushing clients to use the web client more and the viclient less yet seems we keep losing functionality.  Has anyone else had this issue with VDP 6.1?  If so is there some fix to allow "RECENT TASKS" to actually show the same status in the web client and in the viclient ( They are both talking to the same vCenter so why dont they contain the same information????)  The exact builds I am using are the following

vCenter Server 6.0.0 build 3018524

VDP 6.1.1.19

ESXi 6.0.0 build 3620759

And before its suggested the run the VDP 6.1.2 ( latest version) I already tried that and that version had bigger issues freezing up and failing so the version I am using seemed to be more stable :-)

I have about 50 VMs configured in SRM 5.8, and I'd like the have one recovery plan that can be used to start everything up at as quickly as possible.  But I need to have some delays between the VMs to spread out the startup workload.  I thought I could use priority groups for this.  There can be only five priority groups, but I thought that would be spread out the workload enough to avoid timeouts in service startups or in waiting for VMware tools, etc.  The problem is that IP customization, which includes a startup and shutdown of the guest VM, is not affected by priority groups.  In the case of a recovery plan test, every VM startup includes seven sub-steps:  1) Configure Storage 2) Configure Test network 3) Guest Startup 4) IP customization 5) Guest Shutdown 6) Power On 7) Wait for VMware Tools.  Priority groups only affect the last two sub-steps.

So, not only is the description of this feature very deceptive about what it does, it gives me no way to avoid my hosts and storage from getting slammed with 50 simultaneous VM startups (and shutdowns.)

VMware support suggested I use the base VM boot delay to introduce delays for each VM.   That could work, but would also affect ever restart in my production environment.  Does anybody have any better ideas?  Does anybody else agree this feature should be modified to include the IP customization steps?

Community,

I'm posting this weird error in case it helps someone else. It feels like a silly oversight on my part after finding/fixing it.

I was running into a reoccurring error when trying to do a database level backup on my SQL server.  I was attempting to do a database level backup using the VDP SQL agent on a SQL 2012 server.

The database backup failed with the following information in the VDP plugin log:

"Error: Miscellaneous error."

(Not very useful, so I began pulling logs from the appliance and reviewing them.  Not real helpful there either.

(The job initially failed with permissions errors until I changed the backup agent windows service to run as an account that had appropriate SQL permissions, this seemed like a 'no-brainer'.)

Logs

I checked the logs on the SQL server, it and the Windows Event Log and found this mirrored in both:

SQL Shows (readerrorlog):

Error: 18210, Severity: 16, State: 1.

BackupIoRequest::ReportIoError: write failure on backup device '%server name removed%_%instance name removed%_SUSDB_SQL-Production-Daily-1465216797749-3006-SQL'. Operating system error 112(There is not enough space on the disk.).

Error: 3041, Severity: 16, State: 1.

BACKUP failed to complete the command BACKUP DATABASE SUSDB. Check the backup application log for detailed messages.

Error: 18210, Severity: 16, State: 1.

BackupVirtualDeviceFile::RequestDurableMedia: Flush failure on backup device '%server name removed%_%instance name removed%_SUSDB_SQL-Production-Daily-1465216797749-3006-SQL'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).

Thoughts:

1. Free space? Maybe it needs to copy DB files?

-The database was 1.7GB.  Both the OS drive (C:) and the data drive (D:) had plenty of free space.  I can't see how there wouldn't be enough room on the VM for a copy of the database if that's what's taking place first.

-The VDP appliance is only at 30% utilization of his storage space.

2. Permissions?

-I was running the backup agent windows service as a service account who is presently a 'sysadmin' to remove permissions issues.  That same account had been doing a powershell SQL backup for a long time, that runs successfully.

Simple Solution

The backup agent service simply wanted to write logs to the 'C:\Program Files\avp\var\' and 'C:\Program Files\avp\var\client logs\' folders and didn't have permissions because the service account wasn't listed on the NTFS permissions.  Once I added the service account and give him write permissions, the jobs worked as expected.

Environment

VDP 6.1

vCenter 6

ESXi 5.5U2

SQL Server VM - Server 2012

SQL 2012

Hello,

After a Storage VMotion of the VDP appliance, it became unresponsive. I'm not able to manage it through vCenter nor its web interface. I guess it's something related to its internal proxy.

When I try to enable it, it fails. I've already rebooted the VDP appliance and the VCSA. Also, I've checked via CLI the health of the VDP services and the backup data. Seems fine.

I've also tried to unregister VDP VM and register again. I need help

My environment:

ESXi 6.0 U2

VCSA 6.0 build 3634794

VDP 6.1

Here it comes some screenshots and logs of the issue:

/usr/local/avamar/var/vdr/server_logs/vdr-server.log

2016-06-15 11:48:12,020 ERROR [Timer_CustomValue]-vi.ViJavaAccess: doGetAveVirtualMachine(): Unable to obtain local AVE virtual machine from its IP address [10.12.73.253]
2016-06-15 11:48:12,021 ERROR [Timer_CustomValue]-vi.ViJavaAccess: getAveVirtualMachine(): Unable to obtain local AVE virtual machine. Might be an Avamar physical machine.
2016-06-15 11:48:12,021 ERROR [Timer_CustomValue]-vi.ViJavaAccess: Failure to determine local AVE instance-uuid
2016-06-15 11:48:12,061 ERROR [Timer_CustomValue]-schedule.CustomValueUpdateTimerTask: Could not update the custom values.
java.lang.NullPointerException        at java.lang.String.contains(Unknown Source)        at com.emc.vdp2.schedule.CustomValueUpdateTimerTask.checkAndUpdateAnnotation(CustomValueUpdateTimerTask.java:97)        at com.emc.vdp2.schedule.CustomValueUpdateTimerTask.run(CustomValueUpdateTimerTask.java:72)        at java.util.TimerThread.mainLoop(Unknown Source)        at java.util.TimerThread.run(Unknown Source)
2016-06-15 11:48:16,869 ERROR [Timer_ServiceHealthValidator]-vi.ViJavaAccess: doGetAveVirtualMachine(): Unable to obtain local AVE virtual machine from its IP address [10.12.73.253]
2016-06-15 11:48:16,869 ERROR [Timer_ServiceHealthValidator]-vi.ViJavaAccess: getAveVirtualMachine(): Unable to obtain local AVE virtual machine. Might be an Avamar physical machine.
2016-06-15 11:48:16,869 ERROR [Timer_ServiceHealthValidator]-schedule.AlarmCheckTimerTask: Failed to get aveVM. aveVM is null
2016-06-15 11:48:16,869 INFO  [Timer_ServiceHealthValidator]-schedule.ServiceHealthValidatorTimerTask: Starting Service Health Check on Wed Jun 15 11:48:16 BRT 2016
2016-06-15 11:48:16,878 INFO  [Timer_retireVM]-schedule.RetireVirtualMachineTask: Searching for virtual machine registrations to retire...
2016-06-15 11:48:16,879 ERROR [Timer_HFSValidator]-vi.ViJavaAccess: doGetAveVirtualMachine(): Unable to obtain local AVE virtual machine from its IP address [10.12.73.253]
2016-06-15 11:48:16,879 ERROR [Timer_HFSValidator]-vi.ViJavaAccess: getAveVirtualMachine(): Unable to obtain local AVE virtual machine. Might be an Avamar physical machine.
2016-06-15 11:48:16,880 ERROR [Timer_HFSValidator]-schedule.AlarmCheckTimerTask: Failed to get aveVM. aveVM is null
2016-06-15 11:48:17,136 INFO  [Timer_retireVM]-connection.Mcsdk10StubManager: ServiceInstanceMoref desc=Service Id: urn:uuid:08FCE6D79ECB43283E1466002109508 name=urn:uuid:08FCE6D79ECB43283E1466002109508 value=SERVICE
2016-06-15 11:48:17,186 INFO  [Timer_retireVM]-connection.McAccessManager: Creating new mcsdk stub handler for connection key: [1151234205, Service Id: urn:uuid:08FCE6D79ECB43283E1466002109508] on Thread: [Timer_retireVM]
2016-06-15 11:48:17,240 INFO  [Timer_HFSValidator]-schedule.HFSValidatorTimerTask: HFS ValidRangeHours 48
2016-06-15 11:48:17,240 INFO  [Timer_HFSValidator]-schedule.HFSValidatorTimerTask: Last validated checkpoint happended on Tue Jun 14 15:06:25 BRT 2016
2016-06-15 11:48:17,360 INFO  [Timer_ServiceHealthValidator]-schedule.ServiceHealthValidatorTimerTask: Internal proxy configured?true
2016-06-15 11:48:17,360 INFO  [Timer_ServiceHealthValidator]-services.BaseCLI: GsanCLI: Updating service state.
2016-06-15 11:48:18,421 INFO  [GsanCLI-update]-services.BaseCLI$4: GsanCLI-update: State update complete. New state: running
2016-06-15 11:48:18,421 INFO  [Timer_ServiceHealthValidator]-services.BaseCLI: McsCLI: Updating service state.
2016-06-15 11:48:18,441 INFO  [McsCLI-update]-service.AdapterUtils: MCS Web Services URL: https://vdp6.epl.gov.br:9443/services/mcService  MCUserId="MCUser"
2016-06-15 11:48:18,632 INFO  [McsCLI-update]-connection.Mcsdk10StubManager: ServiceInstanceMoref desc=Service Id: urn:uuid:08FCE6D79ECB43283E1466002111165 name=urn:uuid:08FCE6D79ECB43283E1466002111165 value=SERVICE
2016-06-15 11:48:18,650 INFO  [McsCLI-update]-connection.McAccessManager: Creating new mcsdk stub handler for connection key: [1151234205, Service Id: urn:uuid:08FCE6D79ECB43283E1466002111165] on Thread: [McsCLI-update]
2016-06-15 11:48:18,699 INFO  [McsCLI-update]-services.McsCLI: MC Connection with Key: [McConnectionKey URI: https://vdp6.epl.gov.br:9443/services/mcsdk10 Username: MCUser Locale: en_US mcDomain: / trustAllSSLCerts: true productName: VDP] isConnected: [true]

Dear all, I have been knocking my VDP about trying to fix it. It had lost data02, and I got that back by running a filecheck discarding the metadata stuff. So, I was then able to gain access to the rollback, but rollback failed.

Now the status looks like this:

root@vdp:~/#: dpnctl status

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: gsan status: up

dpnctl: INFO: MCS status: down.

dpnctl: INFO: Backup scheduler status: down.

dpnctl: INFO: axionfs status: up.

dpnctl: INFO: Maintenance windows scheduler status: enabled.

dpnctl: INFO: Unattended startup status: enabled.

dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

I found this article: Starting the MCS service in VMware vSphere Data Protection 5.1.x fails with the error: gsan rollbacktime does not match … and attempted to run the command specified in there, but it is not successful.

I get this:

root@vdp:~/#: dpnctl start mcs --force_mcs_restore

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: Restoring MCS data...

Permission deniedPermission denied at /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi/XML/LibXML.pm line 587.

Could not create file parser context for file "/usr/local/avamar/var/mc/server_data/prefs/mcserver.xml": Permission denied at /usr/local/avamar/lib/MCServer.pm line 128

dpnctl: ERROR: cannot restore MCS data without a password for MCS administrative user id

dpnctl: ERROR: MCS restore did not succeed, so not restarting MCS

dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

So my question is really... How do I specify the password for the MCS administrative user id so that I may force MCS up again?

If I try to start MCS using dpnctl start mcs I get the following:

root@vdp:~/#: dpnctl start mcs

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: Starting MCS...

dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-6153

dpnctl: ERROR: error return from "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start" - exit status 13

dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

This is what the dpnctl.log looks like:

root@vdp:~/#: tail /usr/local/avamar/var/log/dpnctl.log

2016/06/30-12:50:18 [ "/bin/cat /tmp/dpnctl-get-mcs-status-status-5787 2>&1" exit status = 0 ]

2016/06/30-12:50:18 dpnctl: INFO: "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --test" exit status = 13

2016/06/30-12:50:18 rm -f /tmp/dpnctl-get-mcs-status-status-5787 /tmp/dpnctl-get-mcs-status-output-5787

2016/06/30-12:50:18 dpnctl: INFO: "rm -f /tmp/dpnctl-get-mcs-status-status-5787 /tmp/dpnctl-get-mcs-status-output-5787" - exit status 0

2016/06/30-12:50:18 dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

2016/06/30-12:50:18 rm -f /usr/local/avamar/var/dpnctl.lck

2016/06/30-12:50:18 ======= dpnctl 7.1.80-129 (1.139), running as admin, RETURNING at 2016-06-30 14:50:18 CEST =======

2016/06/30-12:50:18 [user "admin"] program (pid 5787) exit status = 1 (error)

2016/06/30-12:50:18 ======= dpnctl 7.1.80-129 (1.139), running as root, ENDING at 2016-06-30 14:50:18 CEST =======

2016/06/30-12:50:18 [user "root"] program (pid 5729) exit status = 1 (error)

Not a lot to work with here... Logfile has absolutely no info that I can understand...

Message was edited by: Ludvig F Aarstad: added more info

Wow, it was as simple as typing su - admin before running the dpnctl start mcs force_mcs_restore command...

Message was edited by: Ludvig F Aarstad: Added solution...

Did you run a VDP performance test and did it pass?

hello

can we have some news on a possible new version of vdp ?

thanks,

Hi All

I have been using VDP for a few weeks now and it has been running without too many issues. In the last day or so I have seeing backups fail on 2 of my appliances. I would like to know the reason for the failure so I putty to the appliacne and started to look through the logs in /usr/local/avamarclient/var-proxy-x but I am finding it extremely difficult, none of the logs are labeled with the friendly VM names but insteand the log file names use the vCenter machine ID or some such identifier. I was wondering if somone could give some assistance with

1. Easiest way to find the log for a particular VM

2. What are the best logs to look at to ascertain the actual reason for the backup failure

If anyone has any other tips and tricks for interpreting the logs please let me know.

Thanks a lot guys,

Good day,

I'm using vSphere Data Protection basic 5.5 to backup my VMs. Some of my backup job failing to backup. Error from client log file:

2013-12-11T21:01:09.512-5:00 avvcbimage Warning <18652>: Too many extra snapshot files (1) were found on this Windows VMs datastore. This can cause a problem for the backup or restore.

2013-12-11T21:01:09.512-5:00 avvcbimage Info <18678>: Extracted NEW vmnumber1-flat.vmdk base disk from the VMX file

2013-12-11T21:01:09.512-5:00 avvcbimage FATAL <0000>: [IMG0002] The datastore information from VMX '[Datastore1] NEW vmnumber1/vmnumber1.vmx' will not permit a restore or backup.

2013-12-11T21:01:09.512-5:00 avvcbimage Info <9772>: Starting graceful (staged) termination, Too many pre-existing snapshots on the datastore will not permit a restore or backup. (wrap-up stage)

No snapshots present on that VM.

No delta files in the vm folder.

Tried consolidating but still no luck.

Any help/pointers is appreciated.

-Ali

I have vmware data protection 5.5.6

at this time are not running backups.

I'm trying to move the virtual machine to another datastore and I get the following error:

The method is disabled by 'vSphere Data Protection'

Error stack:

Call "VirtualMachine.Relocate" for object "XXXX" on vCenter Server "XXXX" failed.

this happens to me all the virtual machines that are scheduled backup.

Please your help to move the machines.

Hi,

We have a problem since we upgrade to ESXI 6.0

Many Backups have these error in Veeam : invalid CBT data

After search i find a kb in vmware : VMware KB: Backing up a virtual machine with Changed Block Tracking (CBT) enabled fails after upgrading to or instal…

And i install the good patch yesterday.

But this night we have the same problem and my question is :

Do that backup need to turn one time after patch with error forautomatically reset the CBT?

thank you for your help.

Hello,

as stated above my backup job is failing and I don't know why.

From the vSphere Web Client I get the following message on the task console:

And from the logs I deducted that the important part is the one below:

/usr/local/avamarclient/var/Backup_test....log

All my software if version 6.0.

If there is anymore information I can provide please tell me.

Many of my customers have very simple installs with a single SAN.

I have a test environment right now where I have 2 esxi hosts and a san. One of the hosts has a local datastore. I am using the local datastore to house VDP and my backups. The idea being 1 this is a lab and 2 if i lose the SAN i probably wont lose the local datastore.

That being said i have 2 issues.

1. First i cannot add the vcenter server to the backup job. I suppose this could possibly be by design given my next question

2. Can you restore anything without the vcenter server? The file level recovery allows you to log directly into the VDP appliance from a protected host. My main purpose of a vm level backup is to restore everything if i lose the datastore. So if i have a virtualized vcenter server (i assume most do) and i lose my primary datastore i am going to lose both the protected virtual machines and the vcenter server. I have not going to lose the VDP data since it would reside on a seperate datastore. Can you initiate a restore directly from the VDP appliance without vcenter? If not this seems like a glaring oversite that should be changed. I would prefer to be able to backup vcenter with VDP and initiate a restore of vcenter directly from the VDP appliance if i lose everything.

Also are there any plans to make vdp managed from the vcenter client, the web client is not that great compared to the standard client.

hi guys

suddenly my Backup Scheduler won't start (I can see that in status VDP configure tab). It was working fine.

I manually start it. No go.

I used command: dpnctl stop mcs - dpnctl start mcs. Same thing it won't work. Logs for this process

any idea what could be causing the issue? Which logs to check?

thanks

Hi,

I have a strange issue with vdp 6:

after executing a backup job the vdp appliance shutdown and generate an error saying: A query send operation failed

any help please?

thanks,

Noura

Hi,

I have imported the VDR OVF appliance into VC, powered it on and got a VM Console.  VM has DHCP address and I can see it on the network.  I have logged into the console as the default user and set my own root password.  I have logged out and back in to test it's correct.

I browse to https://vdr-server:5480 and get the login prompt for the web console and supply the credentials root & my password.

I then have a 2-3 minute wait and then get the following error;

!https://10.105.6.22:5480/images/exclamation.gif!Invalid server response, CIM server is unreachable. Please try again.

Can anyone help please?

Hi all,

We're using VDR 1.0.2. Often I get the the following error until I reboot the vdr-appliance:

Failed to create snapshot for XXXX, error -3902 ( file access error)

When I get this error-message, I although can create a snapshot manually. We' re using a CIFS-Share for our vdr-backup-destination.

Any hints?

Thanks a lot.

Tom

I've just started playing with VCB, and I have a basic question.  I'm testing vcbmounter, with syntax like this:

vcbmounter -h  supposed to be the actual ip address (or hostname) of the guest operating system?

I'm hoping my VCB Proxy doesn't need access to the same networks that my guest operating systems are on.  My VM's are on different VLANs, and in one case a completely different physical network - but all using the same iSCSI SAN.  So, my VCB Proxy has access to two networks - The iSCSI network and the VMWare management network (where my service consoles and vCenter server reside).

I'm hoping that the VCB Proxy (where I'm running vcbmounter) doesn't need access to the networks that my VM's are on.

Can someone clarify for me?