Hi,
I have depolyed VDP 6.1 and have it up and running. now im trying to connect to the appliance frpm vsphere Web client. After pressing connect the screen goes grey and nothing happens.
No error message no time out, just grayed out with timeglas not moving.
DNS is ok
account for vdp in vcenter ok
all services for appliance are ok
vcenter 5.5 ESXI 5.5
Regards
mgd
I am getting an error when trying to backup my vCenter server through VDP:
VDP: Backup job (vCenter VMs) failed to backup client V Center 2 64. Execution error: E10056:Restore failed due to existing snapshot.. Job Id: vCenter VMs-1348589067036. (Full client path: )
As far as I can tell there are no snapshots for the vCenter VM in question. How should I proceed in resolving this and getting a successful backup of my vCenter server?
estimnated time to complete: 10 minutes. But it has been much longer than 10 minutes. Powered off and powered on. Didn't help
We have using CA Arcserve and VCB to perform the backup of VM's
Recently i've noticed that some VM's are no longer reporting performance data in the "Performance" Tab of VCenter.
Furthering digging, show the account used to perform the backup snapshots is causing an an error "Another task is already in progress"
I suspect one of back jobs has been cancelled and the snapshot 'clean up'process that arcserve uses has left the VM in some state....other than normal.
What steps can I take to fix this error...i've tried rebooting the VM but this don't fix the issue.
Most backup jobs are hanging at 92% completion. I see some posts about recovery jobs hanging, but nothing about backups. Logs on this beast are not user frendly so I've got no direction yet. Any ideas what might be hanging these?
One of my VDP appliances won't run integrity checks. When I try to run an integrity check manually I get the following error.
My issue is almost identical to the following post:
http://communities.vmware.com/message/2149800#2149800
However I've tried rebooting the vdp and I've checked the service status, and they're all running. Wierd thing is when I try to run a manual integrity check the Maintenace service stops itself then restarts right before I recieve this error message above. I also have the following errors every morning in the event log.
Hi,
I cancelled in the middle of snapshot deleting process all the snapshot configuration file deleted. I able up the vm but the data's are available before the snapshot date. In datastore i can see delta & snapshot files how merge/recover data from delta.vmdk.
WHMSG.vmdk
WHMSG-flat.vmdk - 40GB
WHMSG_1.vmdk
WHMSG_1-flat.vmdk - 500 GB
WHMSG_1-00001-delta.vmdk - 75GB -> i want restore/merge this vmdk.
note: 2 Hard disk (40GB & 500 GB)
Pls help
First thing first, here is the background information.
1. vSphere has been upgraded from 4.1 to 5.0 update 2 and the problem started. We use Symantec Netbackup 7.5 with NBD transport mode.
2. We have mixed storage of HDS (VMFS) and Netapp (NFS). So far the issue only exhibits on Netapp NFS datastores. Most of the VMs being backed up are running on netapp too. Don't know if it helps but it is better to have more information.
3. We have large amount of VMs (400+) being backed up nightly, most of them are incremental backup. Before backup start, Netbackup calls vCenter to create a snapshot. After backup completes, Netbackup again calls vCenter to remove snapshot.
The problem we have is sometimes random VMs would have delta files (vmname-000001.vmdk) left behind in the VM folder. It is not used by vmx file which means it is completed orphaned so I can remove it manually. And manual removal seems to be the only way to get rid off the orphaned delta files (I am still searching for a script to identify and remove such files. I am grateful If you can share). I tried create a new snapshot then delete all, I also tried "consolidate" option in snapshot manage. Neither of them helped.
The reason why I know it is happening on random VMs is we have a script to search through each datastore for delta files (trust me regular script to search snapshot doesn't work). And every day we have different affected VMs. And funny thing is normally the orphaned delta files are only 16MB or 17MB.
In the screenshot above, the delta file vmname-00002.vmdk is the orphaned file I am talking about. Looking at the vmx file, it is pointing to the base/flat file which is normal. Because the snapshot is considered as "deleted/cleaned". We know this by looking at the vCneter history.
I have created a ticket to VMware support but was told Symantec is needed because we or VMware supoport can't manually reproduce the issue in house. They believe something is wrong while netbackup calls the API. Symantec support is engaged too but they don't seem to have a clue either.
I am wondering if anyone knows where else to look.
Hey Guys,
I'm trying out the new VDP 6.0 to see if it can replace our existing backup solution and so far so good except for File Level Restores.
I manage to get the VM's backed up and I know that in order to do a "File Level Restore" or FLR, I have to connect to https://<IP_of_your_VDP_appliance>:8543/flr from the VM itself. I would like to know if I can do a restore from another machine like my own workstation? This would make it a little easier for me or the helpdesk guys to perform restores but I can't figure out how to do this.
Anyone?
Nick
Hello,
This has been an ongoing issue with the last few VDP versions.
I update vSphere to the latest version (in this case 6.5.0 5224529) and then the VMware tools on VDP appliances throws a warning that it is out of date.
I learned the hard way that you cannot simply perform the "Upgrade VMware Tools" automatically... it just uninstalls it and then cannot install the new version.
In the past I would just update to the newest version of VDP (in this case 6.1.4) and the problem would go away.
However, it hasn't gone away...
1. Is this problem really hurting anything? Backups seem to be ok.
2. Is there a KB or something that acknowledges this issue and/or provides a fix?
Thanks!
I am helping a buddy out and running into roadblocks for his problem.
I'm not a Veeam guy and stay away from the snapshots when I can so with that....
I am seeing a ton of ghost? missing in action snapshots or phantom open snaps when i browse the datastore but when in snapshot manager there is nothing there.
I have read taking a snapshot and checking the Quiesce Guest file system, then deleting the open snap. However this works 75% of the time I run into a VM that I have to power the system down clone the machine and wait.... then power on the clone and delete the troublesome VM. Time to time I get this error
"An error occurred while consolidating disks: msg.fileio.lock."
I am wondering if this is the case of Veeam or snapshots in general for any other snap based backup.
Personally I am a user of BackupExec and just doing file level backups.
I am adding a screenshot from a datastore of the problem I am seeing.
Any thoughts would be great.
Hi
We use Veeam BR to backup our Vmware environment, recently we have started so see Failed to create NFC download stream. The strange thing is
that Veeam retryies the job and then it goes sucessfull.
Veeam support response
So, SOAP connection to the host from the Veeam B&R server works intermittently - on the first run of the job it doesn't work, but works on retry. You could supply the VMware support with this information.
[22.01.2013 18:25:37] <01> Error Client error: Cannot authenticate user.
[22.01.2013 18:25:37] <01> Error Soap fault. Detail: '', endpoint: ''
[22.01.2013 18:25:37] <01> Error SOAP connection is not available. Connection ID: [XXXXXXXXXX].
[22.01.2013 18:25:37] <01> Error Failed to create NFC download stream. NFC path: [nfc://conn:XXXXXXXX,nfchost:host-43586,stg:datastore-50477@FSU298/FSU298.vmx]
(X = masking my vcentername
/regards
Christer
Used Capacity on one of our VDP reached 96.46% and Backup Scheduler was stopped. I'm able to start Backup Scheduler. But although I deleted many backups the used capacity is still the same. Is any way to freeing up space or should I better do rollback to the last validated checkpoint?
Good Morning All,
Thanks for taking the time to look!
I have an issue with numerous virtual servers on our server estate.
When VEEAM backs-up the server it takes a snapshot, when the back-up is complete it removes the snapshot, this then causes the server to hang and loose connectivity for anywhere up to 1 hour, thus causing an outage to end users.
I have approached VEEAM and they advised this was HA monitoring, I have disabled this but still the issue occurs.
I have also disabled CBT (Change Block Tracking) in the VEEAM back-up job which requires that all snapshots be removed, the same thing happens.
The odd thing is, this does not occur on every single back-up, almost on a monthly basis.
Any feedback or input would be greatly appreciated.
Thanks,
San
I've just updated VDP to 5.1.10.32 following Administration Guide. I removed snapshot after update but I'm not able to set disks to "Independent - Persistent“ mode again. I've got "File []/vmfs/volumes/5093aae5-979ec04c-8556-180373ede2e6/vSphere Data Protection/vSphere Data Protection_2-000001.vmdk was not found“ message first and "Cannot complete operation due to concurrent modification by another operation." messages now. No change after reboot.
Could it be related to not running VMware Tools in vCenter Appliance?
Hi,
I've recently inherited a vCenter environment where a small number of the VMs are being backed up using VDP 6.1. The backups are currently failing and I'm not really getting anywhere with troubleshooting as I'm struggling to get the logs.
The job fails immediately, from inside the vSphere client I recieve;
Reason: VDP: An unexpected error occurred with the following error code: 30983.
Log file retrieved is empty.
When trying to download the client logs from the appliance website I receive the following error:
Download failed: I/O error.
Possible cause could be insufficient disk space. Please check available space on the download location.
Definitely sufficient space. I've connected to the appliance and check the client job logs manually and there doesn't seem to be any there.
There is 8TB of storage available so we've not run out there.
Not to sure where to go from here.
Thanks.
Is there any backup solution (freeware, if possible) for VMware ESXi free version?
Regards
marius
Dear all, I have been knocking my VDP about trying to fix it. It had lost data02, and I got that back by running a filecheck discarding the metadata stuff. So, I was then able to gain access to the rollback, but rollback failed.
Now the status looks like this:
root@vdp:~/#: dpnctl status
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: down.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: axionfs status: up.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
I found this article: Starting the MCS service in VMware vSphere Data Protection 5.1.x fails with the error: gsan rollbacktime does not match … and attempted to run the command specified in there, but it is not successful.
I get this:
root@vdp:~/#: dpnctl start mcs --force_mcs_restore
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: Restoring MCS data...
Permission deniedPermission denied at /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi/XML/LibXML.pm line 587.
Could not create file parser context for file "/usr/local/avamar/var/mc/server_data/prefs/mcserver.xml": Permission denied at /usr/local/avamar/lib/MCServer.pm line 128
dpnctl: ERROR: cannot restore MCS data without a password for MCS administrative user id
dpnctl: ERROR: MCS restore did not succeed, so not restarting MCS
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
So my question is really... How do I specify the password for the MCS administrative user id so that I may force MCS up again?
If I try to start MCS using dpnctl start mcs I get the following:
root@vdp:~/#: dpnctl start mcs
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: Starting MCS...
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-6153
dpnctl: ERROR: error return from "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start" - exit status 13
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
This is what the dpnctl.log looks like:
root@vdp:~/#: tail /usr/local/avamar/var/log/dpnctl.log
2016/06/30-12:50:18 [ "/bin/cat /tmp/dpnctl-get-mcs-status-status-5787 2>&1" exit status = 0 ]
2016/06/30-12:50:18 dpnctl: INFO: "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --test" exit status = 13
2016/06/30-12:50:18 rm -f /tmp/dpnctl-get-mcs-status-status-5787 /tmp/dpnctl-get-mcs-status-output-5787
2016/06/30-12:50:18 dpnctl: INFO: "rm -f /tmp/dpnctl-get-mcs-status-status-5787 /tmp/dpnctl-get-mcs-status-output-5787" - exit status 0
2016/06/30-12:50:18 dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
2016/06/30-12:50:18 rm -f /usr/local/avamar/var/dpnctl.lck
2016/06/30-12:50:18 ======= dpnctl 7.1.80-129 (1.139), running as admin, RETURNING at 2016-06-30 14:50:18 CEST =======
2016/06/30-12:50:18 [user "admin"] program (pid 5787) exit status = 1 (error)
2016/06/30-12:50:18 ======= dpnctl 7.1.80-129 (1.139), running as root, ENDING at 2016-06-30 14:50:18 CEST =======
2016/06/30-12:50:18 [user "root"] program (pid 5729) exit status = 1 (error)
Not a lot to work with here... Logfile has absolutely no info that I can understand...
Message was edited by: Ludvig F Aarstad: added more info
Wow, it was as simple as typing su - admin before running the dpnctl start mcs force_mcs_restore command...
Message was edited by: Ludvig F Aarstad: Added solution...
Hi all,
we have a fairly large vSphere 5.0 environment with 2x vCenter servers, each with ca. 2000 VMs using about 600 TB of FC-attached SAN storage.
The majority (~90%) of these VMs are backed up with Symantec NetBackup 7.5 via the vStorage API (doing VMware snapshots through vCenter).
While the environment was growing the backup environment also grew by adding more and more media servers to it. But we are still using only one vCenter server per location.
Since a while we are having trouble with a growing number of backup job failures: It looks like the backup servers' connections to vCenter sometimes fail or time out in the middle of a job.
And the vCenter servers (VMs with 8 vCPUs and 32 GB RAM) appear to be heavily loaded (esp. CPU, but also RAM).
My question here is: Does this happen, because we have reached an architectural limit of such kind of backup setup? Or in other words: Do we need to add additional vCenter servers and spread hosts and VMs among them?
We expect further growth of this environment and need to take a decision quickly. I cannot provide any details about how the backups are done right now. I'd just like to hear some opinions from people who run comparable environments and also use snapshot based backups. Should one vCenter server be able to handle backups of more than 2000 VMs? Is it worth trying to scale up the one vCenter server?
Thanks for any comments!
Andreas
