Hi all,
On 9 September at 13:41 Thinware vBackup job failed with the error below.
Irrelevant info skipped for brevity.
Version: 4.0.1.2747 (Version 4.0, Update 1)
Process Started: 09/09/2017 14:41:26
Execution Arguments:
- Virtual Machine = guest1
- Server = host1:22
- Job Type = Backup-Image-SSH
- Backup Root = D:\
- Disk Exclusion = None
- Backups to Keep = 1
- Quiesce Guest File System = True
- Snapshot Virtual Machine Memory = False
- Create Snapshot Task Timeout = 15
- Remove Snapshot Task Timeout = 15
- Encrypt Temporary Password File = True
- Compression Level = Advanced
- Maximum Log Size = 1024
- Backup Type = Image-Full
Server Platform: VMware ESXi 6.0.0 3620759
Virtual Machine Guest OS: Microsoft Windows 7 (64-bit)
License:
- Type: Perpetual
- Status: Active
- Edition in Use: Professional
Backup Folder: D:\\guest1
Backup: 170909_144141
No existing temporary snapshots found.
Temporary Snapshot: vBackup-TMP-170909_144141
VMware Virtual Disk Manager: VMware Virtual Disk Manager - build 2248791
VMware Virtual Disk Manager response for Hard disk 1: [datastore2-SATA] guest1/guest1.vmdk:
SSL_VerifyCbHelper: Certificate verification is disabled, so connection will proceed despite the error
[NFC ERROR] NfcNet_Recv: requested 264, recevied only 0 bytes
[NFC ERROR] NfcFile_Receive: Failed to get message
[NFC ERROR] NFC_NETWORK_ERROR
Failed to convert disk: Operation completes asynchronously (0x300004650).
E Unable to complete backup. Unable to verify successful transfer of virtual machine virtual disk file D:\\guest1\170909_144141\guest1.vmdk.
e Unable to remove temporary snapshot vBackup-TMP-170909_144141. Unable to retrieve snapshot ID. Error while creating 'snapshot' file. Failed to login: Operation timed out.
Temporary files successfully removed.
Backup data folder deleted in cleaning up after job failure.
Process Completed: 09/09/2017 16:17:19
Elapsed Time: 1 hour, 35 minutes and 53 seconds
This has left me with the following files on host1 (VMware ESXi, 6.0.0, 3620759):
drwxr-xr-x 1 root root 2660 Sep 12 04:20 .
drwxr-xr-t 1 root root 3780 Apr 19 09:47 ..
-rw------- 1 root root 31391404032 Sep 12 14:37 guest1-000001-delta.vmdk
-rw------- 1 root root 329 Sep 9 13:41 guest1-000001.vmdk
-rw------- 1 root root 12884901888 Apr 19 19:57 guest1-3d58c3bf.vswp
-rw------- 1 root root 20346 Sep 9 13:41 guest1-Snapshot19.vmsn
-rw-r--r-- 1 root root 131 Sep 9 13:41 guest1-aux.xml
-rw------- 1 root root 644245094400 Sep 9 13:41 guest1-flat.vmdk
-rwx------ 1 root root 398 Sep 9 13:41 guest1-vss_manifests19.zip
-rw------- 1 root root 503 Sep 2 19:25 guest1.vmdk
-rw-r--r-- 1 root root 583 Sep 9 13:41 guest1.vmsd
-rw-r--r-- 1 root root 3324 Sep 9 13:41 guest1.vmx
-rw------- 1 root root 0 Apr 19 19:57 guest1.vmx.lck
-rw-r--r-- 1 root root 3317 Sep 9 13:41 guest1.vmx~
-rw------- 1 root root 8684 Sep 9 13:42 old-guest1.nvram
-rw------- 1 root root 3855 Sep 12 04:20 old-guest1.vmxf
-rw-r--r-- 1 root root 1252233 Jun 24 14:06 vmware-1.log
-rw-r--r-- 1 root root 725229 Sep 12 12:10 vmware.log
-rw------- 1 root root 1047527424 Apr 19 19:57 vmx-guest1-1029227455-1.vswp
Host1 appears to be running fine but this is clearly a messy situation.
I'm concert to touch it before resolving this.
I don't know what happens if I:
- restart the guest or the host
- take a snapshot
- rerun the backup
Even though the vmdk file is pretty big disk space should not be an issue:
host1 - 3.2TB free on datastore
guest1 OS - 85GB free
backup destination media D: - 3.4TB free
But I've noticed the following in vSphere:
Provisioned Storage: 1.18TB
Not-shared Storage: 642.21GB
Used Storage: 642.21GB
Is it possible that it runs out of space during snapshotting / imagining?
I would like to know what the safest recovery procedure from this situation is.
And what the root cause of me ended up here was.
Thanks
Adam