It seemed like a good idea, and it still seems like a good idea when it works, but when my combination of Data Protection Manager, Hyper-V, and iSCSI gets grumpy, it gives me ulcers.
My recent issue started, as it usually does with this setup, with a power outage. The power went out, and when it came back on all the servers came back up properly. “No Problem”, I thought, however a little bit later I noticed that DPM was having issues backing up the virtual servers on one of the virtual hosts. I tried to kick the backup jobs off manually, but they failed with a VSS error of event ID 12305, “Volume/disk not connected or not found” and that the VSS provider was in a “bad state”. I tried some of the easy things that I found on the ‘net but to no avail. I figured I’d reboot the box and that it would “figure itself out “when it came back up (which is often the case when my DPM backups delightfully start bombing out on a virtual host). This time however, the server came up and my iSCSI virtual machines were gone, unlisted in the Hyper-V manager. Despite the fact that Hyper-V has delisted virtual machines on me several times in the past as well, it never ceases to cause me to question my career choice when it happens.
Investigating, I saw that the iSCSI drive along with the virtual disks were there so my hope was that the VHDs were okay. When Hyper-V was trying to add the virtual machines it was kicking out an error along the lines of an OID that couldn’t be found. Distressed, I decided to just recreate the virtual machines from the VHDs (not a first for me either), however I made the fortunate mistake of letting Hyper-V store the virtual machines in the default directory on the system partition. The first two machines started up fine, but the third kicked out an error that it couldn’t write the memory file. I had forgotten that Hyper-V keeps a swap file of the memory for the virtual machine and I had run out of space for such files on the system partition. I figured that I’d have to recreate the virtual machines, again, but in their original directories on the iSCSI drive. Before I went through that work again though, I figured I would take a shot at modifying the XML config for the servers that I had just created.
It was then in the Hyper-V ProgramData folder that noticed something peculiar – the links to the original non-operational virtual machines weren’t working. When I pulled the directory listing I found them pointing at ‘F:’, but there was a different drive on ‘F:’ and the virtual machine iSCSI drive had been moved to ‘H:’. It turns out that after the power outage the server had decided to snag an unassigned iSCSI drive that was on an attached Netgear ReadyNas box and assign it to ‘F:’. The Virtual Host had worked fine through the week because the drive mapping didn’t take affect until after I rebooted the server. After reassigning the drives my Hyper-V and DPM are happy again, but I’m sure they’ll get back at me eventually.