Wednesday 10 April 2019

Hyper-V virtual machines in starting state for a long time (Veeam issue)

The Issue

At work we have a two Hyper-V host set up with one live Hyper-V server and one replication server. We run 5 virtual machines on Hyper-V and use Veeam to backup and replicate all the virtual machines.

The last few times that I restarted the main Hyper-V server for windows updates I noticed that the virtual machines took a long time to start up.

The host server restarted as normal but all the virtual machines showed as 'starting...' in Hyper-V Manager with no sign of any of the virtual machines starting up.

After around an hour and a half of waiting all the virtual machines suddenly started up as nothing had happened. When I checked all the virtual machines they were working normally.

I restarted one of the virtual machines to see if the slow start up happened again. This time there was a slight delay of around thirty seconds but nothing as extreme as the first boot. So it looked like it was only an issue when the host server rebooted.

I looked through the event viewer but couldn't see anything regarding the slow boot.

The Fix

After a few hours of search on the internet I finally found an article that looked like it matched my problem. It was from a Veeam forum thread. I contacted my IT support company to get there opinion on the article and they confirmed that I had the same issue.

I have to schedule in an update for Veeam Backup and Replication to Version 9 Update 4

The Problem

The problem of the slow boot was caused by Veeam Backup and Replication 9 not cleaning up replication points correctly in virtual machine configuration files. Hyper-V configuation files with the extension .vmxc are used by Veeam, to temporarily store replication point information. The information should be deleted soon after the relication has completed but for some reason this clean up doesn't happen correct in this version of Veeam Backup and Replication. The .vmcx files should be quite small but in our example they were around 50 mb each. We ran the script to check how many replication points were stored in the files and found it to be an average of 14000 replication points!

We haven't updated Veeam yet but hopefully this should fix the issue. I'll update this article to confirm if the fix works or not.