5

Server 2012 VMs don’t boot after a restart

2012mileage

 

It’s patch week again which means many guest reboots in our VMware environment. Lately, we’ve been receiving alerts on many of our Server 2012 VM’s, and upon investigating we’ve found these guests to be hung at the Windows boot screen. After a few hours they might come up, but more often than not a hard reset is required to bring them back.

It turns out this is a known issue and VMware has released KB2092807 to address it. Each processor has a TimeStampCounter (TSC) which is essentially an odometer that counts how many cycles the processor has run since its last reboot. When Windows reboots, it resets this counter if the value is equal to or larger than 18,014,398,509,481,984. That is to say, the VM has been running for about two months, or longer, without being powered off. The problem is that Windows only resets the counter on CPU0, and a discrepancy between multiple vCPU’s can cause the OS to hang at boot up. This issue only occurs on version 10 virtual hardware due to the hardware clock mechanism. Version 11 hardware is not impacted.

They do offer a fix, but this requires a slight change of each VM. You must add an advanced parameter that will reset the TSC of all vCPU’s on a soft reset. What they don’t tell you is if you want to use the GUI (Web or C# Client) to make the change, you’ll have to power down the VM first. Otherwise the option to edit advanced settings is greyed out.

 

advances_no_edit

 

  • If you have the time and flexibility to power down each VM, simply navigate to theadvancedparameters of your VM’s settings…
    • Web Client: Settings -> VM Options -> Advanced -> Edit Configuration
    • C# Client: Edit Settings -> Options -> General -> Configuration Parameters
  • Add a new row with the following
    • Name: monitor_control.enable_softResetClearTSC
    • Value: True

advanced_parameters

 

If you’re like the rest of us, you can’t afford to bring down each VM and edit it manually. Thankfully, VMware has offered a great PowerCLI script to get the job done. The script adds the advanced option to all affected VM’s. Don’t worry about the line where it searches for guestid like windows8*Guest. Server 2012 VM’s show up as windows8Server64Guest. The script then performs a stationary migration (the VM doesn’t change hosts) which applies the new configuration.

Note: VMware removed line 13 from their KB because they discovered that self-migration didn’t always perform the Fast Suspense Resume (FSR) required to activate the changes. Instead they recommend vMotioning your VM’s. 


ForEach ($vm in (Get-VM)){
$vmv = Get-VM $vm | Get-View
$name = $vmv.Name
$guestid = $vmv.Summary.Config.GuestId
$state = $vmv.Summary.Runtime.PowerState
$vmx = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmx.extraConfig += New-Object VMware.Vim.OptionValue
$vmx.extraConfig[0].key = "monitor_control.enable_softResetClearTSC"
$vmx.extraConfig[0].value = "TRUE"
if ($guestid -like "windows8*Guest") {
($vmv).ReconfigVM_Task($vmx)
if ($state -eq "poweredOn") {
# $vmv.MigrateVM_Task($null, $_.Runtime.Host, 'highPriority', $null)
}
}
}
Testing the script, my VM updated successfully while powered on.
powercli_reconfig_advanced
reconfigure_vm
After the configuration change, the TSC of each vCPU will reset to 0 upon a soft reboot and Windows will boot without issue.

Matt Bradford

5 Comments

  1. I updated all my VMs to hardware version 11 instead of using your script. After a couple of minutes alls VMs finished successfully and the problem is fixed.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.