Since joining VMware, I’ve built, rebuilt and updated several lab environments (home lab and work labs). One of the problems I keep running into is incorrect/missing/disabled NTP on my ESXi hosts. Because lab gear is often abused well-loved, BIOS time is not always correct (long periods of being powered off), settings are all sorts of jacked up (it’s a lab, we’re engineers, let’s break stuff!!!), and documentation is non-existent. The problem is not just in a lab environment – the number of customer environments I’ve seen that aren’t set to use NTP is much higher than I would have thought (I wrote about a case here where incorrect time on hosts caused some of the problems with a VMware Horizon View environment). Getting ESXi host time set correctly is one of the first steps in building out a new vSphere environment as authentication fails, VM guests get incorrect time, HA configuration can fail, log files get incorrect time stamps making security analysis or troubleshooting difficult, etc. This TechTarget tip (https://searchvmware.techtarget.com/tip/Network-time-synchronization-for-VMware-ESXi-Timing-is-everything) does a good job explaining the basics of ESXi time synchronization, including how to manually set time and NTP on ESXi hosts. Manually setting time, NTP, and DNS forwarding servers (if you’re using hostnames for NTP instead of IP addresses) on a bunch of hosts is a waste of time – let’s automate!
Before we get too carried away, let’s first make sure our vCenter server has the correct time. If you installed your vCenter Server on a Windows host, it should get it’s time from your Active Directory. For a vCenter Server Virtual Appliance (VCVA), you need to set the NTP settings manually. To do this, log into the admin interface of the VCVA (https://<VCVA-name-or-IP>/5480/). On the vCenter Server tab, click either the ‘Time’ menu button or the ‘Configure Time’ link.
You have several options for time synchronization on the VMware vCenter Server Appliance:
- No synchronization – just don’t…
- NTP synchronization – sync with an upstream NTP server. Use an authoritative source on your own network if available, otherwise use something like ntp.org’s servers.
- VMware Tools synchronization – Use the pre-installed VMware tools package to read time from the ESXi host that the VCVA is running on. If the host’s time is incorrect or changes, the VCVA time will change too. If you vMotion the VCVA from one host to another and the hosts time is not in sync, the VCVA will experience a time shift.
- Active Directory synchronization – If you joined your VCVA to your Microsoft Active Directory, VCVA will read the time from your domain’s authoritative time source (usually the domain controller holding the PDC Emulator FSMO role).
I prefer Active Directory if one is available and correctly configured to pull time from a trusted higher stratum server. See this TechNet article for tips on how to configure an Active Directory forest for reliable time synchronization. Next I would choose the NTP synchronization option, followed by VMware tools sync, and never the no time synchronization.
Once the VCVA has accurate time, we can focus on the ESXi hosts. As I mentioned above, this can be done manually in either the vSphere Client (the legacy C# client) or the vSphere Web Client. Here’s the screenshot for where to click in the vSphere Web Client for a single host.
That’s too many clicks for me, so I put together a quick script in PowerCLI that will do several things for me:
- Clear existing NTP servers from ESXi hosts.
- Manually set the time on all ESXi hosts to match the local system time (the system you are running the script on). This is very helpful if you have servers with date-time values that are way out of whack. The ESXi NTP client will not correct time on the host when the offset (i.e. difference) between the host time and the NTP server time is greater than the preset sanity limit of 1000 seconds. Per VMware KB 1005092, If you have a greater than 1000 second offset you may find entries like the following in /var/log/messages or /var/log/hostd.log (better yet, see them in VMware Log Insight!!!)
ntpd[263140]: time correction of seconds exceeds sanity limit (1000); set clock manually to the correct UTC time. [info 'ha-eventmgr'] Event 91 : NTP daemon stopped. Time correction 1206 > 1000 seconds. Manually set the time and restart ntpd.
- Configure DNS servers on the management network (useful if you find your NTP servers by name instead of IP.
- Configure NTP servers on ESXi hosts, configure the NTP daemon to start automatically with the host, open the ESXi firewall for outbound NTP requests, and restart the NTP service to be sure everything is working correctly.
I have a simple menu driving the script to make changes easy without having to remember to pass variables and whatnot. You’ll see the first menu option for ‘Delete all existing DNS Servers values’ is not working. PowerCLI can’t write null values with the Set-VMHostNetwork -DNSAddress command, and the UpdateDnsConfig method in VMware.Vim.HostDnsConfig was not behaving for me (probably because my PowerCLIfu is weak). Adding new DNS servers will overwrite any existing values, so the option is kinda silly to have I guess….
Feel free to modify or suggest improvements. The code is on github (https://github.com/joshuatownsend/set-vmware-ntp-dns), and displayed below for you to use:
<###################################################################### Menu-driven PowerCLI script to delete, add, and update DNS and NTP on VMware ESXi hosts. **Save as set-ntp.ps1 **Switch to directory ps1 is saved in and run ** MAKE SURE LOCAL SYSTEM TIME IS CORRECT!!!! v1.0 27 February 2014: Initial draft v1.1 27 February 2014: Changed menu option 1 to dark gray cause no worky. Written By Josh Townsend All Code provided as is and used at your own risk. ######################################################################> $xAppName = ‘set-ntpdns’ [BOOLEAN]$global:xExitSession=$false # Add PowerCLI snapin if not already loaded if ((Get-PSSnapin -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue) -eq $null ) {Add-PsSnapin VMware.VimAutomation.Core; $bSnapinAdded = $true} # Prompt for vCenter or ESXi server $vCenter = Read-host "Enter the name of your vCenter Server or ESXi host" $domain = Read-host "Domain name (leave blank if using local ESXi account)" $user = Read-host "vCenter or ESXi user account" $password = Read-host -Prompt "Enter password" -AsSecureString $decodedpassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($password)) Connect-VIServer $vCenter -user $domain\$user -Password $decodedpassword #Some other connection options: #Option 1 - Show menu of recently used ESXi or vCenter Servers #Write-host "Choose from the list below or enter the name of your vCenter Server or ESXi Host, then enter credentials as prompted" -ForegroundColor Yellow #Connect-VIServer -Menu $true #option 2 - Hard code it. This leaves your password in plain text. Consider using the new-VICredentialStore option to securely store your credentials! #Connect-VIServer -Server 10.10.10.10 -User root -Password vmware $esxHosts = get-VMHost Function Clear-DNS{ #foreach ($esx in $esxHosts) #{ #Write-Host "Deleting existing DNS servers from $esx" -ForegroundColor Green # Get Current Values for required properties #$currenthostname = Get-VMHost | Get-VMHostNetwork | select hostname #$currentdomainName = Get-VMHost | Get-VMHostNetwork | select domainname # ------- UpdateDnsConfig ------- #$config = New-Object VMware.Vim.HostDnsConfig #$config.dhcp = $false #$config.hostName = $currenthostname #$config.domainName = $domainname #$_this = Get-View -Id 'HostNetworkSystem-networkSystem' #$_this.UpdateDnsConfig($config) #} #Write-Host "Old DNS Values Cleared! Press any key to return to menu..." -ForegroundColor Green Write-Host "Sorry, this function not supported because we cannot delete or write null values for DNS Server Addresses using PowerCLI. `nUse the Add Additional menu option to overwrite existing DNS servers. `nPress any key to return to the menu." -ForegroundColor Magenta $x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown") LoadMenuSystem } Function Clear-NTP{ foreach ($esx in $esxHosts) { Write-Host "Deleting existing NTP servers from $esx" -ForegroundColor Green $existingNTParray = $esxhosts | Get-VMHostNTPServer Remove-VMHostNTPServer -NtpServer $existingNTParray -Confirm:$false } Write-Host "Old NTP Values Cleared! Press any key to return to menu..." -ForegroundColor Green $x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown") LoadMenuSystem } Function Clear-NTPDNS{ Clear-NTP Clear-DNS Write-Host "Done! Press any key to return to menu...." -ForegroundColor Green $x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown") LoadMenuSystem } Function Manual-Time{ # First set all hosts in vCenter to use time of local system to prevent too large drift for NTP to correct Write-Host "Updating manual time on $esx to match local system time" -ForegroundColor Green $esxhosts | Where-Object { $t = Get-Date $dst = $_ | %{ Get-View $_.ExtensionData.ConfigManager.DateTimeSystem } $dst.UpdateDateTime((Get-Date($t.ToUniversalTime()) -format u)) } Write-Host "Done! Press any key to return to menu...." -ForegroundColor Green $x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown") LoadMenuSystem } Function Set-DNS{ # Prompt for Primary and Alternate DNS Servers $dnspri = read-host "Enter Primary DNS" $dnsalt = read-host "Enter Alternate DNS" # Prompt for Domain $domainname = read-host "Enter Domain Name" foreach ($esx in $esxHosts) { Write-Host "Configuring DNS and Domain Name on $esx" -ForegroundColor Green Get-VMHostNetwork -VMHost $esx | Set-VMHostNetwork -DomainName $domainname -DNSAddress $dnspri , $dnsalt -Confirm:$false Write-Host "Done! Press any key to return to menu...." -ForegroundColor Green $x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown") LoadMenuSystem } } Function Set-NTP{ #Prompt for NTP Servers $ntpone = read-host "Enter NTP Server One" $ntptwo = read-host "Enter NTP Server Two" foreach ($esx in $esxHosts) { Write-Host "Configuring NTP Servers on $esx" -ForegroundColor Green Add-VMHostNTPServer -NtpServer $ntpone , $ntptwo -VMHost $esx -Confirm:$false Write-Host "Allow NTP queries outbound through the firewall on $esx" -ForegroundColor Green Get-VMHostFirewallException -VMHost $esx | where {$_.Name -eq "NTP client"} | Set-VMHostFirewallException -Enabled:$true Write-Host "Starting NTP service on $esx" -ForegroundColor Green Get-VmHostService -VMHost $esx | Where-Object {$_.key -eq "ntpd"} | Start-VMHostService Write-Host "Configuring NTP Client Policy on $esx" -ForegroundColor Green Get-VMHostService -VMHost $esx | where{$_.Key -eq "ntpd"} | Set-VMHostService -policy "on" -Confirm:$false Write-Host "Restarting NTP Client on $esx" -ForegroundColor Green Get-VMHostService -VMHost $esx | where{$_.Key -eq "ntpd"} | Restart-VMHostService -Confirm:$false } Write-Host "Done! Press any key to return to menu...." -ForegroundColor Green $x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown") LoadMenuSystem } Function Set-NTPDNS { Set-DNS Set-NTP Write-Host "Done! Press any key to return to menu...." -ForegroundColor Green $x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown") LoadMenuSystem } # Menu to select functions function LoadMenuSystem() { [int]$xMenuChoiceA = 0 [BOOLEAN]$xValidSelection=$false while ( $xMenuChoiceA -lt 1 -or $xMenuChoiceA -gt 8 ){ CLS # ------ Menu Choices ------- Write-Host “Choose an option below to modify all ESXi hosts DNS and NTP settings.`n” -ForegroundColor Magenta Write-host "`t1. Delete all existing DNS Servers values (not working)" -ForegroundColor DarkGray Write-host "`t2. Delete all existing NTP Server values" -ForegroundColor Cyan Write-host "`t3. Delete all existing DNS & NTP Servers (not working)" -ForegroundColor Cyan Write-host "`t4. Manually set time on all hosts to match your local system time.*" -ForegroundColor Cyan Write-host "`t*This option prevents NTP sync problems due to large offset" -ForegroundColor Gray Write-host "`t5. Add additional DNS Server values**" -ForegroundColor Cyan Write-host "`t**This will overwrite any existing values" -ForegroundColor Gray Write-host "`t6. Add additional NTP Server values and set NTP to start with host.***" -ForegroundColor Cyan Write-host "`t***Verifyr DNS values are correct if using names instead of IPs." -ForegroundColor Gray Write-host "`t7. Add both NTP & DNS Server values and set NTP to start wtih host" -ForegroundColor Cyan Write-host "`t8. Quit and exit`n`n" -ForegroundColor Yellow # ------ Menu Choices ------- # Get and Validate Choice [Int]$xMenuChoiceA = read-host "Please select an option [1-8]" if( $xMenuChoiceA -lt 1 -or $xMenuChoiceA -gt 8 ){ Write-Host “`tInvalid Selection.`n” -Fore Red;start-Sleep -Seconds 1 } # Survey Says.... Switch( $xMenuChoiceA ){#… User has selected a valid entry.. load menu 1{ Clear-DNS } 2{ Clear-NTP } 3{ Clear-NTPDNS } 4{ Manual-Time } 5{ Set-DNS } 6{ Set-NTP } 7{ set-NTPDNS } default { $global:xExitSession=$true;break } } } } LoadMenuSystem If ($xExitSession){ Exit-PSSession #… User quit & Exit } Else { .\set-ntp.ps1 #… Loop the function }
Enjoy!
If you found this helpful, consider voting for my site in the vSphere-Land Top vBlog 2014 poll here: https://www.surveygizmo.com/s3/1553027/Top-VMware-virtualization-blogs-2014
John says
Hi,
great post, seems very useful. but the script cannot be copied well: after paste the whole script will be in one line w/o enters.
could you please upload the script as a .ps1 file, or github link?
thanks
Josh Townsend says
Hi John,
Thanks for letting me know that copying the script from the syntax highlighter isn’t working well. I’ve added the link to the github repo above the code. I’ll see if I can find a better syntax highlighter for WordPress…
Herceg Andras says
thank you for the github link!