Since joining VMware, I’ve built, rebuilt and updated several lab environments (home lab and work labs). One of the problems I keep running into is incorrect/missing/disabled NTP on my ESXi hosts. Because lab gear is often
abused well-loved, BIOS time is not always correct (long periods of being powered off), settings are all sorts of jacked up (it’s a lab, we’re engineers, let’s break stuff!!!), and documentation is non-existent. The problem is not just in a lab environment – the number of customer environments I’ve seen that aren’t set to use NTP is much higher than I would have thought (I wrote about a case here where incorrect time on hosts caused some of the problems with a VMware Horizon View environment). Getting ESXi host time set correctly is one of the first steps in building out a new vSphere environment as authentication fails, VM guests get incorrect time, HA configuration can fail, log files get incorrect time stamps making security analysis or troubleshooting difficult, etc. This TechTarget tip (http://searchvmware.techtarget.com/tip/Network-time-synchronization-for-VMware-ESXi-Timing-is-everything) does a good job explaining the basics of ESXi time synchronization, including how to manually set time and NTP on ESXi hosts. Manually setting time, NTP, and DNS forwarding servers (if you’re using hostnames for NTP instead of IP addresses) on a bunch of hosts is a waste of time – let’s automate!
Before we get too carried away, let’s first make sure our vCenter server has the correct time. If you installed your vCenter Server on a Windows host, it should get it’s time from your Active Directory. For a vCenter Server Virtual Appliance (VCVA), you need to set the NTP settings manually. To do this, log into the admin interface of the VCVA (https://<VCVA-name-or-IP>/5480/). On the vCenter Server tab, click either the ‘Time’ menu button or the ‘Configure Time’ link.
You have several options for time synchronization on the VMware vCenter Server Appliance:
- No synchronization – just don’t…
- NTP synchronization – sync with an upstream NTP server. Use an authoritative source on your own network if available, otherwise use something like ntp.org’s servers.
- VMware Tools synchronization – Use the pre-installed VMware tools package to read time from the ESXi host that the VCVA is running on. If the host’s time is incorrect or changes, the VCVA time will change too. If you vMotion the VCVA from one host to another and the hosts time is not in sync, the VCVA will experience a time shift.
- Active Directory synchronization – If you joined your VCVA to your Microsoft Active Directory, VCVA will read the time from your domain’s authoritative time source (usually the domain controller holding the PDC Emulator FSMO role).
I prefer Active Directory if one is available and correctly configured to pull time from a trusted higher stratum server. See this TechNet article for tips on how to configure an Active Directory forest for reliable time synchronization. Next I would choose the NTP synchronization option, followed by VMware tools sync, and never the no time synchronization.
Once the VCVA has accurate time, we can focus on the ESXi hosts. As I mentioned above, this can be done manually in either the vSphere Client (the legacy C# client) or the vSphere Web Client. Here’s the screenshot for where to click in the vSphere Web Client for a single host.
That’s too many clicks for me, so I put together a quick script in PowerCLI that will do several things for me:
- Clear existing NTP servers from ESXi hosts.
- Manually set the time on all ESXi hosts to match the local system time (the system you are running the script on). This is very helpful if you have servers with date-time values that are way out of whack. The ESXi NTP client will not correct time on the host when the offset (i.e. difference) between the host time and the NTP server time is greater than the preset sanity limit of 1000 seconds. Per VMware KB 1005092, If you have a greater than 1000 second offset you may find entries like the following in /var/log/messages or /var/log/hostd.log (better yet, see them in VMware Log Insight!!!)
ntpd: time correction of seconds exceeds sanity limit (1000); set clock manually to the correct UTC time. [info 'ha-eventmgr'] Event 91 : NTP daemon stopped. Time correction 1206 > 1000 seconds. Manually set the time and restart ntpd.
- Configure DNS servers on the management network (useful if you find your NTP servers by name instead of IP.
- Configure NTP servers on ESXi hosts, configure the NTP daemon to start automatically with the host, open the ESXi firewall for outbound NTP requests, and restart the NTP service to be sure everything is working correctly.
I have a simple menu driving the script to make changes easy without having to remember to pass variables and whatnot. You’ll see the first menu option for ‘Delete all existing DNS Servers values’ is not working. PowerCLI can’t write null values with the Set-VMHostNetwork -DNSAddress command, and the UpdateDnsConfig method in VMware.Vim.HostDnsConfig was not behaving for me (probably because my PowerCLIfu is weak). Adding new DNS servers will overwrite any existing values, so the option is kinda silly to have I guess….
Feel free to modify or suggest improvements. The code is on github (https://github.com/joshuatownsend/set-vmware-ntp-dns), and displayed below for you to use:
If you found this helpful, consider voting for my site in the vSphere-Land Top vBlog 2014 poll here: http://www.surveygizmo.com/s3/1553027/Top-VMware-virtualization-blogs-2014