I needed to grab some stats from my ESX hosts for off-line analysis so I fired up my trusty ESXTOP intent on using batch mode to capture a .csv formatted output. I started to manually select the counters I was interested in while working in ESXTOP interactive mode (you can save your selected counters to the esxtop configuration file with the ‘w’ command) and thought that there must be a better way. I found that better way in the VMware Performance Community: https://communities.vmware.com/docs/DOC-3930. There is now a -a switch that can be used to include ALL performance counters. I’m sold.
I wanted detailed information, so I decided on a 15 second capture interval to run for a 2 hour window. Here’s the command I used:
esxtop -a -b -d 15 -n 480 > /tmp/esxtopout.csv
where -a is for ALL, -b is for batch mode, -d is for delay, and -n is for the number of iterations ((60/15)*60*2). I wrote out the results to a .csv in /tmp. The resulting CSV weighed in at a whopping 100MB after 2 hours.
The CSV can be analyzed in Excel (pivot tables work well for this) or in Windows Perfmon. I opened the log in Perfmon as I was after basic Min/Average/Max counters and Perfmon makes those easy to see. When adding the CSV log to Perfmon, you are prompted to select counters. I added all instances of Commands/sec, Reads/sec, and Writes/sec from Physical Disk (I was gathering some IOPS counts for a new storage proposal). I got a bit more than I bargained for: a mostly unresponsive Perfmon window and the ugliest darn graph I’ve ever seen.
Switching from a graph view to the report view allows you to easily view and remove specific counters that you are not interested in, or open the Properties of the data set, switch to the data tab and bulk select counters that you want to remove. I was not interested in vmhba1:x, specific VM’s or worlds, so I killed all of those, leaving just the base iSCSI device (vmhba32 in my case).
After some cleanup the graph looked a bit better and more importantly, I was able to easily read my Min/Average/Max stats:
Here are the takeaways –
- ESXTOP is a powerful utility for performance monitoring
- All stats (-a) can result in a huge file – use it wisely in batch mode; else use interactive mode to select your counters and write them to the user-defined configuration file. Invoke the config file with the -c option when running in batch mode.
- Consider using vscsiStats for more granular reporting of storage performance and storage workload characterization.
- ESXTOP physical disk stats do not include NFS volumes.
Do you use other tools or methods to collect basic disk IO counters for storage sizing purposes? If so, leave a comment describing your approach!
