ESXTOP Batch Mode Analysis with Windows Perfmon

I needed to grab some stats from my ESX hosts for off-line analysis so I fired up my trusty ESXTOP intent on using batch mode to capture a .csv formatted output.  I started to manually select the counters I was interested in while working in ESXTOP interactive mode (you can save your selected counters to the esxtop configuration file with the ‘w’ command) and thought that there must be a better way.  I found that better way in the VMware Performance Community: http://communities.vmware.com/docs/DOC-3930.  There is now a -a switch that can be used to include ALL performance counters.  I’m sold.

I wanted detailed information, so I decided on a 15 second capture interval to run for a 2 hour window.  Here’s the command I used:

esxtop -a -b -d 15 -n 480 > /tmp/esxtopout.csv

where -a is for ALL, -b is for batch mode, -d is for delay, and -n is for the number of iterations ((60/15)*60*2).  I wrote out the results to a .csv in /tmp.  The resulting CSV weighed in at a whopping 100MB after 2 hours.

The CSV can be analyzed in Excel (pivot tables work well for this) or in Windows Perfmon.  I opened the log in Perfmon as I was after basic Min/Average/Max counters and Perfmon makes those easy to see.  When adding the CSV log to Perfmon, you are prompted to select counters.  I added all instances of Commands/sec, Reads/sec, and Writes/sec from Physical Disk (I was gathering some IOPS counts for a new storage proposal). I got a bit more than I bargained for: a mostly unresponsive Perfmon window and the ugliest darn graph I’ve ever seen.

image

Switching from a graph view to the report view allows you to easily view and remove specific counters that you are not interested in, or open the Properties of the data set, switch to the data tab and bulk select counters that you want to remove.  I was not interested in vmhba1:x, specific VM’s or worlds, so I killed all of those, leaving just the base iSCSI device (vmhba32 in my case).

After some cleanup the graph looked a bit better and more importantly, I was able to easily read my Min/Average/Max stats:

image

Here are the takeaways –

  • ESXTOP is a powerful utility for performance monitoring
  • All stats (-a) can result in a huge file – use it wisely in batch mode; else use interactive mode to select your counters and write them to the user-defined configuration file.  Invoke the config file with the -c option when running in batch mode.
  • Consider using vscsiStats for more granular reporting of storage performance and storage workload characterization.
  • ESXTOP physical disk stats do not include NFS volumes.

Do you use other tools or methods to collect basic disk IO counters for storage sizing purposes?  If so, leave a comment describing your approach!

Comments

  1. when i import the esxtop with an interval of 15 seconds in perfmon.
    are the values i see in perfmon the real values.
    for example if i see a physical path read latency of 5 then its 5 or do i have to do something with that interval of 15 ?
    or did perfmon this already for me ?

  2. Hy
    I have executed the following command: “esxtop -b -d 2 -n 100 -c .esxtopstoragecounters > /tmp/esxtopstoragecounters.csv”
    How can I now get the .csv file from the Host to analyze with Permon?
    We use ESXi 5.0.0 installed on an SD-Card.
    Thanks for your help.

  3. Hy
    I have executed the following command: “esxtop -b -d 2 -n 100 -c esxtopstoragecounters > /tmp/esxtopstoragecounters.csv”
    How can I now get the .csv file from the Host to analyze with Permon?
    We use ESXi 5.0.0 installed on an SD-Card.
    Thanks for your help.

    • Hello Ivo,
      You can use WinScp tool to get the file from ESX to your local Computer and then can use perfmon to Analyze the file.

  4. Mike Schreiner says:

    okay i’m gonna try this today at 12:45 PM!!!

  5. William – thanks for the link. The article you referenced on interpreting ESXTOP (http://communities.vmware.com/docs/DOC-9279) is a great one. As you suggested, building out a counter list is a great way to automate reviewing ESXTOP stats in Perfmon. I’d also love to build out a PAL (http://www.codeplex.com/PAL) template to interpret and report on ESXTOP stats when I find some time….

Trackbacks

  1. […] Read Joshua Townsend’s lessons here. VMware ESXTOP Learning Lessons: VMToday Categories: VMware, Virtualization Tags: esxtop, perfmon, performance, Virtualization, vmtoday, VMware Comments (0) Trackbacks (0) Leave a comment Trackback […]

  2. […] and looking for CMDS/s in the output.  I published a couple articles on using esxtop here and here.  The numbers from PerfMon and esxtop get you pretty close but can be skewed by a few things […]

  3. […] necessary.  If you want to capture ESXTOP realtime CPU ready and then analyze it later, use ESXTOP Batch Mode then analyze in Excel or Windows Perfmon.  ESXTOP counters are explained here: Interpreting […]

  4. […] The CPU Ready summation in milliseconds counter in the vCenter Client is not always the most accurate or easy to interpret stat – to better quantify the problem it might be best to go to the ESX command line and run ESXTOP.  CPU Ready over 5% could be a sign of trouble, over 10% and there is a problem.  Running ESXTOP in batch mode and then analyzing the output using Windows Perfmon or Excel might be a good way to go on this to get a view over several hours rather than the realtime stats we were looking at.  I wrote a post a while back with more info on ESXTOP batch mode: http://vmtoday.com/2009/09/esxtop-batch-mode-windows-perfmon/ […]

  5. […] ESXTOP Batch Mode Analysis with Windows Perfmon […]

  6. […] This article (High CPU Ready, Poor Performance here) goes through a similar scenario and provides some great insight on what to look for and even some other ways to capture and analyse the CPU Ready % (see ESXTOP Batch Mode Analysis with Window Performance here) […]

Drop a comment below:

%d bloggers like this: