VMtoday

I recently posted an article on how specific actions during the upgrade of a VMware Virtual Machine’s hardware from v4 to v7 can cause problems with certain services, including DNS, DHCP, and WINS. In that case, the problem was related to Microsoft Windows leaving non-present devices with networking configurations and  the failure of the VMware Upgrade Helper service to copy WINS settings when updating the NIC.  As my fellow blogger and VMUG leader, Jason Boche, responded on Twitter: “Same gotchas, different version.”  And right he is – anyone with experience in P2V or V2V, or who has been working with VMware long enough to have done a 2.5 to 3.0 upgrade experienced the same gotchas.

There are other issues with VMware virtual hardware upgrades, however, that you may not have experienced.  One such issue that I have experienced is highlighted in VMware Knowledge Base article 1013109: “Upgrading virtual hardware in ESX 4 may cause Windows 2008 disks to go offline“.  The problems described in the article are unique to Windows 2008 Enterprise and Datacenter editions only.  The problem is pretty well described in the title of the article – Upgrading virtual hardware in ESX 4 may cause Windows 2008 disks to go offline.  In this case, like with the ghost NIC’s I described last week, is more of a Microsoft issue, but it will rear its head when a VMware Administrator least desires it.  With this particular problem, the Windows Virtual Disk Service (part of the native Storage Management suite) is set to not auto-mount newly discovered disks that do reside on a shared bus.  Microsoft has a MSDN article on the VDS SANS policy here.  Upgrading the virtual hardware version causes the disks to be re-discovered and not auto-mounted.  This can potentially impact all non-system disks on a VM.

You may also experience similar issues when upgrading the vSCSI adapter in a VM from a standard LSI Logic Parallel SCSI adapter to a (new in vSphere 4.0) paravirtualized SCSI (pvSCSI) adapter, move virtual disks to new vSCSI adapters to increase the number of concurrent disk IO operations, or when you change the SCSI node ID of a virtual disk.  These may all trigger a re-discovery of the disks by the Windows Virtual Disk Service, leaving data disks offline on Windows 2008 Enterprise and Datacenter Edition guests.

In my opinion, these issues are not reasons to forgo upgrading your virtual hardware version.  However, when your upgrade/migration plans call for upgrading the virtual hardware version of your guests you should be prepared to resolve any issues caused by ‘ghost hardware’, offline disks, and the like.  Both the MSDN and VMware articles I cited above offer workarounds for the offline disk issue.  Here are the links again:

I recently completed a VMware VI 3.5 to vSphere upgrade in a small environment (5 hosts, 80 VM’s).  Being a small environment, the upgrade was planned for one big overnight blitz.  Unfortunately, the size of the environment did not afford a test environment to uncover potential issues before the upgrade.  The upgrade to vSphere itself went swimmingly (the vCenter server had been upgraded a couple weeks earlier).  However, some things in the environment started to go wonky once the upgrade was complete.  Specifically, name resolution (DNS), DHCP, WINS, Group Policy, and really anything Microsoft Active Directory related just did not work.

Let me explain a bit about the environment so you can better understand what the problem was and how it was corrected. The environment was an all Microsoft shop, except for VMware of course. The company follows a virtualize-first policy and is about 90% virtualized, including the Active Directory Domain Controllers. The DC’s are Windows 2008 and serve up DHCP, DNS, and WINS in addition to their Directory Services roles.

The problems really began after I upgraded the virtual hardware version from v4 to v7 (check out page 97 of the vSphere Upgrade Guide for the upgrade procedure).  When a Windows server is upgrade from VMware Hardware Version 4 to 7, the VMware Upgrade Helper Service handles the reconfiguration of network adapters on the upgraded virtual machine.  The VMware Upgrade Helper Service is installed with VMware Tools and is one of the reasons, along with getting drivers installed for the new hardware, for upgrading VMware Tools before upgrading the hardware version.  If you review the Event Viewer Application log on an upgraded machine you will see several entries from VMUpgradeHelper (Source) with several different Event ID’s (26, 280, 272, 108, & 105).  An examination of these events will show that the VMware Upgrade Helper service 1.) Backed up the network configuration at OS shutdown, 2.) Started Automatically with the OS, 3.) Checks the device ID for the network adapter, 4.) If the device ID has changed (as a result of a hardware upgrade), the backed up configuration is restored and Event ID 269 is logged.

This behavior should be transparent for most configurations, with the exception of a slightly longer boot time following the upgrade.  However, I did notice a few problems with the NIC settings being restored under certain conditions.  First, on servers with a statically configured IPv4 stack, IP addresses and DNS server addresses were restored, but the WINS server addresses were not restored.  I suspect this is an oversight in the VMware Upgrade Helper service, but is probably not a major issue for many servers/environments as WINS is infrequently used.  However, for a WINS server itself to lose its configuration to use itself as a WINS server, bad things happen.  There are several ways to correct this – scripts, DHCP Options, etc.  In the end, this wasn’t really a show stopper for me in this small environment.

The second, and bigger issue for me, was that after the virtual hardware was upgraded and the VMware Upgrade Helper Service did it’s job my Active Directory and related services were not available.  DNS was not functioning, DHCP was not handing out addresses, and I couldn’t connect to AD using ADUC, GPMC or LDAP.  It took me a few minutes to figure out what was going on.  This seems to be what happened: the virtual hardware upgrade caused a new virtual network adapter to be installed in the VM and all of the settings, including the MAC, address to be restored.  The HW v4 NIC was removed from the machine, but Windows held onto the device as a ‘ghost NIC’ in Device Manager.  The core AD services, including DNS and DHCP, were still attempting to bind to the ghost NIC.  This behavior persisted through service restarts and reboots of the guest.  It wasn’t until I examined the IP configuration on the new NIC and clicked Apply (instead of canceling out) that I was prompted with a message indicating that there was more than one network interface configured with the same IP address, queuing me into the solution.

The error message should be familiar to anyone who has performed a Physical-to-Virtual migration (P2V) and is easily corrected by removing the old device through Windows Device Manager.  The device is hidden so you first have to expose it before deleting it.  Check http://support.microsoft.com/kb/315539 for details or simply follow my instructions below.  To expose the non-present NIC, open a command prompt and enter:

set devmgr_show_nonpresent_devices=1

You can then open Device Manager (enter devmgmt.msc at the command prompt to save some time).  In Device Manager, click View | Show Hidden Devices.  Expand Network Adapters and find the grayed-out entry for the old NIC as pictured below.

Select the ghost NIC and right-click | Uninstall to remove it.

The final gotcha for me on this is that the set devmgr_show_nonpresent_devices=1 command does not work on Windows 2008 (or Vista, Windows 7, or Windows 2008 R2).  To see and remove ghost NICs from Windows 2008, and environmental variable must be defined.  To set the variable, open Server Manager from the Windows Start Menu.  Highlight ‘Server Manager (%SERVERNAME%)’ in the left-side tree-view pane.  Click ‘Change System Properties’ in the right-hand pane.  Switch to the Advanced tab and click ‘Environment Variables.  Create a new System variable by clicking the New button.  The Variable name should be ‘devmgr_show_nonpresent_devices’ and the value should be ‘1′ as pictured below.

Click OK to close out of any open Windows.  A reboot is not necessary for the variable to take effect, although you may have to close out of all open Device Manager Windows and then reopen devmgmt.msc.  Click View | Show Hidden Devices and remove the ghost NIC as described above.  A quick reboot after I removed the ghost NIC from the domain controllers and all Active Directory, DNS, DHCP, and WINS services immediately began operating normally.  This second issue is more of a Microsoft problem in my opinion, and has been around for some time.

Before you start getting all upset and the FUD starts flying (“this is Microsoft/VMware’s latest attempt to break VMware/Microsoft?”), it wasn’t really vSphere that broke Active Directory; It was me.  A little better planning and not rushing through the last wee hours of the upgrade Window could have saved some trouble.  If you are planning a similar upgrade, it would be best to upgrade your domain controllers/DNS servers one at a time and remediate the issues I have decribed before upgrading the next.  This will ensure continued availability of your Active Directory and other critical services during your upgrade.

A reader named Mark contacted me today and asked if there was a way to reduce the size of the batch output from an ESXTOP run.  And he asks for good reason: Depending on the number of VM’s on your host, the delay between ESXTOP samplings and the number of samples you collect, using the All Stats option (-a) can yield a massive file in a short period of time.  If written to a partition on your ESX Service Console you run the risk of filling the partition, and forget about actually being able to analyze the data in PERFMON or Excel.  For example, on an ESX host running ~15 VM’s I produced 100MB worth of CSV using the -a switch, sampling every 15 seconds, for just under 2 hours.  ESXTOP uses 10-second intervals by default; I used -d 15 to change the sampling delay.  Had I went with the default my output would have been bigger.

To reduce the size of your output, you can change your sampling delay to something larger, say 30-seconds.  I suppose you could also capture statistics when the host is not busy so you get fewer characters in the results, but that’s just being goofy.

A better way to reduce your ESXTOP output size is to selectively include only the statistics you are interested in, and is really what Mark was asking.  After all, all statistics from ESXTOP can be too many statistics, and chances are you already know what stats you are interested in.  Here’s how you can narrow down the collected stats for easier analysis and smaller output:

1. Enter ESXTOP in interactive mode on the Service Console by simply typing esxtop at the # prompt
2. Switch to a component you are NOT interested in capturing statistics on by pressing the corresponding menu option (c: ESX cpu, m: ESX memory, d: ESX disk adapter, u: ESX disk device, v: ESX disk VM).
3. Press f when viewing the component you do not want to capture.  A list of fields will be displayed.  You can toggle the fields on and off by pressing the letter corresponding to each field.  An * indicates that the field is on.  You want to turn off all of the fields you don’t want to collect.
4. Repeat steps 2 & 3 for the remaining components, leaving only what you want to capture.
5. Switch to the component you want to capture in batch mode and repeat step #3, except you will now enable what you want to capture.
6. Press W (capital W – case sensitive) to write out the ESXTOP configuration file.  You can accept the default or create new configuration files.  You may want to create a CPU-only config file, memory-only, and so forth.
7. Press CTRL+C to stop ESXTOP.
8. Now, invoke ESXTOP in batch mode, calling your updated or new configuration file you created in step #6 using the -c switch.  Here’s an example:# esxtop -b -d 30 -n 480 -c .esxtopcpustats > /tmp/esxtop_cpu_stats.csv where .esxtopcpustats is an ESXTOP config file with only CPU stats.  -d sets your capture interval to 30 seconds, and -n sets the number of samples to 480 (or 4 hours with a delay of 30 seconds).

Once your capture is complete you can replay the sampling in ESXTOP using replay mode (-R), or you can copy the .csv to a Windows system and use PERFMON or Excel to analyze the stats.  If using PERFMON or Excel you will notice that the system summary information displayed at the top of an interactive ESXTOP session is included in the output (console memory, console cpu, etc.).  As far as I know, there is no way to disable this, nor would you want to as it includes the time stamp necessary to interpret your data.

It is possible to use the vSphere CLI or the vSphere Management Assistant (vMA) to run RESXTOP, a version of ESXTOP designed for remote administration of ESXi or ESX.  You may note, however, RESXTOP from the vSphere CLI only works from a Linux client.  Using either of these tools will help you to automate ESXTOP statistics collection from multiple hosts using customized configuration files.

VMware vCenter collects performance statistics, tasks and events for historical performance analysis and auditing.  The collection level and retention of performance statistics can be controlled through the vCenter GUI (see Administration | vCenter Server Settings | Statistics).   The level of statistics collection and retention periods can have a dramatic impact on your vCenter Server’s performance if not carefully planned and monitored.  In particular, the vCenter database can grow quite large and the database server required to support the increase in statistics increases in size and performance characteristics (increased disk IO capacity, CPU, and memory).  Fortunately, VMware has provided a vCenter database sizing tool within the vCenter client (see picture).  This is all well and good for initial sizing, and my experience shows that vCenter’s sizing estimates are fairly accurate assuming the environment remains healthy.

I recently migrated an environment from vCenter 2.5 to 4.0 and in the process switched from a Windows 2003 32-bit vCenter host and a SQL 2005 server (remote to vCenter) to a Windows 2008 64-bit vCenter server with a SQL 2008 server (again, a remote SQL server).  I experienced a few issues during the migration and thought I had worked through them all (I’ll post on those at a later date).  However, after a bit of time I found that performance statistics for objects in the vCenter were missing of not rendering at an acceptable pace.  Upon further investigation, I discovered warnings in the vCenter Service Status node indicating that performance rollups within the vCenter database were not taking place.

In a SQL-backed vCenter, statistics rollups are handled by the SQL Server Agent (note: if you are using SQL Server Express, statistics rollups are handled by vCenter itself as SQL Express does not offer SQL Server Agent jobs).  KB 1003570 describes this process (it applies to vCenter 2.5, but the principles in it can be applied to 4.0).  To troubleshoot and resolve the issue I opened SQL Server Management Studio and checked several items:

1. Is the SQL Server Agent running?
2. Are there statistics rollup jobs defined for SQL server agent?
3. Are those jobs running?

In my case, the SQL Server Agent was running (you are prompted to configure this during the vCenter install).  However, when I checked for the presence of rollup jobs, I discovered that only a Past Day job had migrated with the database to the new SQL server.  Upon investigating the job history for that job I discovered that the job had not run since the migration (note to self: add these checks to your standard vCenter migration checklist).

To remediate the problem I completed the following steps:

1. Remove the bad ‘Past Day stats rollupVirtualCenter’ job from the list of SQL Server Agent Jobs.
2. Recreate the three standard stats rollup jobs.  To recreate the jobs, find SQL scripts on your vCenter server in C:\Program Files (x86)\VMware\Infrastructure\VirtualCenter Server.  The .sql scripts you’ll need are stats_rollup1_proc_mssql.sql, stats_rollup2_proc_mssql.sql, and stats_rollup3_proc_mssql.sql.  Run these scripts in SQL Query Analyzer against your VirtualCenter Database in order from 1 to 3.  These scripts should create the rollup jobs and their associated stored procedures (this procedure is detailed at http://communities.vmware.com/thread/123715).
3. After recreating the jobs I took a backup of the vCenter database.  The Past Day job soon kicked off to begin a stats rollup (this runs every 30 minutes by default).

I checked the server several hours later and discovered that rather than completing successfully, the Past Day job was still running and the drive holding my vCenter database transaction log was full.  Back to the drawing board..

1. I disabled the Past Week and Past Month rollup jobs to avoid job conflicts.
2. I backed up the vCenter database and then performed a shrink of the log file to get it back down to size.
3. The vCenter was running as a VM, so I was able to quickly increase its disk size and use diskpart from within the guest to extend the partition.  The space required to process weeks of performance statistics is not included in the vCenter Database Sizing tool as it is assumed that the rollup/purge jobs will run as designed.

I wanted to see how bad the problem was before kicking off another job so I ran:

select count(*) from vpx_hist_stat1

against the vCenter database in SQL Query Analyzer.  The query ran for several hours (never a good sign) and eventually returned well over 20 million rows of performance statistics (thanks to http://communities.vmware.com/message/1318736 for pointing me in this direction).  I investigated options to truncate the tables (see above link), and also looked at a script from VMware KB 1000125: Purging old data from the database used by vCenter Server.  In the end, I decided to try to let the Past Day stats job run.

I stopped the vCenter  Server Service to prevent new statistics from being written to the database.  I also disabled the Past Week and Past Month SQL Agent jobs to prevent job conflicts and then manually started the Past Day job.  I had to stop the job several times as it filled the 100GB transaction log volume.  A backup & shrink operation gave me back the space on the log volume.  I saw about 300GB of transaction logs written over the course of this process, but the Past Day job eventually completed.

Finally, I re-enabled the Past Week and Past Month jobs and manually ran both of them (Past Week first, then Past Month), followed by a backup and shrink of the vCenter database.  I was impressed with the performance increase I saw in the vCenter client.  Lists and performance graphs rendered much faster than when stats rollups were not taking place.

It would be a good idea to include checking stats rollup job status and a count of rows from the vpx_hist_stat tables in the vCenter database in your regular maintenance tasks.  For other vCenter Database best practices, check out breakout session PO2061 from VMworld 2008.  If you did not attend or subscribe to VMworld, Scott Lowe covered the session in this post.  A VMworld 2009 “online only” session entitled VM3237 vCenter Databases: Setup, Management and Best Practices was also offered (subscription required).  I have not viewed this session so I cannot comment on its content.

Microsoft published a document named “Getting to Know Hyper-V: A Walkthrough from Initial Setup to Common Scenarios” last week.  According to Microsoft, “this guide provides detailed step-by-step walkthroughs for testing Hyper-V on a pre-production environment. You can use this guide to become familiar with Hyper-V and the process of creating and managing virtual machines. Also included in this guide are useful scenarios that you can test to better understand how Hyper-V can address the business goals of your organization.”  The document serves as a sort of evaluators guide for Hyper-V, stepping the reader through everything from enabling VT in BIOS through virtual networking.  It also includes some sections on using snapshots, base virtual machine templates, and managing Hyper-V based virtual machines remotely with Hyper-V Manager.  If you want more in-depth documentation on Hyper-V you can go through http://technet.microsoft.com.

As a side note, Microsoft has published the Microsoft Manual of Style for Technical Publications to help standardize technical documentation.  I have long been a fan of Microsoft’s technical documentation for its easy to read style, although it sometimes lacks the depth that I desire.

While we’re on the topic of virtualization documentation, I have also been quite pleased with VMware’s technical documentation over the years, and have found it to be continually increasing in quality, providing very specific technical guidance and references to additional resources.  I have also been pleased to see that VMware has improved delivery options for documentation.  VMware offers several formats for documentation delivery, including web-based and PDF’s.  Start with the Documentation Roadmap for a quick introduction to the available documentation, and where to find what you need.

You can find web-based vSphere documentation here: http://pubs.vmware.com/vsp40/.   The web-based documentation is great for running searches on.  All vSphere documentation can be accessed through this page: http://www.vmware.com/support/pubs/vs_pages/vsp_pubs_esx40_vc40.html.  If you want to do a full grab of all of VMware’s documentation for an in-house repository (e.g. SharePoint), check out xtravirt’s VMware Documentation Downloader script.

If you are looking for quick and easy evaluator guide-type documentation from VMware, check out these resources: ESXi Installable and vCenter Server Setup Guide and the Virtualization Kit (registration required) at http://www.vmware.com/resources/wp/virtualization101_register.html.

There is a ton of less formal VMware documentation in several places:

• Technical resources and case studies here: http://www.vmware.com/resources/techresources/
• Proven practices around Strategy, Applications, Security, Management, and Availability at VIOPS.
• Official VMware Blogs at http://www.vmware.com/vmtn/planet/vmware/.
• Community blogs aggregated by VMware at Planet v12n: http://www.vmware.com/vmtn/planet/v12n/
• VMworld Recorded Sessions & Labs (VMworld 2009 Sessions available as of today, September 14th) at http://vmworld.com.
• The VMware Community Forums: http://communities.vmware.com/
• And, 3rd party books like Scott Lowe’s Mastering VMware vSphere 4.

Do you have other sources of virtualization documentation or easy methods of searching documentation to find exactly what you need when you need it?  If so, leave a comment!

I recently received an invitation to the VMware Workstation beta program for the next version, currently code-named ‘Iron’.  I have yet to join the beta, but the invitation got me thinking about what new features I would like to see in VMware Workstation.  Once I join the beta I will not be allowed to publicly discuss the new features, so I’ll toss out some ideas now and see how many show up in future releases:

• Group Policy management of VMware Workstation installations.  I’ve spent the past four years in software development environments, and VMware Workstation has been a key component in those environments as it allows developers to create unique, portable environments for whatever it is that those crazy coders cook up next.  The problem, however, is that coders are crazy and sometimes do things in VM’s’ that are not, um, ideal.  I would love to be able to exert control over the VMware Workstation installations through Group Policy to help control the environment a bit.  Here are a few policy elements I can envision:
  • Allow certain types of network configurations, prevent others.  For example, I want to allow only host-only configurations so developers cannot use VM’s to get around security mechanisms or host lockdowns, or prevent bridged/NAT’d configurations so VM’s are less likely to be an entry point for malware.
  • Prevent inefficient  resource configurations by restricting VM’s from being configured with more than a certain percentage of the hosts memory, CPU count, etc.  I’ve seen people configure guests with the same amount of memory as the host and performance in both suffers.  Controlling this could save on help desk calls.
  • Control around disk configurations.  For example, I can picture policies that would block VMDK’s from being placed on removable media to prevent data leakage, or prevent users from creating a VMDK that would fill their local disk.
  • A policy that defines master image location and prevents VM’s not obtained from the central image repository from being started.  This would allow network admins to maintain some control over the base images on the network.  ACE capabilities probably come in here….
• Monitoring, Alerting, and Reporting of key aspects of VMware Workstations installations.  As a network manager, I want to be able to report on the VM’s that are in place on various hosts around the environment, and monitor their usage.  A report of all VM’s on a host with MAC address, host OS, IP address, and even a copy of the .vmx could be useful.
• Integration with VMware Lab Manager – Lab Manager is an awesome tool, but the SMB’s that I service are not always able to afford the resources needed to run every VM a development team can dream up, and there is plenty of horsepower on a modern workstation.  I would love to see a simple integration with Lab Manager that would allow users to be able to pull images from library or their workspace to their local workstation, and push images back to Lab Manager as necessary.
• The ability to use the host as parent, and guests VM’s as snapshots.  The physical host could be the standard image provisioned by IT and the VM’s as snapshots would allow users to customize the image or do revision testing of software without affecting the base image.
• 3D/Aero enabled within guest.  It doesn’t work in current versions, but I can see this coming along soon after watching VMware View and PC-over-IP demos of great 3D performance in VDI environments.
• Improved Unity experience.  It’s already pretty good, but can always be better.
• vShield Zones implemented in Workstation to control communication between VM’s, and of course, vShield Zones controlled through Group Policy.

That’s about all I can come up with off the top of my head.  I am not a heavy Workstation user myself, so some of my wish list may already be in the product.  I also know that some of the items on my list are in ACE, but ACE is not appropriate for all environments.  I am wondering how the continued drive to the data center and cloud, including virtual desktop technologies continuing to gain market share, may push Workstation/Fusion/Player out of the enterprise.

What features do you want to see in new versions of VMware Workstation (or Fusion)?  How can you envision Workstation integrating with other VMware products?  Leave a comment with your ideas!

I had some folks from our .NET development team come to me with a problem today – their ASP.NET code was taking forever to recompile after updates to the code base. But these guys are cool – they came with a proposed solution (most people who grace my office door are simply dropping off problems). Their solution? A RAMDisk mounted in a VMware Windows guest.  I give them credit for a novel approach, but I could see some issues:

• What would happen if the balloon driver kicked in and demanded the memory the RAMDisk was running on?
• A reservation would get around the balloon driver issue, but there is no way to specifically target the 512MB of RAMDisk, all memory in the VM must be reserved.
• I’m a pragmatic Windows systems administrator at heart, with a heap of systems and processes to manage and monitor.  I don’t want the additional burden of making sure the RAMDisk loads at boot, keeps a consistent image across boots, can be easily updated by new code pushes, and remains compatible with new VM hardware and Tools versions.
• A RAMDisk would take from what are already memory constrained VM’s, possibly hurting performance more than helping.
• If the disk subsystem is slow enough to get you thinking down the path of a RAMDisk, maybe it’s time for a new SAN…

I did some Googling around and couldn’t find any decent info.  I did find a few hits on people running VMware guests entirely inside a RAMDisk – a concept that peaked my interest almost enough to think about trying it just to say I did….  Have any of you experimented with a RAMDisk inside a VMware guest?  If so, what did you take away from the setup?  Was there a performance gain?  Where there gotcha’s?  Leave a comment if you have experience, guesses, or advice on this idea.

I needed to grab some stats from my ESX hosts for off-line analysis so I fired up my trusty ESXTOP intent on using batch mode to capture a .csv formatted output.  I started to manually select the counters I was interested in while working in ESXTOP interactive mode (you can save your selected counters to the esxtop configuration file with the ‘w’ command) and thought that there must be a better way.  I found that better way in the VMware Performance Community: http://communities.vmware.com/docs/DOC-3930.  There is now a -a switch that can be used to include ALL performance counters.  I’m sold.

I wanted detailed information, so I decided on a 15 second capture interval to run for a 2 hour window.  Here’s the command I used:

esxtop -a -b -d 15 -n 480 > /tmp/esxtopout.csv

where -a is for ALL, -b is for batch mode, -d is for delay, and -n is for the number of iterations ((60/15)*60*2).  I wrote out the results to a .csv in /tmp.  The resulting CSV weighed in at a whopping 100MB after 2 hours.

The CSV can be analyzed in Excel (pivot tables work well for this) or in Windows Perfmon.  I opened the log in Perfmon as I was after basic Min/Average/Max counters and Perfmon makes those easy to see.  When adding the CSV log to Perfmon, you are prompted to select counters.  I added all instances of Commands/sec, Reads/sec, and Writes/sec from Physical Disk (I was gathering some IOPS counts for a new storage proposal). I got a bit more than I bargained for: a mostly unresponsive Perfmon window and the ugliest darn graph I’ve ever seen.

Switching from a graph view to the report view allows you to easily view and remove specific counters that you are not interested in, or open the Properties of the data set, switch to the data tab and bulk select counters that you want to remove.  I was not interested in vmhba1:x, specific VM’s or worlds, so I killed all of those, leaving just the base iSCSI device (vmhba32 in my case).

After some cleanup the graph looked a bit better and more importantly, I was able to easily read my Min/Average/Max stats:

Here are the takeaways -

• ESXTOP is a powerful utility for performance monitoring
• All stats (-a) can result in a huge file – use it wisely in batch mode; else use interactive mode to select your counters and write them to the user-defined configuration file.  Invoke the config file with the -c option when running in batch mode.
• Consider using vscsiStats for more granular reporting.
• ESXTOP physical disk stats do not include NFS volumes.

Do you use other tools or methods to collect basic disk IO counters for storage sizing purposes?  If so, leave a comment describing your approach!

I have been meaning to write this up for a while; Scott Drummonds’ ‘Love Your Balloon Driver’ post today at his Virtual Performance blog gave me a nice reminder.  I actually caught a sneak peak at the graphs with an explanation from Scott at his instructor-led lab at VMworld 2009.  Scott calls out that the only workload they discovered suffers from balloon driver activity is Java.  The reason for Java’s problems with balloon driver activity is that Java itself runs in a VM and so the guest OS cannot properly determine which pages should be swapped out when the balloon driver calls for it.

My experiences causes me to agree with Scott and the whitepaper he cites – in a properly designed and equipped environment the balloon driver is not detrimental for most every workload to a point.   However, I recently discovered in a client site that the balloon driver can cause significant issues when the environment is poorly designed and under-sized.  Here the background:

I was called into an already established environment where the client was running on an older blade with VMware ESX 3.5.  The blade maxed out at 16GB RAM and had dual dual-core CPU’s with no hope for an upgrade.  On the blade was a single guest VM running Windows 2003 with SQL 2005, in it’s full 32-bit glory.  The VM was configured with 4 vCPU’s and 16GB of memory.  Some of you can probably already guess where this is going….

The x86 Windows guest had PAE configured, and SQL took advantage of AWE to use the additional memory beyond the 4GB limit of a 32-bit system.  Additionally, the Windows guest had the /3GB switch enabled in boot.ini.  Finally, as per SQL best practices, the ‘Lock Pages in Memory‘ permission was granted to the SQL Server service account.  What the guest was left with was 1GB of kernel mode memory and 15GB of User Mode/Extended addressable memory.

And here’s the problem.  The client was using ESX, not ESX 3.5, so the Service Console required memory.  In this case, the service console had approximately 512MB allocated to it.  Futhermore, VM’s require some overhead on ESX to run.  The memory overhead consumed by a Windows guest on ESX 3.5 with 4 vCPU and 16GB of memory is a bit more than 512MB.  On a properly sized ESX server with multiple similar guests/workloads, you could probably gain much of the overhead back through transparent page sharing; but in this case I had a 1:1 P2V ratio.  If you are any good at math you see that the environment is running about 1GB short of memory.  A quick check of the balloon driver stat in vCenter show that the balloon driver was constantly active and demanding about 1GB back from the guest… constantly.

Under normal circumstances this might not be an issue, but in this case the Windows guest was being absolutely punished.  The guest CPU’s were pegged at 100% with an excessive amount of kernel time, often indicating IO issues.  And indeed I did experience terrible disk and network performance on the guest.  At the root of the problem is this – the Lock Pages in Memory permission allows SQL to get a firm grasp on the user mode memory available to it (15GB) and lock it up.  This left the already starved (because of the 3GB switch in the boot.ini) guest kernel with it’s 1GB the only thing the balloon driver could really swap out.

The client suggested a reservation of 16GB on the VM, knowing that memory reservations prevent balloon driver activity.  I calmly asked them to back away from the keyboard as I explained how if a starved guest was bad, how much worse a starved Service Console would be.  In the end the fix was quiet easy – I convinced the customer that they should reduce the amount of memory allocated to the guest by about 1GB, enough to let the 512MB SC and the 512MB of overhead run without contention.  I was able to show them the difference between allocated and active memory in vCenter – the 1GB being surrendered was not really being actively used, SQL just had it locked up.  In fact, surrendering the 1GB of memory back to ESX breathed new life into the guest VM, bringing its performance back in line with expectations.

Ideally, I would have brought in a bigger ESX server that could serve additional VM’s, driving greater levels of efficiency across the environment.  It just wasn’t an option for the client in this case.  In the end, the problem was fixed and I was reminded just how fun it can be to explain some of these backwards sounding virtualization concepts to customers – fewer vCPU’s can lead to better performance of guests, less guest memory can fix performance issues, and increasing the quantity of similar guests on a host can drive better performance to a point because of transparent page sharing.

Stay tuned over the next few weeks as I digest and write on my VMworld experience – from VMUG activities to Paul Maritz’s press conference announcing the vCloud Express, and plenty of great sessions in between.  Like many of you, I returned from VMworld with quite a backlog of work but I’ll do my best to squeeze in some posts and tweets.

The VMware ecosystem is rich and thriving, with partners, vendors, and the virtualization community at large continuing to drive innovation and producing complementary products. VMworld is where this ecosystem comes together to show the latest and greatest they have to offer, and to give a glimpse into what lies ahead. With VMworld 2009 just a few days away, the ecosystem is pulsing with increase vigor, as evidenced by the increased amount of email, Twitter traffic, and general buzz in the community.

The VMworld 2009 Sponsor and Exhibitor list is available at http://www.vmworld.com/community/conferences/2009/sponsors-exhibitors and gives good insight into who and what is part of the VMware ecosystem.  If you are heading to VMworld next week, be sure to check out the exhibitor list ahead of time so you can target your time on the exhibition floor (or so you know which booths to hit up for the best schwag, tchotchke, booth babes, and T-shirts).