I’ve had several people ask me about using CD/DVD drives on VMware View virtual desktops.  Specifically,

• Can I use a CD or DVD drive with VMware View Desktops?
• Why doesn’t the CD or DVD drive on my re-purposed PC with the VMware View Client pass through to the View desktop?
• How can I make a CD or DVD drive work on my View Desktop?
• Are there specific CD/DVD drives that are supported on VMware View?
• How can I present a CD image (.ISO) to all View Desktops?

CD/DVD Devices in a View Environment – What works, what doesn’t?

First, the bad news.  When using PC over IP (PCoIP) as the connection protocol, IDE, SCSI, or SATA attached CD or DVD (or BluRay, HD-DVD, LaserDisc) drives local to the VMware View Client are not passed through to the VMware View virtual desktop.  This quite simply means that the internal CD drive of your re-purposed desktop will not be mapped to the View desktop when using PCoIP.  That said, mapping local CD or DVD drives to a View virtual desktop is supported when using Microsoft Remote Display Protocol (RDP).  Unfortunately, user experience (video performance, audio quality) declines and low/latent bandwidth  can increase if you switch from PCoIP to RDP.

The better approach to presenting a local CD/DVD drive to a View Desktop is to use a USB-connected drive.  This approach will work with both re-purposed PC’s acting as your View client as well as with Thin Clients and Zero Clients.  USB-connected devices can be passed through from a View Client to a VMware View virtual desktop while using PCoIP.  Teradici (the creators of PCoIP and maker of most of the Tera chips in industry leading Thin and Zero clients) has published a list of tested USB CD/DVD drives on their KB here http://techsupport.teradici.com/ics/support/default.asp?deptID=15164&task=knowledge&questionID=566.  Note that if you use some of these drives with a Windows XP View desktop, you may experience a View disconnect when ejecting a disc from the drive.  Other USB CD and DVD drives may also work – I grabbed this drive (OWC SuperSlim USB 2.0 Enclosure) from a colleague and was able to pass it through to my Clearpath Cloud hosted Windows 7 View desktop with no problems.

CD/DVD Drive Performance in a View Environment

While a USB connection will get your CD or DVD drive connected to your View desktop, performance may not be amazing.  For many tasks like software installs, reading images, or copying files performance will be acceptable.  A Teradici client can support up to 15Mbps for USB connected devices.  On a low bandwidth connection, or a connection with high latency, a local USB connected CD/DVD may not work well, if at all.  On a fast LAN connection, I would expect that redirected CD/DVD performance would be acceptable for many tasks, including reading medical images (PACS, DICOM viewer, RIS, HIS).  Do not expect to stream media or watch movies from a USB connected CD/DVD drive – but because View is typically a business solution, and watching the latest Hollywood hit at work is not often a business requirement, this should present few problems in all but the smallest niche cases (so get back to work!).  Disc burning would work with this configuration too – better for burning small images than a mass disc duplication effort.

Alternative Methods and Niche Cases

I’ve run into a few situations over the past decade or so of IT work where a desktop needed to have a CD/DVD perpetually connected to support some crummy application (FoxPro comes to mind here).  If you are supporting an environment like this, you can configure your parent VM with a persistently connected ISO on a VMFS or NFS vSphere datastore. Using this method, you could present the same CD image (.ISO) to all the VM’s based on the parent VM.  The connected ISO setting carries over to View Composer replicas and linked clones.  If you wanted to target this configuration to just a specific pool of View desktops, create a new snapshot of the parent image with the virtual CD/DVD drive set to ‘Connect at power on’ and an ISO file selected in the ‘Datastore ISO File’ field, as pictured below:

Note that in the VMware View Optimization Guide for Windows 7, and many other View tuning guides, it is recommended to remove the virtual CD/DVD drive from the parent VM.  If you follow this recommendation, you could still leverage a virtual CD/DVD drive, such as Microsoft Virtual CD Control Tool or UltraISO can also be used inside the View guest operating system, with an ISO library established on a low cost shared storage tier network share to give users a self-service way to mount a CD or DVD image.

Restricting Access to CD/DVD Devices in a VMware View Environment

While VMware-empowered IT is all about giving users freedom and control, there may be times where you want to restrict access to USB CD/DVD devices (or any other USB devices) in your VMware View environment.  You can do this in several ways – a complete solution will likely use a combination of the following methods based on user requirements, administrative overhead, and security posture:

• Use VMware View Group Policy administrative templates (.adm) applied to the View Client to control either ‘Connect all USB devices to the desktop on launch’ or ‘Connect USB devices to the desktop when they are plugged in’.
• If using RDP, use the same View Client (vdm_client.adm) administrative template to disable the ‘Redirect supported plug-and-play devices’
• Use built-in Active Directory Group Policy Objects (GPO), applied to your View Desktops OU’s, such as

• Administrative Templates | System | Removable Storage Access
• Administrative Templates | Windows Components | Remote Desktop Services | Device and Resource Redirection
• Administrative Templates | System | Device Installation | Device Installation Restrictions
• Use your Thin Client or Zero Client vendor’s management software to restrict access to, or disable, USB devices on Thin Clients.

Serial port devices in a VMware View environment

I’ve also had a couple questions about serial device (DB9 COM port) connections to View desktops.  Most of the above applies to serial devices too.  If serial devices are required in your environment, pick up a USB to Serial adapter (something like this will do Cables To Go 26886 USB To DB9 Male Serial Adapter, Blue (0.45 Meters/1.5 Feet)) and use GPO’s to control access and redirection of the USB device.

Wrap-up

Hopefully this answers your questions on using CD and DVD drives in a VMware View virtual desktop environment.  A combination of VMware View settings, vSphere VM configuration settings, and Microsoft Group Policy Objects (GPO’s) give you the ability to use and control removable media devices in your View environment.

Have you struggled with any other devices in your VMware View environment – leave a comment with your questions or suggested work-arounds!

Note: This article is cross-posted at the Clearpath Solutions Group blog at http://clearpathsg.com/weblog.

I have found VMware vSphere Resource Pools to be an often misunderstood element, and incorrectly implemented by even seasoned VMware administrators.  I recently found an example of an incorrect implementation of Resource Pools being used to organize VM’s in vCenter’s Hosts and Clusters view, without considering the potential performance impact of this configuration.  I thought that I might try to explain the resource pool piece a bit better, particularly around Shares.

First, a few technical points to remember:

• Resource Pools can only be used when VMware Distributed Resource Scheduler (DRS) is enabled on a cluster.  DRS is available in vSphere Enterprise and Enterprise Plus editions.
• VMware Shares are like stock/shares in a company –> Shares are proportional.  The more  shares you issue (on your company or in your VMware environment), the less valuable the individual share is.  The more shares you have, the greater your voting power/ability to claim a right to resources is.
• The vSphere Hosts Cluster itself is technically a Resource Pool too.
• Shares only come into play when there is resource contention (Memory or CPU).  If the host/cluster has spare resources available for a guest it allocates availability of those resources based on the configured vRAM size and vCPU count, taking into consideration any limits (Mem/CPU) on the VM.
• Resource Pools can be used to apply resource Shares to VM’s, as well as Reservations and Limits.  Shares, Reservations, and Limits can also be set on a per-VM basis.  The values on Resource Pools and individual VM’s combined can do funny things.

Now, consider the following screen shot of the environment running on my laptop lab:

I have 16 VM’s total, 4 in ResourcePool1, and 1 VM in ResourcePool2.  The resource pools are set to ‘Normal Shares’ with no other configuration changes to the default.  The VM’s are all the same size as far as vRAM and vCPU are concerned.

If we look at the ‘Resource Allocation’ tab of the Cluster, we see that the VM’s in the Cluster resource pool each have 5% of the Shares for CPU, and each ‘Normal’ resource pool has 21% of the total shares of the cluster.  If we divide the 21% shares by the 4 VM’s in ResourcePool1, we find that each VM gets 5.25% of the cluster’s total shares.  That means that the VM’s in this ‘Normal’ pool actually get a bit more CPU time under resource contention than those in the parent cluster pool.  The single VM (MoveVM1) in ResourcePool2 gets 21% of the shares all to itself – much larger than the remaining VM’s in the Cluster, even though it may not be a VM of higher importance in your environment.

Now consider Memory resource allocation:

The ‘Normal’ shares value for memory on a resource pool is 163840.  Compare this to the 640 shares each single VM gets in my current config (VM-* each have 64MB RAM, so a very low share value per VM).  We’ve effectively configured only a fraction of 1% (basically 0% shares) to the VM’s in the parent cluster pool, while the few VM’s in the ResourcePool1 and ResourcePool2 get to share 48% of cluster resources per pool.  This still leads to an imbalance for the VM’s in each pool as compared to VM’s in the sibling pool and to the VM’s in the parent pool.  I.E. – MoveVM1 gets all 48%; while the four VM’s in ResourcePool1 share 48% (48/4=12% each).

Now let’s say that ‘MoveVM1′ was just a VM that a well-intentioned administrator spun up for some testing, putting it in a resource pool for ease of locating it in an otherwise busy vCenter view.  This otherwise inconsequential testing VM now has the right to claim 48% of the memory (vRAM) in the cluster and 21% of CPU time when there is resource contention in the cluster.  Now if this lowly testing VM is the cause of resource contention on an otherwise healthy cluster, it’s getting its cake and eating it too.  When the going get’s tough, the least valuable workload gets the priority in this configuration!  We’ve created a case where the meek (VM) shall inherit the resources.  All your resources are belong to us.  You get the idea – a big fat oops!

Hopefully you can see how using Resource Pools as units of organization can negatively affect the performance of an environment under resource contention, even if those resource pools are configured with ‘default/normal’ values.  The risk becomes more severe if the values of those resource pools were to change (say, move to a ‘High’ value for shares on ResourcePool2 in my lab).

Best Practices

• It is best to use folders the ‘VM’s and Templates View’ in the vSphere Client for organizing VM’s and only (sparingly) use resource pools to achieve specific performance goals under resource contention (shares) and/or resource allocation goals (limits & reservations) under normal operating circumstances with resource availability.
• Use the Resource Allocation tab to review and tune your Resource Pool (and individual VM) Shares settings.  Export the resource allocation list from vCenter and use Excel if you want to experiment with values and do what-if calculations.
• Restrict permissions to create and modify resource allocation on VM’s and on Resource Pools in vCenter to only VMware Administrators who understand the implications of these changes, and who are capable of doing the math to properly design a resource allocation model.  If you don’t have an expert on staff, engage a consultant to help (might I suggest Clearpath Solutions Group ).
• Check out VMware vCenter Operations Manager (vCenter Ops/ vCOPs) for advanced monitoring and capacity planning (the product formerly known as VMware CapIQ is included with vCenter Ops).  Good monitoring and forecasting can help to avoid resource contention in highly consolidated, oversubscribed, or performance-sensitive environments (and what environments are not all three of these today?).
• Review the VMware vSphere Resource Management Guide to better understand the use and function of Resource Pools. The Guide can be found here: http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-resource-management-guide.pdf.
• Buy  Duncan Epping’s and Frank Denneman’s book, VMware vSphere 5 Clustering Technical Deepdive to really understand how VMware vSphere Distributed Resource Scheduler (DRS) (as well as VMware High Availability (HA) VMware Distributed Power Management (DPM), and VMware Storage DRS) function and are best configured.

Now that I’ve settled in at Clearpath Solutions Group, I’ll be writing occasional blog posts on the company blog site.  My first article went live today – a fancy little piece on using Microsoft Key Management Services with VMware View to activate Windows 7.  Here’s a quick excerpt:

When deploying or recomposing a pool of Windows 7 (or Windows Vista) virtual desktops on VMware View, you may receive an error message stating: “composer agent initialization error (16): Failed to activate software license (waited 0 seconds).”  This error occurs when you use a Multiple Activation Key (MAK) to activate the parent VM.  View Composer QuickPrep, by default, uses Microsoft Key Management Services (KMS).  VMware KB Article 1026556 provides workarounds for activating MAK clients, or bypassing activation by QuickPrep so you can use a trial key.  The workarounds are appropriate for test/pilot cases, but will eventually cause problems such as MAK key exhaustion.  The proper way to license and activate a deployment of Windows 7 on VMware View is to use a KMS.

Read the rest of my post on how to setup and use a KMS for VMware View on the Clearpath Blog: http://clearpathsg.com/using-kms-vmware-view-windows-activation.

Update: This issue was fixed as of 3/15/2012 in ESXi 5.0 Update 1 per the original knowledge base article: VMware KB 2008144. To download ESXi 5.0/vCenter Server 5.0 Update 1, see the VMware Download Center. 

I recently stumbled on two vSphere 5 ESXi networking bugs that I thought I would share. The issues are very similar from a cursory level, but have different symptoms, troubleshooting steps, and implications for your architecture, so I’m going to split the issues into two separate posts. Because troubleshooting these issues was a real pain, I’ll provide some details on how to identify these issues in your environments and wrap up with a third post on what I believe to be some best practices to avoid these same problems and achieve greater redundancy and resiliency in your vSphere environments.

The Problem

Today, we’ll look at an ESXi 5 networking issue that caused massive iSCSI latency, lost iSCSI sessions, and lost network connectivity. I’ve been able to reproduce this issue in several environments, on different hardware configurations. Here’s the background information on how all this started: I upgraded an ESXi 4.1 host to ESXi 5 using vSphere Update Manager (VUM). Note that I did use the host upgrade image that contained the ESXi500-201109001 iSCSI fixes – if you are upgrading to vSphere 5 and have iSCSI in your environment, use this image. Here’s a quick look at how the networking was configured on this host:

The iSCSI networking was configured in a very typical setup, and per best practices, as outline in VMware’s documentation, as well as from many vendors (see EMC’s Chad Sakac’s ‘A Multivendor Post on using iSCSI with VMware vSphere’), with two vmnic uplinks, two vmknics, with one active adapter on the correct layer-2/layer-3 network, and the other unused.

After the upgrade, the standard vSwitch with two vmnics for uplinks (Broadcom NetXtreme II BCM5709 1000Base-T) and two vmknics that serviced the software iSCSI adapter failed to pass traffic (vmkping to the iSCSI targets failed) and could not mount ANY iSCSI LUN’s. VM network, management, and vMotion ports were not affected.

If I let the host sit long enough, it *might* find a couple paths to the storage, but even then performance was deteriorated per the vmkernel.log:

WARNING: ScsiDeviceIO: 1218: Device naa.60026b90003dcebb000003ca4af95792 performance has deteriorated. I/O latency increased from average value of 5619 microseconds to 495292 microseconds.

Troubleshooting

I’m going to dump a whole bunch of my troubleshooting steps on you – hopefully they not only help folks dealing with this particular bug, but help with general network and configuration troubleshooting in VMware vSphere. During troubleshooting, I removed the vmk binding for these two on the iSCSI adapter, removed the software iSCSI Adapter itself, removed the vmknics on the vSwitch, and removed the vSwitch itself. I then recreated the vSwitch, set vSwitch MTU to 9000, recreated two vmk ports, set 9000MTU, assigned IP, and set failover order for multipath iSCSI. I then re-created the software iSCSI adapter and bound the two vmk ports. I was able to pass vmk traffic and mount iSCSI LUN’s. Great – problem solved!?!?! Not so much – I rebooted the host and the problem returned.

Here are my next troubleshooting steps:

• I repeated the procedure above and re-gained connectivity, but the problem returns on subsequent reboots. I can verifiably recreate the problem.
• I verified end-to-end connectivity for other hosts on the same Layer 1, Layer 2, and Layer 3 network as the iSCSI initiator and iSCSI targets.
• I verified the ESXi host’s networking configuration using the vSphere client, double-checking the vSwitch, vmnic uplinks, and vmknic configurations. Everything looked good so I canceled out.
• I then reinstalled ESXi from scratch (maybe something was left over from 4.1 that a clean install would weed out), built up the same configuration, and was again able to re-create the problem.
• I poured over logs (vmkernel.log, syslog.log and storagerm.log primarily). I could see an intermittent loss of storage connectivity, failure to log into the storage targets (duh – there is no connectivity, no vmkping) and high storage latency on hosts where I had rebuilt the iSCSI stack and run a few VM’s.
• I switched out the Broadcom NIC for an Intel NIC (the Broadcom had hardware iSCSI capabilities – I wanted to be sure the hardware iSCSI was not interfering).
• I verified TOE was enabled.

The ‘Ah-Ha’ Moment

Next, I verified the ESXi host’s networking configuration using the vSphere client one more time – the properties of the vSwitch, the properties of the vmkernel (vmk) ports, the manual NIC teaming overrides, IP addressing, etc. Everything looked correct – I MADE NO CHANGES – but when I clicked OK (last time I canceled) to close the vSwitch properties and was greeted with this warning:

Wait a second… I didn’t change anything, why am I being prompted with a you’re ‘Changing an iSCSI Initiator Port Group’ warning? I like to live dangerously, and wanted to see what would happen, so I said ‘Yes’.

Much to my surprise, after only viewing and closing the vSwitch and iSCSI vmk port group settings, I was able to complete a vmkping on the iSCSI-bound vmk’s. And moreover, I completed a Rescan of all storage adapters and my iSCSI LUN’s were found, mounted, and ready for use. Problem solved? Nope. The same ugly issue re-appeared after a reboot.

While the problem wasn’t solved, I now had something to work with. My go-to troubleshooting question “What Changed?” could maybe be answered. Even though I didn’t change anything in the vSwitch Properties GUI, something changed. To see what changed in the background, I compared the output of the following ESXi Shell (or vCLI, or PowerCLI) commands before and after making ‘the change’ happen (by viewing the properties of the vSwitch/vmk ports), but found no changes.

• esxcfg-vswitch -l
• esxcfg-vmknic -l
• esxcfg-nics -l

Then, I made backup copy of esx.conf

 cp /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

Then I caused ‘the change’ and then compared checksums using md5sum, but found no differences:

 md5sum /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

I compared the running .conf and the backup .conf, but found no differences:

 diff /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

Call in Air Support
At this point, I was out of ideas so I called for help: “Hello, 1-866-4VMWARE, option 4, option 2 – help!”

After repeating many of the same troubleshooting steps, the support engineer decided that I had hit on a known, and not yet patched, bug. The details of the bug are included in KB 2008144: Incorrect NIC failback occurs when an unused uplink is present. That’s right – my iSCSI traffic, vmkpings, etc were being sent down the wrong NIC – the UNUSED NIC. Ouch. The bug caused the networking stack to behave in a very unpredictable way, making my troubleshooting steps next to useless, and any other advanced troubleshooting ideas I had (sniffing, logs, etc.)

Once I knew what the issue was, I could see a bit of evidence in the logs:

WARNING: VMW_SATP_LSI: satp_lsi_pathIsUsingPreferredController:714:Failed to get volume access control data for path "vmhba33:C0:T0:L4": No connection

NMP: nmp_DeviceUpdatePathStates:547: Activated path "NULL" for NMP device "naa.60026b90003dcebb0000c7454d5cc946".

WARNING: ScsiPath: 3576: Path vmhba33:C0:T0:L4 is being removed

Notice the NULL path – the path can’t be interpreted correctly when being sent down the wrong (unsued) vmnic that is on a different subnet and VLAN. The gotcha on this issue is that I had followed best practices where applicable, and accepted default settings on the vSwitch and vmknics.

The Quick Fix
VMware KB 2008144 offers two workaround for this bug. The quick fix for the problem is to simply change the Failback setting on either the vSwitch running the software iSCSI vmknic’s to “No” (default is yes), or to change the setting on the vmknic itself if you have other port groups on the vSwitch (such as a VM Network port group to give your guest VM’s access to the iSCSI network).

Changing Failback = No on the iSCSI vmknics and then rescanning the storage adapters fix the glitch immediately.

Architecture Changes
The second workaround from VMware is “Do not have any unused NICs present in the team.”. This translates to a slightly different architecture than that described in many documents. To achieve this workaround, the configuration would have to change to two vSwitches, each with a single vmnic uplink and a single vmk port, bound to the iSCSI adapter. This change does not impact redundancy or availability when compared with the single-vSwitch:two-vmk configuration that I was running with as one of the vmnics was set to unused anyway. This workaround does add a bit more complexity, as there are a few more elements to configure, monitor, manage, and document.

This problem seems to only present itself on vSphere Standard Switches (vSwitch), although I could not get confirmation of this (please post a comment if you know!). Assuming this is true, a vDistributed Switch (vDS) could be used for Software iSCSI traffic. Mike Foley has a write-up on how to migrate iSCSI from a vSwitch to a vDS on his blog here: http://www.yelof.com/?p=72.

A Couple More Notes
My troubleshooting fix of viewing the vSwitch settings and clicking ok seemed to temporarily resolve the issues because it triggered an up/down event on the vmk of the unused uplink. This caused the network stack to re-evaluate paths and start using the correct, Active, uplink.

Note that this problem can occur outside of my iSCSI use case – any vSwitch, Port Group, or VMKNIC with an unused adapter set in the NIC Teaming Failover Order are susceptible to this bug, so watch for it on redundant vMotion networks (vMotion randomly fails), VM Network networks (sudden loss of guest connectivity), or even your management network (hosts fall out of manageability from vCenter, and can’t be contacted via SSH, vSphere client, etc.
Leave a comment if you’ve experienced this bug – your notes on the problem may help others find and fix the issue until VMware releases a fix. I understand that a fix for this particular bug is not due out until at least vSphere 5 Update 1.

I’ll have another (shorter) writeup on the 2nd networking bug I found in ESXi 5 later in the week – check back here for a link once it is published.

I’ve been asked several times recently to recommend training resources for VMware, so I thought I might write my responses up in a blog post to help out folks in the community who are looking for the best resources to gain VMware knowledge, prepare for their VCP and other certifications, and continue on their journey to becoming a virtualization rockstar.

I’ve picked up a bunch of certifications over the past 10 years.  For me, certification is not the means to an end, but the end of some long, intensive studying and lab work, then doing some deep dive studying and doing.  By the time I get to the test, passing should be a forgone conclusion.  I’ll save details of my lab for a future post and focus on the books and other learning resources that I use.  When getting into a new or updated technology, I start out my studying with a good overall survey of the technology I want to learn.  I like a good book that hits all of the major components, provides background information to help explain why the technology, component, or module really matters and how it fits into the big picture.  Then I get into technology specific books – deep dives, command line references, and architecture books.

Books

Mastering VMware vSphere 5, by Scott Lowe

My go-to book for VMware vSphere has been Scott Lowe’s Mastering VMware vSphere 4.  Scott’s updated book, Mastering VMware vSphere 5 started shipping yesterday.  Scott covers everything from the basics of what a hypervisor is to VMware vSphere best practices.  This is a great book to accompany lab work as it includes licensing, planning and installation, setting up virtual networking, storage basics, security, resource allocation, HA, DRS, and even some automation with the CLI and PowerCLI (PowerShell).  The book is well written, taking you methodically through vSphere, while providing plenty of helpful hints along the way.  Do yourself a favor and click the picture to the left to order it from Amazon now (paperback or Kindle format).  This book is a great way to get started with studying for your VCP certification.

VMware vSphere 5 Clustering Technical Deepdive

Once I have the basics down, I get into the deep dive work. The first deep-dive book for VMware vSphere 5 is VMware vSphere 5 Clustering Technical Deepdive, by Duncan Epping and Frank Denneman.  This is Duncan and Frank’s second book that focuses on the clustering and high availability technologies available in VMware vSphere.  Readers of Duncan and Frank’s first book, VMware vSphere 4.1 HA and DRS Technical deepdive (Volume 1), got an incredibly deep look at how to configure VMware HA and DRS.  The new vSphere 5 Clustering Technical Deepdive includes Storage DRS as well.  I’ve talked to several readers of both these books and Duncan and Frank’s blogs who have remarked that 1.) I’ve been doing it wrong all along, 2.) I totally understand how HA and DRS work after reading this, and 3.) My environment really is resilient and reliable thanks to this book.

Pearson and VMware teamed up earlier this year to create VMware Press.  There are several books coming from VMware Press, as well as other authors/publishers that are now available for pre-order from Amazon.com.  These include:

There are not many vSphere 5 specific books out yet, but many of the vSphere 4 resources are still very useful.  My library includes these:

Video Training

If you are not a big reader or you are looking for additional topics, check out TrainSignal’s VMware Training Videos.  TrainSignal offers a whole slew of courses (many taught by VMware vExperts), including:

• vSphere 5 Training
• VMware View Administration Training
• vSphere Troubleshooting
• vSphere Performance Monitoring
• vSphere Security Design
• vSphere PowerCLI.

I have a couple of TrainSignal DVD’s and found them to be good quality with deep technical content.

Blogs and BrownBags

Once you are comfortable with the material, you can start to study for your VCP.  Several bloggers have published collections of materials to help you prepare for the VCP, VCAP, and even the VCDX.  I recommend Simon Long’s collection here: http://www.simonlong.co.uk/blog/vcp-vsphere-upgrade-study-notes/ and Cody Bunch’s VCP4 Resource Page and BrownBag sessions.

Instructor Led & Certification

Finally, once you are all read up, head to a VMware Education instructor led class.  You need to take a VMware Authorized Training course to qualify to sit for the VMware Certified Professional (VCP) certification exam.  VMware also offers a nice catalog of eLearning courses.  If you want to get a discount on eLearning, Instructor-Led training, and certification exams from VMware, check out the VMUG Advantage program.

My last post described a problem I experienced with VMware HA after upgrading to vSphere 4.1.  Here is my experience with a similar issue after applying the ESXi410-201010401-SG patch to one of my test/dev ESXi clusters.  The patch, released on November 15th and weighing in at a hefty 212MB, fixes a number of issues from Likewise authentication on ESXi hosts to allowing configurable NOOP timout and interval values for faster failover of certain iSCSI arrays (like the DS3300 or MD3000i).

The environment where this problem occured has a single vCenter server managing both a production cluster and the test/dev cluster.  After applying this particular update to the ESXi hosts in the cluster, the vCenter server began to crash every 5 minutes or so.  The crash was logged on the vCenter server with Event ID 7031: The VMware VirtualCenter Server service terminated unexpectedly.  My go-to troubleshooting question (“What changed?”) pointed at the ESXi patch, but a VMware KB search and a little Google action yielded no results directly related to ESXi410-201010401-SG and the vCenter Server service terminating unexpectedly.  VMware KB article 1003926 provides some basic troubleshooting steps for vCenter Server, such as checking for port conflicts, vCenter DB health & availability, and log locations.  The environment was healthy until the patch was applied to a sub-set of my ESXi hosts so I could confidently eliminate credentials, port conflicts and the like as the cause of the problem, so I jumped right to the log files for vCenter.  The vpxd-*.log is found in “C:\ProgramData\VMware\VMware VirtualCenter\Logs” on Windows 2008 vCenter servers and “%ALLUSERSPROFILE%\VMware\VMware VirtualCenter\Logs\vpxd.log” on Windows 2003 servers.  I found a few lines of interest in the log file but decided I had better call VMware Support to further analyze the issue.

To make a long story short, what the logs revealed is a bug that is triggered whenever VMware Distributed Resource Scheduler (DRS) ran on the updated test/dev cluster.  Disabling DRS stopped the symptom of the vCenter Server Service terminating unexpectedly, but this was obviously not a long-term solution.  A bit more digging by my VMware support rep led to VMware Distributed Power Management (DPM) being enabled on the cluster as the root cause of the issue.  Disabling DPM but leaving DRS enabled on the cluster fixed the glitch.  I can live without DPM, but DRS is pretty darn handy.

At this point, VMware engineering knows about the issue, and a fix is planned for vCenter 4.1 Update 1.  Interesting that DPM was fingered in this case, as well as in the case I wrote about last week where HA and DPM apparently do not always play well together.  It seems like DPM is not fully baked, even though it is now officially supported.  This is unfortunate as DPM is promising to me – I can imagine the technology behind DPM being used for intelligent load shedding during peak electrical cost hours, power outages, or cooling outages in datacenters with some good integration between a DPM API and environmental management and monitoring systems like APC’s NetBotz.  Anyone else using DPM without having problems?  Any ideas for extending DPM or leveraging it for other purposes in the datacenter – I’d love to hear ideas in the comments.

I recently ran into an issue with one of my vSphere clusters after upgrading from vSphere 4.0 to vSphere 4.1 (with ESXi 4.1 and vCenter 4.1).  After the upgrade, I attempted to enable VMware High Availability (HA) on the upgraded cluster.  Each of the ESXi hosts in the cluster appeared to have been properly configured for HA (as observed in the ‘Recent Tasks’ pane of the vSphere Client).  Despite having appeared to configure HA correctly, I found that each host in the cluster was displaying an error on the Summary tab of the vSphere Client that read ‘Error <date> <time> HA agent on <host> in cluster <clustername> in <datacenter> has an error: Error while running health check script’.

I’ve dealt with HA errors in the past, so I quickly jumped into my standard troubleshooting and quick-fixes proceedure:

1. Verify host connectivity.
2. Right-click on each host and choose ‘Reconfigure for VMware HA’
3. Disable & Re-enable HA on the cluster.
4. Disable HA, place hosts into Maintenance Mode & Reboot (one at a time).  Re-enable HA.
5. Get frustrated that a quick fix is not probably not in my future….
6. Verify host name resolution for each host in the cluster from the service console/tech support mode of each host.
7. Review log files on vCenter Server and each host for glaring issues.  All Greek to me in this case….
8. Call VMware Support.

VMware Support reviewed the log files I had attached to my Service Request (SR) when I opened the case and had me try a few different things to fix the issue.  First, we verified the steps I had taken and collected some fresh logs.  Next, the support rep had me verify that Distributed Power Management (DPM) was not enabled on the cluster as there apparently is a known issue (although a KB is not available at this time) with configuring HA when DPM is enabled under certain circumstances.  I did not have DPM enabled on this particular cluster so I didn’t spend time chasing down this particular bug.

Finally, the following proceedure, run on each ESXi server in the cluster, resolved the issue (Note – this procedure is safe to do during normal operations as it does not affect running VM’s):

1. Verify SSH or Console access to the host (this requires enabling Remote SSH/Tech Support Mode on ESXi hosts on the Configuration tab | Security Profile node of the vSphere Client, or by pressing F2 to login to ESXi 4.1 | troubleshooting options | enable remote SSH.
2. Disable HA on the affected cluster.
3. Right-click | Disconnect each host in the cluster from the ‘Hosts & Clusters’ view of the vSphere Client.
4. SSH to the host and run the following commands:

services.sh stop
/opt/vmware/uninstallers/VMware-vpxa-uninstall.sh
/opt/vmware/uninstallers/VMware-aam-ha-uninstall.sh
services.sh start

5. In the vSphere Client, right-click on each host and Connect.
6. Enable HA on the cluster.

This procedure cleanly removes the VMware vCenter agent and the VMware HA agent from the ESX or ESXi host.  Reconnecting the host to vCenter pushes the vCenter management agent back to the host and installs it cleanly.  Enabling HA on the cluster re-installs the HA agent.  After completing these steps I had no further issues with HA on the cluster – case closed.  I hope this is helpful for anyone else who might be experiencing HA errors after upgrading to vSphere 4.1.

For those wanting to learn HA best practices or go a bit deeper into the inner workings of VMware HA, I highly recommend Duncan Epping’s VMware HA Deep Dive article and/or VMware vSphere 4.1 HA and DRS Technical Deepdive (Volume 1) book.

I ran into an issue with a customer today where a VM was performing terribly.  From within the guest OS (a Windows 2003 application server running .NET in IIS which I will call BigBadServer) things appeared sluggish and CPU time was high.  The amount of time being spent on the kernel was notably high.  The VM in question had 4 vCPU’s and a good helping of memory.

I don’t have access to the VMware client at this particular site – just some of the guests, so I was flying blind.  Gut feeling told me that I was dealing with a resource contention issue.  I had the VMstats provider running in the guest (http://vpivot.com/2009/09/17/using-perfmon-for-accurate-esx-performance-counters/) showed me that there was no ballooning or swapping going on, and that the vCPU’s were not limited and the CPU share value seemed to be at the default.

I strongly suspected that the physical server running VMware ESX was oversubscribed on physical CPU (pCPU) resources.  Essentially, the guest VM’s that are sharing the resources of the physical machine are demanding more resources than the machine can handle.  To verify this theory, I had the client check the ‘CPU Ready’ metric on BigBadServer and bingo!

CPU Ready is a measure of the amount of time that the guest VM is ready to run against the pCPU, but the VMware CPU Scheduler cannot find time to run the VM because other VM’s are competing for the same resources.

From the stats the customer provided on our phone call, the CPU Ready for any one of the 4 vCPU’s on the BigBadServer was on average 3723ms (min: 1269ms, max:8491ms).  (Update 8/25/2010 to clarify summation stat) The summation for the entire VM was around 12,000ms on average and peaked around 35,000.  The stats came from the real-time performance  graph/table in the vSphere client. The real-time stats in the vSphere Client update every 20 seconds, so the CPU Ready summation value should be divided by 20,000 to get a percentage of CPU ready for the 20 second time slice.  If I take the worst case scenario of 8491ms per vCPU, this VM spent nearly 43% (8491/20,000) of the 20 second time slice waiting for CPU resources.

The CPU Ready summation in milliseconds counter in the vCenter Client is not always the most accurate or easy to interpret stat – to better quantify the problem it might be best to go to the ESX command line and run ESXTOP.  CPU Ready over 5% could be a sign of trouble, over 10% and there is a problem.  Running ESXTOP in batch mode and then analyzing the output using Windows Perfmon or Excel might be a good way to go on this to get a view over several hours rather than the realtime stats we were looking at.  I wrote a post a while back with more info on ESXTOP batch mode: http://vmtoday.com/2009/09/esxtop-batch-mode-windows-perfmon/

To help quantify the problem a bit more, the BigBadServer is on an ESX 4.0 server with about 10 other servers.  The physical blade has two dual-core CPU’s (AMD Opteron 2218HE’s which are not hyperthreaded).  The other VM’s on the blade have different vCPU and vMemory configurations.  3 VM’s (including BigBadServer) have 4 vCPU’s.  A couple have 2 vCPU’s, and the remainder are configured with 1 vCPU.  In ESX 4.x, the VMware console OS actually runs as a hidden VM, pegged to pCPU #1.

I generally recommend a pCPU:vCPU ration of 1:4 for mid-sized VMware deployments of single vCPU VM’s.  The blade we are running on is a 1:5 with several multi-vCPU VM’s.  The multi-vCPU’s start to skew the ratio recommendation and require some advanced design decisions.  VMware’s scheduler requires that all the vCPU’s on a VM run concurrently (even if the Guest OS is trying to execute a single thread).  Also, the VMware CPU Scheduler prefers to have all the vCPU’s from a VM run on the same pCPU.  As workloads are bounced around between pCPU’s, the benefits of CPU cache are lost.  This is one of those ‘more-is-less’ situations that you run into on virtualized environments.

What this CPU Scheduler nonsense means in this case is that the 4 vCPU’s on BigBadServer have to wait until all logical pCPU’s on the box are idle (including the one that runs ESX itself) before it can run.  If ESX can’t accomplish that (we are experiencing resource contention) it starts prioritizing workloads according to what it can best run.  It is much easier to schedule the smaller VM’s, so it tends to run those on pCPU more frequently.  The larger VM’s tend to suffer a bit more than the smaller ones.  We are competing with 2 other VM’s with 4 vCPU’s that use up all of the logical pCPU’s when they need to run, as well as with the smaller VM’s.

I suggested a few ways to fix this issue for the BigBadServer web server:

1. Using Shares and/or Reservations on the VM.  This probably won’t work in our situation as the physical server is too over-subscribed.  We might see a slight improvement in BigBadServer (or we might not see any change), but possibly at the extreme expense of the other VM’s sharing the blade.
2. Reduce the number of vCPU’s on BigBadServer AND the other multi-vCPU VM’s on the same physical server.  This would reduce resource contention and open up a whole bunch of scheduling options for the VMware CPU Scheduler.  This is the quickest/cheapest fix, but will not work if the VM’s really do need 4 vCPU’s.  A little workload analysis should determine which can be made smaller (the vCenter server graphs/stats should be enough for this).  For what it’s worth, by our analysis BigBadServer seems to be happier with 4 vCPU assuming we can run with a low CPU Ready on those 4.
3. Move the BigBadServer VM to a physical ESX server with fewer multi-vCPU VM’s so there is less contention.
4. Move the BigBadServer VM to a physical ESX server with quad-core pCPU’s (ideally two quad-cores or bigger).  This would give a lot more flexibility to the VMware CPU Scheduler and allow it to run quad-vCPU VM’s on the same pCPU for greater efficiency.
5. Split BigBadServer into 2 smaller VM’s – The server currently runs a couple sites.  We could split them onto two servers – one for Project1 and one for Proejct2.  This configuration would take some design, testing, and time but could scale out better, give more flexibility and availability in the long run.

I’m not sure which way the customer will go on this one yet, but I feel good having armed them with enough knowledge and options to make an informed decision.

To avoid problems like this in the future, I recommend these rules of thumb:

• Design your hosts for your guests.  Taking your Guest VM sizes into account when designing your environment and choosing physical hardware is crucial if you need bigger VM’s.
• Don’t make your VM’s bigger than you have to.  It is always easier to add resources than take them away.  Hot Add of CPU and Memory in vSphere make adding incredibly easy.
• Monitor your environment for CPU Ready, Swapping, and other metrics that can indicate an inefficient design.
• Call for help when you can’t figure out what is going on (I’m happy to help!).  VMware is super powerful, but some things can be downright backwards when it comes to resource allocation on a fixed set of hardware.

If you are looking for some resources to help explain CPU Scheduling a bit more, I recommend:

• VMware’s Official documentation of CPU Scheduler in vSphere 4.1 – http://www.vmware.com/files/pdf/techpaper/VMW_vSphere41_cpu_schedule_ESX.pdf.
• A nice summary of co-scheduling from VMware’s Performance Blog: http://blogs.vmware.com/performance/2008/06/esx-scheduler-s.html
• Description and stats on Ready Time metrics for VI3: http://www.vmware.com/pdf/esx3_ready_time.pdf
• Understanding Virtual Center Performance Statistics: http://communities.vmware.com/docs/DOC-5230.pdf

(Updated 8/25/2010 to include a few additional reference links and corrected summation divided by time slice to get accurate values)

I posted an article in December on how the SVGA driver included with VMware Tools caused the guest VM to freeze.  I referenced VMware’s KB Article 1011709, which directed you to not use the SVGA drivers included with VMware Tools.  KB1011709 has since been updated (as of February 25, 2010) to indicate that the VMware Tools package included with ESX 4.0 Update 1 includes a new WDDM driver that is fully supported.  If you have updated to Update 1, you should upgrade VMware Tools to take advantage of the new driver.

If you followed the KB1011709′s original advice and did a custom install of VMware Tools (leaving out the SVGA driver through a custom install), you may have to do a re-install of VMware Tools before the new driver is available.   Once you get VMware Tools upgraded, the new driver can be found in the guest VM at C:\Program Files\Common Files\VMware\Drivers\wddm_video.  These drivers are not automatically installed, so you’ll have to update your guest’s video adapter driver in Device Manager.

It’s a bummer that the WDDM SVGA drivers are not automatically installed.  You could  probably copy these drivers to other VM’s and use Windows Device Manager to replace the standard driver with the newer WDDM driver without having to do the uninstall, reboot, reinstall of VMware tools on all of your VM’s.

Just as I was about to publish this, I saw a TweetDeck pop-up from @jasonboche saying that he had published pretty much the same update here: http://www.boche.net/blog/index.php/2010/03/28/windows-2008-r2-and-windows-7-on-vsphere/.  Not only does he have pretty pictures to go with his post, but also points out that VMware Tools installs/upgrades executed with VMware Update Manager (VUM) will not install the upgraded SVGA driver.  He also recommends updating templates to include the upgraded drivers.  Great points, Jason.

I am finishing up an installation of an EMC Clariion CX4 SAN. One of the final steps of the installation is to configure PowerPath/VE on the ESXi hosts. PowerPath/VE is EMC’s multipathing extension module for VMware (and Hyper-V), designed to replace the Native Multipathing Plugin (NMP) for increased I/O performance and failover management.  To simplify and automate the installation of PowerPath/VE, I decided to use VMware Update Manager (VUM) to push the extension to the ESXi 4.x hosts in the environment.

The process of setting up an additional VUM patch repository to host PowerPath/VE (and other 3rd party extensions such as the Cisco Nexus 1000v) is pretty straight forward.  3rd party extensions are supported in VUM beginning with vSphere 4.0 Update 1.  Chad Sakac has posted a great video guide on YouTube that covers the setup:

I opted to use the tomcat installation on the environment’s vCenter server to host the PowerPath/VE repository.  To accomplish this, I simply created a new directory in the tomcat root directory.  The default path for the root directory on a vSphere vCenter Server is “C:\Program Files\VMware\Infrastructure\tomcat\webapps” (or C:\Program Files (x86)\VMware\Infrastructure\tomcat\webapps on a 64-bit installation).

I created a directory named ‘depot’ and within that directory created a PowerPathVE folder.  I extracted the contents of the VUM folder from the PowerPath .zip file that I downloaded from http://powerlink.emc.com.  A screenshot of the directory is below:

PowerPath/VE Depot Directory Tree

After creating the directory for the patch repository, I simply added an Extension Repository to VMware Update Manager as Chad shows in his video.  I would like to call out one caveat – Because vCenter may not listen on standard HTTP/HTTPS ports, I used https://vcenter.domain.local:8443/depot/PowerPathVE/index.xml as the path to the source.

VUM Patch Source

Once PowerPath was added to an Extension Baseline in VUM, I simply had to scan my hosts for updates and remediate.  Installation of PowerPath/VE requires the host to be in Maintenance Mode and concludes with a reboot.  Pretty simple.

Then all you have to do is fight through an overly-complex licensing setup (seriously, a 112 page PDF on how to install licenses???), a bit of configuration, and you are multi-pathing with the best of them.  If you are interested in learning more about PowerPath/VE, start with this whitepaper: EMC PowerPath/VE for VMware vSphere Best Practices Planning.  For a bit of real-world insight into the performance increase you might see with PowerPath/VE, check out this blog post from Eric Sloof: Massive I/O power increase using EMC PowerPath/VE.

Update – 3/27/09: VMware published a Knowledge Base article on this procedure a few weeks after I wrote this post.  You can find it in  article 1018740.

Update – 4/15/11: You may have to set the NTFS permissions on the ‘depot’ folder to allow ‘anonymous’ read access when running on a 2008 or 2008 R2 server before you can validate and download from the new repository.

In Part I of this series, I discussed the important of storage performance in a virtual environment (really any environment, virtual or not, where you want acceptable performance), and introduced some of the basic measures of a storage environment.  In Part II, we will look more closely at what may be the most important storage design consideration in a VMware server-consolidation enviornments, many SQL environments, and VDI environments to name a few: IOPS.

If we stick with a single-disk-centric approach as we did in Part I, IOPS is quite simply a measure of how many read and write commands a disk can complete in a second.  IOPS is an important measure of performance in a shared storage environment (such as VMware) and in high-transaction-rate workloads like SQL.  Because hard drives are forced to abide by the laws of physics, the IOPS capabilities of a disk are consistent and predictable given a specific configuration.  The formula for calculating IOPS for a given disk is pretty straight forward (please show your work):

IOPS = 1000/(Seek Latency + Rotational Latency)

Exact latencies vary by disk type, quality, number of platters, etc.  You can look up the tech specs for most drives on the market.  As an example, I have randomly chosen the technical specifications of the Seagate Cheatah 15k.7 SAS drive.  This particular drive has the following performance characteristics:

- Average (rotational) latency: 2.0msec

- Average read seek (latency): 3.4msec

- Average write seek (latency): 3.9msec

Using the read latency number, the math works out like this:

1000
———- = 185 maximum read IOPS
2.0+3.4

The maximum write IOPS will be a bit less (~169IOPS) because of the higher write seek latency.  Writing is more ‘expensive’ than reading and therefore slower.

Fortunately, there are some widely accepted ‘working’ numbers, so you do not have to use this formula for each and every disk you might consider using.  Because rotational latency is based on the rotational speed, we can use the published Rotations Per Minute (RPM) rating of the drive to guess-timate the IOPS capabilities.  Typical spindle speeds (measured in RPM) and their equivalent IOPS are in the table below.

RPM………IOPS

7,200          80

10,000       130

15,000       180

SSD           2500 – 6000

While not a traditional spinning disk, I have also included Solid State Disks (SSD’s) for reference as SSD’s are starting to see increased market adoption.  I have seen a wide range of sizing IOPS for SSD depending on the technology, type (SLC, MLC, etc.)  Check out http://en.wikipedia.org/wiki/Solid-state_drive for an introduction, and ask your vendors for more in-depth technical information.

If you are brand-new to this (and you are still reading, congrats!), you can see how many IOPS your Windows computer is asking for by opening Performance Monitor and looking at the ‘Disk Transfers/sec’ counter under Physical Disk.  This is a sum of the ‘Disk Reads/sec’ and ‘Disk Writes/sec’ counters as you can see in the screenshot below:

If you are after some stats for your VMware ESX environment, check out esxtop and looking for CMDS/s in the output.  I published a couple articles on using esxtop here and here.  The numbers from PerfMon and esxtop get you pretty close but can be skewed by a few things we’ll discuss in later posts.

Now that was fun and all, but let’s get real: Single-disk configurations are uncommon in servers.  As such, we’ll part ways with our Simple Jack single disk approach to storage and begin to look at more real-world multi-disk enterprise-class storage configurations.  A discussion of IOPS in a multi-disk array is a great way to start.  From a very elementary perspective, you can combine multiple hard drives together to aggregate their performance capabilities.  For example, two 15k RPM disks working together to server a workload could provide a theoretical 360 IOPS (180 + 180). This also  scales out so ten 15k RPM disks could provide 1800 IOPS, and 100 15k RPM disks could provide 18,000 IOPS.

Designing your environment so that your storage can deliver sufficient IOPS to the requesting workload is of utmost importance.  If you are working on a storage design, arm yourself with data from perfmon, top, iostat, esxtop, and vscsiStats.  I typically gather at least 24 hours of performance data from systems under normal conditions (a few days to a week may be good if you have varying business cycles) and take the 95th percentile as a starting point.  So from a very simple approach, if your data and calculations show a 1800 IOPS demand at the 95th percentile, you ought to have at least ten 15k RPM disks (or twenty-three 7.2k RPM SATA disks) to achieve performance goals.  It’s amazing how some simple data and a pretty little Excel spreadsheet can help you understand and justify the right hardware for the job.

Now before you go and start filling out that PO form for a nice new storage system based on these numbers there are a few more things we ought to discuss.  RAID, cache, and advanced storage technologies will skew these numbers and need to be understood.  Stay tuned to future articles in this series for more on those topics and more.

Finally, there has been a bunch of activity in the VMware ecosystem of vendors, bloggers, and twittering-type-folks around storage performance.  As this here post sat in my drafts folder, Duncan Epping posted this gem of an article that pretty much included all of the content of this article, as well as future ones in my series: http://www.yellow-bricks.com/2009/12/23/iops/.  Do yourself a favor and read his post and the comments from his readers – both are filled with a ton of great information, including some vendor-specific implementations.
I was led to Duncan’s article by a post by Chad Sakac on his blog: http://virtualgeek.typepad.com/virtual_geek/2009/12/whats-what-in-vmware-view-and-vdi-land.html.  This is also a great read that covers some of the same information with a focus on VMware View/VDI and is also worth a few minutes of your time.  Also check out http://vpivot.com/2009/09/18/storage-is-the-problem/ for a rubber-meets-the-road post from Scott Drummonds on the importance of storage performance vis-a-vis IOPS in a VMware-virtualized SQL environment.

I recently ran into an issue when installing my first Windows Server 2008 R2 virtual machine.  The VM would hang/freeze randomly when used through the VMware vCenter Client’s console.  It turns out this is a known issue (see this VMware KB Article) with the SVGA driver that is installed as part of the default installation of VMware Tools.  While the article does not explain why you should disable the SVGA driver, it’s advice is correct if you want to avoid problems in your guest VM.  To correct my problem, I removed the SVGA driver from the Windows Device Manager and rebooted.  If you are having problems removing the SVGA driver before the VM hangs, use Remote Desktop to access the guest machine to perform the driver uninstall.  I have not observed hanging/freezing in the VM since removing the SVGA driver from my Windows 2008 R2 guest.  Note that this same issue is present in Windows 7.

I recently posted an article on how specific actions during the upgrade of a VMware Virtual Machine’s hardware from v4 to v7 can cause problems with certain services, including DNS, DHCP, and WINS. In that case, the problem was related to Microsoft Windows leaving non-present devices with networking configurations and  the failure of the VMware Upgrade Helper service to copy WINS settings when updating the NIC.  As my fellow blogger and VMUG leader, Jason Boche, responded on Twitter: “Same gotchas, different version.”  And right he is – anyone with experience in P2V or V2V, or who has been working with VMware long enough to have done a 2.5 to 3.0 upgrade experienced the same gotchas.

There are other issues with VMware virtual hardware upgrades, however, that you may not have experienced.  One such issue that I have experienced is highlighted in VMware Knowledge Base article 1013109: “Upgrading virtual hardware in ESX 4 may cause Windows 2008 disks to go offline“.  The problems described in the article are unique to Windows 2008 Enterprise and Datacenter editions only.  The problem is pretty well described in the title of the article – Upgrading virtual hardware in ESX 4 may cause Windows 2008 disks to go offline.  In this case, like with the ghost NIC’s I described last week, is more of a Microsoft issue, but it will rear its head when a VMware Administrator least desires it.  With this particular problem, the Windows Virtual Disk Service (part of the native Storage Management suite) is set to not auto-mount newly discovered disks that do reside on a shared bus.  Microsoft has a MSDN article on the VDS SANS policy here.  Upgrading the virtual hardware version causes the disks to be re-discovered and not auto-mounted.  This can potentially impact all non-system disks on a VM.

You may also experience similar issues when upgrading the vSCSI adapter in a VM from a standard LSI Logic Parallel SCSI adapter to a (new in vSphere 4.0) paravirtualized SCSI (pvSCSI) adapter, move virtual disks to new vSCSI adapters to increase the number of concurrent disk IO operations, or when you change the SCSI node ID of a virtual disk.  These may all trigger a re-discovery of the disks by the Windows Virtual Disk Service, leaving data disks offline on Windows 2008 Enterprise and Datacenter Edition guests.

In my opinion, these issues are not reasons to forgo upgrading your virtual hardware version.  However, when your upgrade/migration plans call for upgrading the virtual hardware version of your guests you should be prepared to resolve any issues caused by ‘ghost hardware’, offline disks, and the like.  Both the MSDN and VMware articles I cited above offer workarounds for the offline disk issue.  Here are the links again:

A reader named Mark contacted me today and asked if there was a way to reduce the size of the batch output from an ESXTOP run.  And he asks for good reason: Depending on the number of VM’s on your host, the delay between ESXTOP samplings and the number of samples you collect, using the All Stats option (-a) can yield a massive file in a short period of time.  If written to a partition on your ESX Service Console you run the risk of filling the partition, and forget about actually being able to analyze the data in PERFMON or Excel.  For example, on an ESX host running ~15 VM’s I produced 100MB worth of CSV using the -a switch, sampling every 15 seconds, for just under 2 hours.  ESXTOP uses 10-second intervals by default; I used -d 15 to change the sampling delay.  Had I went with the default my output would have been bigger.

To reduce the size of your output, you can change your sampling delay to something larger, say 30-seconds.  I suppose you could also capture statistics when the host is not busy so you get fewer characters in the results, but that’s just being goofy.

A better way to reduce your ESXTOP output size is to selectively include only the statistics you are interested in, and is really what Mark was asking.  After all, all statistics from ESXTOP can be too many statistics, and chances are you already know what stats you are interested in.  Here’s how you can narrow down the collected stats for easier analysis and smaller output:

1. Enter ESXTOP in interactive mode on the Service Console by simply typing esxtop at the # prompt
2. Switch to a component you are NOT interested in capturing statistics on by pressing the corresponding menu option (c: ESX cpu, m: ESX memory, d: ESX disk adapter, u: ESX disk device, v: ESX disk VM).
3. Press f when viewing the component you do not want to capture.  A list of fields will be displayed.  You can toggle the fields on and off by pressing the letter corresponding to each field.  An * indicates that the field is on.  You want to turn off all of the fields you don’t want to collect.
4. Repeat steps 2 & 3 for the remaining components, leaving only what you want to capture.
5. Switch to the component you want to capture in batch mode and repeat step #3, except you will now enable what you want to capture.
6. Press W (capital W – case sensitive) to write out the ESXTOP configuration file.  You can accept the default or create new configuration files.  You may want to create a CPU-only config file, memory-only, and so forth.
7. Press CTRL+C to stop ESXTOP.
8. Now, invoke ESXTOP in batch mode, calling your updated or new configuration file you created in step #6 using the -c switch.  Here’s an example:# esxtop -b -d 30 -n 480 -c .esxtopcpustats > /tmp/esxtop_cpu_stats.csv where .esxtopcpustats is an ESXTOP config file with only CPU stats.  -d sets your capture interval to 30 seconds, and -n sets the number of samples to 480 (or 4 hours with a delay of 30 seconds).

Once your capture is complete you can replay the sampling in ESXTOP using replay mode (-R), or you can copy the .csv to a Windows system and use PERFMON or Excel to analyze the stats.  If using PERFMON or Excel you will notice that the system summary information displayed at the top of an interactive ESXTOP session is included in the output (console memory, console cpu, etc.).  As far as I know, there is no way to disable this, nor would you want to as it includes the time stamp necessary to interpret your data.

It is possible to use the vSphere CLI or the vSphere Management Assistant (vMA) to run RESXTOP, a version of ESXTOP designed for remote administration of ESXi or ESX.  You may note, however, RESXTOP from the vSphere CLI only works from a Linux client.  Using either of these tools will help you to automate ESXTOP statistics collection from multiple hosts using customized configuration files.

VMware vCenter collects performance statistics, tasks and events for historical performance analysis and auditing.  The collection level and retention of performance statistics can be controlled through the vCenter GUI (see Administration | vCenter Server Settings | Statistics).   The level of statistics collection and retention periods can have a dramatic impact on your vCenter Server’s performance if not carefully planned and monitored.  In particular, the vCenter database can grow quite large and the database server required to support the increase in statistics increases in size and performance characteristics (increased disk IO capacity, CPU, and memory).  Fortunately, VMware has provided a vCenter database sizing tool within the vCenter client (see picture).  This is all well and good for initial sizing, and my experience shows that vCenter’s sizing estimates are fairly accurate assuming the environment remains healthy.

I recently migrated an environment from vCenter 2.5 to 4.0 and in the process switched from a Windows 2003 32-bit vCenter host and a SQL 2005 server (remote to vCenter) to a Windows 2008 64-bit vCenter server with a SQL 2008 server (again, a remote SQL server).  I experienced a few issues during the migration and thought I had worked through them all (I’ll post on those at a later date).  However, after a bit of time I found that performance statistics for objects in the vCenter were missing of not rendering at an acceptable pace.  Upon further investigation, I discovered warnings in the vCenter Service Status node indicating that performance rollups within the vCenter database were not taking place.

In a SQL-backed vCenter, statistics rollups are handled by the SQL Server Agent (note: if you are using SQL Server Express, statistics rollups are handled by vCenter itself as SQL Express does not offer SQL Server Agent jobs).  KB 1003570 describes this process (it applies to vCenter 2.5, but the principles in it can be applied to 4.0).  To troubleshoot and resolve the issue I opened SQL Server Management Studio and checked several items:

1. Is the SQL Server Agent running?
2. Are there statistics rollup jobs defined for SQL server agent?
3. Are those jobs running?

In my case, the SQL Server Agent was running (you are prompted to configure this during the vCenter install).  However, when I checked for the presence of rollup jobs, I discovered that only a Past Day job had migrated with the database to the new SQL server.  Upon investigating the job history for that job I discovered that the job had not run since the migration (note to self: add these checks to your standard vCenter migration checklist).

To remediate the problem I completed the following steps:

1. Remove the bad ‘Past Day stats rollupVirtualCenter’ job from the list of SQL Server Agent Jobs.
2. Recreate the three standard stats rollup jobs.  To recreate the jobs, find SQL scripts on your vCenter server in C:\Program Files (x86)\VMware\Infrastructure\VirtualCenter Server.  The .sql scripts you’ll need are stats_rollup1_proc_mssql.sql, stats_rollup2_proc_mssql.sql, and stats_rollup3_proc_mssql.sql.  Run these scripts in SQL Query Analyzer against your VirtualCenter Database in order from 1 to 3.  These scripts should create the rollup jobs and their associated stored procedures (this procedure is detailed at http://communities.vmware.com/thread/123715).
3. After recreating the jobs I took a backup of the vCenter database.  The Past Day job soon kicked off to begin a stats rollup (this runs every 30 minutes by default).

I checked the server several hours later and discovered that rather than completing successfully, the Past Day job was still running and the drive holding my vCenter database transaction log was full.  Back to the drawing board..

1. I disabled the Past Week and Past Month rollup jobs to avoid job conflicts.
2. I backed up the vCenter database and then performed a shrink of the log file to get it back down to size.
3. The vCenter was running as a VM, so I was able to quickly increase its disk size and use diskpart from within the guest to extend the partition.  The space required to process weeks of performance statistics is not included in the vCenter Database Sizing tool as it is assumed that the rollup/purge jobs will run as designed.

I wanted to see how bad the problem was before kicking off another job so I ran:

select count(*) from vpx_hist_stat1

against the vCenter database in SQL Query Analyzer.  The query ran for several hours (never a good sign) and eventually returned well over 20 million rows of performance statistics (thanks to http://communities.vmware.com/message/1318736 for pointing me in this direction).  I investigated options to truncate the tables (see above link), and also looked at a script from VMware KB 1000125: Purging old data from the database used by vCenter Server.  In the end, I decided to try to let the Past Day stats job run.

I stopped the vCenter  Server Service to prevent new statistics from being written to the database.  I also disabled the Past Week and Past Month SQL Agent jobs to prevent job conflicts and then manually started the Past Day job.  I had to stop the job several times as it filled the 100GB transaction log volume.  A backup & shrink operation gave me back the space on the log volume.  I saw about 300GB of transaction logs written over the course of this process, but the Past Day job eventually completed.

Finally, I re-enabled the Past Week and Past Month jobs and manually ran both of them (Past Week first, then Past Month), followed by a backup and shrink of the vCenter database.  I was impressed with the performance increase I saw in the vCenter client.  Lists and performance graphs rendered much faster than when stats rollups were not taking place.

It would be a good idea to include checking stats rollup job status and a count of rows from the vpx_hist_stat tables in the vCenter database in your regular maintenance tasks.  For other vCenter Database best practices, check out breakout session PO2061 from VMworld 2008.  If you did not attend or subscribe to VMworld, Scott Lowe covered the session in this post.  A VMworld 2009 “online only” session entitled VM3237 vCenter Databases: Setup, Management and Best Practices was also offered (subscription required).  I have not viewed this session so I cannot comment on its content.

Microsoft published a document named “Getting to Know Hyper-V: A Walkthrough from Initial Setup to Common Scenarios” last week.  According to Microsoft, “this guide provides detailed step-by-step walkthroughs for testing Hyper-V on a pre-production environment. You can use this guide to become familiar with Hyper-V and the process of creating and managing virtual machines. Also included in this guide are useful scenarios that you can test to better understand how Hyper-V can address the business goals of your organization.”  The document serves as a sort of evaluators guide for Hyper-V, stepping the reader through everything from enabling VT in BIOS through virtual networking.  It also includes some sections on using snapshots, base virtual machine templates, and managing Hyper-V based virtual machines remotely with Hyper-V Manager.  If you want more in-depth documentation on Hyper-V you can go through http://technet.microsoft.com.

As a side note, Microsoft has published the Microsoft Manual of Style for Technical Publications to help standardize technical documentation.  I have long been a fan of Microsoft’s technical documentation for its easy to read style, although it sometimes lacks the depth that I desire.

While we’re on the topic of virtualization documentation, I have also been quite pleased with VMware’s technical documentation over the years, and have found it to be continually increasing in quality, providing very specific technical guidance and references to additional resources.  I have also been pleased to see that VMware has improved delivery options for documentation.  VMware offers several formats for documentation delivery, including web-based and PDF’s.  Start with the Documentation Roadmap for a quick introduction to the available documentation, and where to find what you need.

You can find web-based vSphere documentation here: http://pubs.vmware.com/vsp40/.   The web-based documentation is great for running searches on.  All vSphere documentation can be accessed through this page: http://www.vmware.com/support/pubs/vs_pages/vsp_pubs_esx40_vc40.html.  If you want to do a full grab of all of VMware’s documentation for an in-house repository (e.g. SharePoint), check out xtravirt’s VMware Documentation Downloader script.

If you are looking for quick and easy evaluator guide-type documentation from VMware, check out these resources: ESXi Installable and vCenter Server Setup Guide and the Virtualization Kit (registration required) at http://www.vmware.com/resources/wp/virtualization101_register.html.

There is a ton of less formal VMware documentation in several places:

• Technical resources and case studies here: http://www.vmware.com/resources/techresources/
• Proven practices around Strategy, Applications, Security, Management, and Availability at VIOPS.
• Official VMware Blogs at http://www.vmware.com/vmtn/planet/vmware/.
• Community blogs aggregated by VMware at Planet v12n: http://www.vmware.com/vmtn/planet/v12n/
• VMworld Recorded Sessions & Labs (VMworld 2009 Sessions available as of today, September 14th) at http://vmworld.com.
• The VMware Community Forums: http://communities.vmware.com/
• And, 3rd party books like Scott Lowe’s Mastering VMware vSphere 4.

Do you have other sources of virtualization documentation or easy methods of searching documentation to find exactly what you need when you need it?  If so, leave a comment!

I needed to grab some stats from my ESX hosts for off-line analysis so I fired up my trusty ESXTOP intent on using batch mode to capture a .csv formatted output.  I started to manually select the counters I was interested in while working in ESXTOP interactive mode (you can save your selected counters to the esxtop configuration file with the ‘w’ command) and thought that there must be a better way.  I found that better way in the VMware Performance Community: http://communities.vmware.com/docs/DOC-3930.  There is now a -a switch that can be used to include ALL performance counters.  I’m sold.

I wanted detailed information, so I decided on a 15 second capture interval to run for a 2 hour window.  Here’s the command I used:

esxtop -a -b -d 15 -n 480 > /tmp/esxtopout.csv

where -a is for ALL, -b is for batch mode, -d is for delay, and -n is for the number of iterations ((60/15)*60*2).  I wrote out the results to a .csv in /tmp.  The resulting CSV weighed in at a whopping 100MB after 2 hours.

The CSV can be analyzed in Excel (pivot tables work well for this) or in Windows Perfmon.  I opened the log in Perfmon as I was after basic Min/Average/Max counters and Perfmon makes those easy to see.  When adding the CSV log to Perfmon, you are prompted to select counters.  I added all instances of Commands/sec, Reads/sec, and Writes/sec from Physical Disk (I was gathering some IOPS counts for a new storage proposal). I got a bit more than I bargained for: a mostly unresponsive Perfmon window and the ugliest darn graph I’ve ever seen.

Switching from a graph view to the report view allows you to easily view and remove specific counters that you are not interested in, or open the Properties of the data set, switch to the data tab and bulk select counters that you want to remove.  I was not interested in vmhba1:x, specific VM’s or worlds, so I killed all of those, leaving just the base iSCSI device (vmhba32 in my case).

After some cleanup the graph looked a bit better and more importantly, I was able to easily read my Min/Average/Max stats:

Here are the takeaways -

• ESXTOP is a powerful utility for performance monitoring
• All stats (-a) can result in a huge file – use it wisely in batch mode; else use interactive mode to select your counters and write them to the user-defined configuration file.  Invoke the config file with the -c option when running in batch mode.
• Consider using vscsiStats for more granular reporting.
• ESXTOP physical disk stats do not include NFS volumes.

Do you use other tools or methods to collect basic disk IO counters for storage sizing purposes?  If so, leave a comment describing your approach!

I have been meaning to write this up for a while; Scott Drummonds’ ‘Love Your Balloon Driver’ post today at his Virtual Performance blog gave me a nice reminder.  I actually caught a sneak peak at the graphs with an explanation from Scott at his instructor-led lab at VMworld 2009.  Scott calls out that the only workload they discovered suffers from balloon driver activity is Java.  The reason for Java’s problems with balloon driver activity is that Java itself runs in a VM and so the guest OS cannot properly determine which pages should be swapped out when the balloon driver calls for it.

My experiences causes me to agree with Scott and the whitepaper he cites – in a properly designed and equipped environment the balloon driver is not detrimental for most every workload to a point.   However, I recently discovered in a client site that the balloon driver can cause significant issues when the environment is poorly designed and under-sized.  Here the background:

I was called into an already established environment where the client was running on an older blade with VMware ESX 3.5.  The blade maxed out at 16GB RAM and had dual dual-core CPU’s with no hope for an upgrade.  On the blade was a single guest VM running Windows 2003 with SQL 2005, in it’s full 32-bit glory.  The VM was configured with 4 vCPU’s and 16GB of memory.  Some of you can probably already guess where this is going….

The x86 Windows guest had PAE configured, and SQL took advantage of AWE to use the additional memory beyond the 4GB limit of a 32-bit system.  Additionally, the Windows guest had the /3GB switch enabled in boot.ini.  Finally, as per SQL best practices, the ‘Lock Pages in Memory‘ permission was granted to the SQL Server service account.  What the guest was left with was 1GB of kernel mode memory and 15GB of User Mode/Extended addressable memory.

And here’s the problem.  The client was using ESX, not ESX 3.5, so the Service Console required memory.  In this case, the service console had approximately 512MB allocated to it.  Futhermore, VM’s require some overhead on ESX to run.  The memory overhead consumed by a Windows guest on ESX 3.5 with 4 vCPU and 16GB of memory is a bit more than 512MB.  On a properly sized ESX server with multiple similar guests/workloads, you could probably gain much of the overhead back through transparent page sharing; but in this case I had a 1:1 P2V ratio.  If you are any good at math you see that the environment is running about 1GB short of memory.  A quick check of the balloon driver stat in vCenter show that the balloon driver was constantly active and demanding about 1GB back from the guest… constantly.

Under normal circumstances this might not be an issue, but in this case the Windows guest was being absolutely punished.  The guest CPU’s were pegged at 100% with an excessive amount of kernel time, often indicating IO issues.  And indeed I did experience terrible disk and network performance on the guest.  At the root of the problem is this – the Lock Pages in Memory permission allows SQL to get a firm grasp on the user mode memory available to it (15GB) and lock it up.  This left the already starved (because of the 3GB switch in the boot.ini) guest kernel with it’s 1GB the only thing the balloon driver could really swap out.

The client suggested a reservation of 16GB on the VM, knowing that memory reservations prevent balloon driver activity.  I calmly asked them to back away from the keyboard as I explained how if a starved guest was bad, how much worse a starved Service Console would be.  In the end the fix was quiet easy – I convinced the customer that they should reduce the amount of memory allocated to the guest by about 1GB, enough to let the 512MB SC and the 512MB of overhead run without contention.  I was able to show them the difference between allocated and active memory in vCenter – the 1GB being surrendered was not really being actively used, SQL just had it locked up.  In fact, surrendering the 1GB of memory back to ESX breathed new life into the guest VM, bringing its performance back in line with expectations.

Ideally, I would have brought in a bigger ESX server that could serve additional VM’s, driving greater levels of efficiency across the environment.  It just wasn’t an option for the client in this case.  In the end, the problem was fixed and I was reminded just how fun it can be to explain some of these backwards sounding virtualization concepts to customers – fewer vCPU’s can lead to better performance of guests, less guest memory can fix performance issues, and increasing the quantity of similar guests on a host can drive better performance to a point because of transparent page sharing.

Stay tuned over the next few weeks as I digest and write on my VMworld experience – from VMUG activities to Paul Maritz’s press conference announcing the vCloud Express, and plenty of great sessions in between.  Like many of you, I returned from VMworld with quite a backlog of work but I’ll do my best to squeeze in some posts and tweets.

I have been pulling my hair out with a small VI3 implementation running against an IBM DS3300 iSCSI array.  Performance, for lack of a better term, sucked.  Granted, the DS3300 is not an enterprise level workhorse of a storage system, but it fit the budget.  Read performance was decent from the array, but write performance was terrible, maxing out at 10Mpbs throughput and insanely high latencies on long writes when the system was under load.  This led to some long P2V operations, poor guest performance, and some questions from the project sponsors on why I couldn’t make the environment sing.

The system was configured with a single controller with dual GigE NIC’s.  The controller had 512MB of battery backed cache (there is also a 1GB cache upgrade option available).  I wrote off some of the poor performance to a single controller with a less-than-optimal amount of cache; blamed the SAS controller to SATA disk command translation overhead; cringed at the 6 disk RAID5 configuration; and engaged in some self doubting.  I convinced the powers that be that we were IO constrained and got some funds to fill out the 3U chassis to a full 12 SATA disks, and reconfigured the array as a RAID10.  Performance gains were almost unnoticeable with these changes.  In addition, I did some basic troubleshooting of the network environment, verifying multiple paths to the storage, setting Flow Control on the switches to receive only, and double-checked my iSCSI initiator settings.  Note: The DS3300 is only supported with the ESX software initiator.  I found documentation on the DS3300 to be lacking, but did discover that the Dell MD3000i is based on the same LSI Engenio array.  Some Googling on the Dell solution led to to the ‘SMcli’ command line interface for both arrays.   The commands are slighly different for the Dell and IBM.  The links to the IBM CLI documentation were broken, so I had to do a bit of trial and error to get the commands right.  I used the Dell documentation as a starting point.  (Rant: Seriously, IBM?  Can you make your documentation any harder to get through – is it a Redbook, is it an Engineering Whitepaper, is it a support document, is it a case study – and why can I only find these with complex Google searches, not on your own product pages, and why can’t you name for documents intelligently, not with some random string of characters).

Update: The IBM System Storage DS3000, DS4000, and DS5000Command Line Interface and Script Commands Programming Guide is here: IBM System Storage DS3000, DS4000, and DS5000Command Line Interface and Script Commands Programming Guide – DS3k4k5kCLIreference, SMCLI

Moving on… I received an automated alert from the DS3300 about an incomplete battery learn cycle.  Using the IBM Storage Manager GUI I generated a  Storage Subsystem Profile’ from the Support tab to check the battery status.  In the profile I discovered that while write cache was enabled, it had a status of “Enabled (Suspended)”.   Ah ha!  Now I’ve got some decent Google material that led me to this: http://communities.vmware.com/thread/195838.  Hot damn I love the VMware Community Forums!

It turns out that in a single-controller configuration the setting for cache mirroring remains enabled by default.  Because there is no 2nd controller to mirror to, the array suspends write caching.  This is probably a safety thing – loss of high availability on the controllers puts data in cache at risk should the only controller fail.  I weighed my options and decided that the poor performance I was experiencing beat HA concerns, so I enabled write cache on the array using this command:

c:\program files\ibm_ds4000\client>smcli -n <ARRAYNAME> -c “set allLogicalDrives mirrorEnabled=false;”

And then followed with this for good measure:

c:\program files\ibm_ds4000\client>smcli -n <ARRAYNAME> -p <arraypassword> -c “set allLogicalDrives writeCacheEnabled=true;”

The results were immediately noticeable:

DS3300 Performance Improvement when Write Cache is Enabled - Click for a Larger View

The screen shot is from Veeam Monitor Free Edition, taken during 4 concurrent V2V operations from Hyper-V to VMware.  With the write cache fully functional, disk usage peaked at 54MBps, latency dropped to about 6ms, and my blood pressure dropped a few notches.

While poking around the CLI I also found that you can dump performance stats from the array (performance is otherwise hard to find on the thing) using this command:

C:\Program Files\IBM_DS4000\client>smcli -n <ARRAYNAME> -c “set session performanceMonitorInterval=5 performanceMonitorIterations=120;save storageSubsystem performanceStats file=\”c:\\ds3300perfstats.csv\“;”

This will give you a 10 minute record of performance from the array which you can analyze using Excel.  The Dell Enterprise Center TechCenter Wiki has a great write-up on how to efficiently analyze the data from this command here: http://www.delltechcenter.com/page/MD3000i+Performance+Monitoring, complete with a YouTube video that walks you through the process:

I am beginning to think that the DS3300 (and MD3000i) may actually be a viable starter solution for SMB’s starting out on a virtualization project.  But I would recommend the cache upgrade, 2nd controller, SAS disks instead of SATA to eliminate the SAS-to-SATA translation overhead and more faster disks instead of fewer slower disks so you can drive throughput and IOPS to a higher level.

Have any of you deployed the DS3300 or MD3000i (or the generic LSI solution)?  Do you have any performance tuning tips for these arrays?  If so, share in the comments!

Here’s the scenario:

After performing maintenance on an ESX server (patches, storage re-scan, reboot), VMFS volumes are no longer visible, even though the hosting LUN can be seen on the Storage Adapters page of the ESX Configuration tab.  Most VMware administrators will see this play out at some point; I saw it in one of my environments today and figured I should make a note of the steps required to correct the issue.

Typically, the root cause of the issue is a change on the storage array that causes the h(id) of the LUN(s) in question to change.  This change could be anything from an array firmware update, LUN removal/recreation, or RAID/LUN reconfiguration.  These changes could cause the h(id) of the LUN to be updated.  When a rescan takes place on the ESX storage adapters (through a manual instantiation, reboot, etc.), the new h(id) is observed.  Because it does not match the previously observed ID, the LUN is tagged as a snapshot LUN and access to that LUN is disabled.

Diagnosis of this problem is fairly easy.  In addition to the behavior I have described, as observed through the Virtual Center Client, the problem can also be confirmed through the ESX command line.

To diagnosis this issue from the console, view the vmkernel log by issuing the following command: tail -f /var/log/vmkernel

You will see messages in the log similar to the following:

Jun  2 16:01:29 esx04 vmkernel: 0:00:31:14.543 cpu3:1039)ALERT: LVM: 4482: vml.0200020000600a0b80005add7800000a494a1d0be6313732362d33:1 may be snapshot: disabling access. See resignaturing section in SAN config guide.
Jun  2 16:01:29 esx04 vmkernel: 0:00:31:14.552 cpu3:1039)LVM: 5579: Device vml.0200010000600a0b80005add7800000a474a1d0bc8313732362d33:1 detected to be a snapshot:
Jun  2 16:01:29 tccesx04 vmkernel: 0:00:31:14.552 cpu3:1039)LVM: 5586:   queried disk ID: <type 2, len 22, lun 1, devType 0, scsi 6, h(id) 5103533129706062046>
Jun  2 16:01:29 esx04 vmkernel: 0:00:31:14.552 cpu3:1039)LVM: 5593:   on-disk disk ID: <type 2, len 22, lun 1, devType 0, scsi 6, h(id) 2153359415130143165>

After confirming that this is indeed the problem you are experiencing, stop and take a deep breath.  The fix is easy, but you need to take steps before fixing it to prevent further damage.  If you are lucky, the problem has only manifested itself on one ESX server (and hopefully that ESX was not hosting any VM’s because you put it into maintenance mode).  Prevent your other ESX servers from rescanning storage – don’t reboot them, don’t manually rescan, don’t update them.

If the affected ESX server was hosting running VM’s, HA (if licensed and properly configured) should have kicked in if applicable and restarted the VM’s on another node in the ESX cluster.

If multiple ESX servers (or all of them) are affected, your VM’s are likely all powered off after hard stops, so there is not much you can do but to get on with fixing the issue and trust your backups (you do have backups, right?).  This is where array-level snapshots come in handy.  In my experience, most if not all VM’s recover after a hard stop like this, but don’t let that keep you from having a robust DR plan.

To correct the issue you must not have any running VM’s on the affected VMFS volumes to alternate volumes.  Shut down the VM’s or use Storage VMotion to move running VM’s to alternate LUN’s.

In the VI Client, select the affected ESX host in the Hosts & Clusters view.  Switch to the Configuration Tab.  Click ‘Advanced Settings’ and then choose the LVM node.  Change the LVM.DisallowSnapshotLun from the default setting of ’1′ to ’0′ and click OK.  Next, rescan your storage from the ‘Configuration | Storage Adapters’ pane.  Your missing VMFS volumes should re-appear.  You’re doing fine, but not done yet.

Even if the other hosts that use the affected VMFS volume appear to be fine, they will most likely lose access to the volume once a rescan/reboot takes place.  You need to perform the LVM.DisallowSnapshotLun = 0 setting change on all ESX servers connected to the volume, followed by a re-scan of your storage.

Once all affected ESX servers see the VMFS volumes, change the LVM.DisallowSnapshotLun setting back to the default of 1.  Migrate back and/or power up VM’s on the volume and see what the damage is.  If you are lucky, everything is good to go.  If not, it’s a great time to check out those backups.

If you do not know what caused the storage change, check your ESX logs to try to determine if the server was rebooted or if storage was rescanned. This will give you an idea of when the change occurred – a starting point to work back from to find the root cause.  Use this command to get started: less /var/log/vmksummary

Here are some suggestions on how to avoid this problem:

1.) Minimize changes to LUN’s once configured on an ESX.

2.) Coordinate Storage Maintenance with VMware maintenance windows.

3.) Have stand-by storage so you can Storage VMotion running VM’s off of the affected LUNS.

4.) Consider NFS, as NFS volumes are not impacted by resignaturing.

For more information on this problem, or to better understand the advanced settings changes involved, check out the VMware SAN Configuration Guide at http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_3_server_config.pdf, page 114, or the VMware iSCSI SAN Configuration Guide at http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_iscsi_san_cfg.pdf, page 117.