VMtoday

VMware News, Views, and How-To's from vExpert Josh Townsend

You are here: Home / Archives for performance

performance

What You Didn’t Know You Needed To Know Before Implementing VMware View

by Leave a Comment

Clearpath Sponsors VMUGI had the privilege of presenting at the Potomac Regional VMware User Group (VMUG) conference yesterday and at the inaugural Boston VMUG last week.  The Potomac Regional event was a combination of Washington DC, Northern Virginia, Maryland, and Federal VMUG’s.  Clearpath participated in both the Potomac and Boston events as a gold sponsor.  My session was entitled, “What You Didn’t Know You Needed To Know Before Implementing VMware View”.

My session provided attendees with an overview of how VMware View integrates with existing infrastructures, Windows images, applications, and processes.  Building on Clearpath’s experience in implementing VMware View for customers of many sizes in a variety of industries, we looked at many of the issues that can cause a View pilot or production implementation to get off track.  Prerequisites and best practices for the pieces of your environment that touch View were reviewed, including Active Directory, Distributed File System (DFS), Group Policies, Load Balancers, ThinApp packaging, and base image optimization.  Finally, we discussed how to integrate the several facets of your IT group that will have to be involved in your View roll-out: application, desktop/helpdesk, network, storage, and infrastructure teams.

The presentation is embedded below.

If you have questions, comments, or need clarification around any of the items covered in my presentation please don’t hesitate to ask!

VMware View Performance Analysis of Logs (PAL) Utility

by 2 Comments

Introducing a new free tool for doing performance analysis on VMware View environments.

Background

I’ve been using the Performance Analysis of Logs (PAL) tool for years (since the first version released in 2007) to assist in analyzing, reporting, and visualizing performance logs in Windows environments.  The real power of PAL is the built-in threshold files that interpret Perfmon counters, providing an explanation of what the counter is, why it matters, and thresholds for warnings and critical to help quickly zero in on performance bottlenecks.  PAL includes templates for Windows, Hyper-V, SQL, Active Directory, IIS, Exchange, .NET, and SharePoint.

I’m often called in to help customers conduct performance analysis and troubleshooting within their VMware View environments.  There are several great tools available for on-going monitoring and alerting in View environments – VMware offers vCenter Operations for VMware View, Xangati makes a good product, and there’s a free PCoIP Log Viewer from Chuck Hirstius at https://mindfluxinc.net/?p=367.  But some customers do not have budget for these products or have an immediate need to do spot troubleshooting for a particular VM, or have problems that extend beyond PCoIP.  I thought why not combine the power of PAL with some VMware View specific data to help analyze Perfmon logs in a View environment?

PAL Report Showing CPU Ready in a View Desktop

A Microsoft engineer, Jeff Stokes, wrote a very simple threshold file for VMware Tools counters for VMware guest VM’s.  It was a great start, but I found it didn’t provide enough detail and didn’t include PCoIP statistics for VMware View desktops.  I used Jeff’s template for inspiration and built my own, including VM Processor, VM Memory, and PCoIP Session statistics perfmon counters, along with explanations of each counter, links to resources for each counter, and ideas for remediating performance problems identified with counters that exceed expected thresholds or are suspected of being a bottleneck.  Windows System stats are included in PAL and are included in the PAL report my threshold file generates.  All told, there are 122 different Windows Performance Monitor (perfmon) counters included in my VMware View Desktop PAL report.

I also built a threshold file that can be used to analyze VMware View Connection Servers and Security Servers.  The standard PAL tool with Windows System Overview thresholds can be used to analyze VMware vCenter Server and VMware Composer servers.  All combined, you can use PAL to analyze all components of your VMware View environment, with the exception of Thin or Zero clients.

The full reports can be information overload – you’ll not use it for day-to-day monitoring.  Rather, use it for a deep analysis for specific VM’s with performance issues.

This is my first version of the VMware View specific thresholds.  I plan to add more analysis, calculated thresholds, etc. in future versions.  Also, stay tuned for a threshold file that is capable of analyzing ESXi ESXTOP batch mode output.

More details, usage instructions, and the PAL Threshold files for download are on the permalink page for this tool here: https://vmtoday.com/scripts-free-tools/view-pal/.  Watch this page for updates to the tool.

As always, I am open for constructive feedback on this tool.  Let me know if it is useful to you, needs some tweaking or tuning, or if you have ideas for cool new features.

Quick tip #1: You can use the View Desktop template to analyze any Windows-based VM, not just View desktops.  In this usage scenario, the PCoIP counters will be ignored.

Quick Tip #2: You can edit my PAL Template file and comment out the inheritence of the SystemOverview.xml file if you just want to collect and report on my VMware counters.

Automate

My fellow vExpert, Jonathan Medd, has a great script for deploying Perfmon Data Collector sets based on an XML template on his blog.  Use this to deploy my template to a batch of View desktops, all Connection Servers, etc.  Find Jonathan’s script here: www.jonathanmedd.net/2010/11/managing-perfmon-data-collector-sets-with-powershell.html

Ryen Tang takes a slightly different approach to deploying Perfmon data collectors – his approach will work with a bit of modifications for my counters.

Laerte Junior has another PowerShell approach for gathering Perfmon data from multiple remote computers here: https://www.simple-talk.com/sql/database-administration/gathering-perfmon-data-with-powershell/

Sample Reports

A quick screenshot of a small part of a View Desktop PAL report is to the right.  Full reports from my lab environment are linked below (warning – these reports are pretty massive pages and will take a bit of time to load, and the data is representative of a lab, not a production environment).  The VMware specific parts of the report will be at the bottom as PAL chains threshold files together, starting with the basic Windows system overview counters.

View PAL Desktop Sample Report

View PAL Connection Server Sample Report

Lag When Dragging a Window Between Monitors in VMware View

by 12 Comments

Multi-Monitor VMware ViewI’ve had several customers ask me for help with the VMware View environment recently, as their users were complaining about performance issues when dragging a window between monitors in a multi-monitor VMware View setup.   Performance was otherwise acceptable on the View desktops.  I recommended several troubleshooting and remediation steps to get windows to stop slowing down when dragging between monitors or “catching” on the border between the two screens, including:

  1. Review the “Multimonitor PCoIP sessions are not displayed as expected KB article” (1030566) from VMware.
  2. Monitor PCoIP statistics (using PerfMon/WMI data, vCenter Operations, the PCoIP Log Viewer utility, or Xangati).  Great data collected, but no problems found.
  3. Tweak PCoIP settings with the PCoIP Group Policy and the PCoIP Configuration Utility.  Again, fine tuning done, but no change in the problem.
  4. Collecting Diagnostic Information from the View Agent, View Client, and View Connection Server.
  5. Review the Symptoms and Resolution steps in VMware KB 1027899: Dual monitors configured using PCoIP does not span both the monitors for VMware View.
  6. Increase video RAM for View desktops in the pool.
  7. Increase client cache size setting in VMware View PCoIP Group Policy Objects (GPO)
  8. Upgrade thin client/zero client firmware.
  9. Upgrade to View 5.1 for PCoIP improvements (and all the other great enhancements in 5.1).  This takes a bit of planning, so not reasonable as a quick fix.

These steps either yielded no results or were not easy to do without some planning and change control.

Then I stumbled on VMware KB article 2010359, “Poor performance when moving windows between multiple displays in View 5.x session.”  The article was just updated yesterday (7/25/2012) to include View 5.1 as an affected product.

From the KB article:

This issue occurs due to the way View handles the transition of a window from one display to another. It relies on the MKS poll rate which is optimized for server configurations to reduce CPU cycles.

This issue is limited to this configuration:

  • Windows 7 View virtual machines (both 32bit and 64bit)
  • Virtual machine hardware version 7 or 8
  • View Agent 5.0
  • Client desktop utilizing multiple displays

For those who don’t know, MKS is:

Mouse Keyboard Screen (MKS) process – A process that is responsible for rendering the guest video and handling guest operating system user input.

See VMware KB 1019471 for more on MKS, VMM, and VMX processes, as well as how to interpret error for all of these services.

The fix is to enter an advanced configuration parameter on the affected virtual machine as follows:

  1. Power off the virtual machine using the vSphere Client.
  2. Right-click the virtual machine and click Edit Settings.
  3. Select the Options tab and under Advanced.
  4. Click General.
  5. Click Configuration Parameters and click Add Row
  6. In the Name field enter mks.poll.headlessRates and in the Value field enter 1000 100 2.
  7. Click OK.
  8. Power on the virtual machine.

When the virtual machine is powered on, it will read the new configuration.

Pretty easy, no?  Not so fast….  The fix requires downtime, for all your View desktops.  And, if you are using VMware View Linked Clones, the change you make to any of the clones will be lost when the VM’s are refreshed at log off, rebalanced, or recomposed.  To get the fix to stick, you need to update the template on which your pool is based on then recompose the pool.  If a recompose is not in your immediate future, you may have to tell your users to wait it out.

If you have a bunch of full desktop pools, dedicated linked clones, or manual pools in View that you want to make this change on, today is your lucky day.  I put together a PowerCLI script to automate the change.  I borrowed from the script from Arne Fokkema to fix the storage vMotion “migration exceeded the maximum switchover time of 100 second(s). ESX has preemptively failed the migration to allow the VM to continue running on the source host” error.

The script should be applied with all target VM’s powered off.  That said, I ran it in the Clearpath lab against a running pool of View Linked Clones and had the .vmx file updated.  The VMware KB clearly states that “When the virtual machine is powered on, it will read the new configuration.”  From my testing, the setting did not seem to take place on the running VM’s until I rebooted them – but your mileage may vary.

Also note (per the VMware KB article 2010359) , there is a risk in making this change to your View desktops – CPU utilization on idle VM’s can increase.  Less idle time means higher CPU usage on your ESXi hosts/cluster and increased CPU Ready in your environment.  I did not see an increase in my environment, but the lab is small.  Test, test, backup, test some more, and backup again before making the change or running my script.

mks.poll.headlessRates CPU Graph

The script is below.  Update the vCenter server variable and the name of the vCenter folder (not the View Administrator folder) that contains your View desktops.

@"
===============================================================================
Title:            Fix-ViewWindowDrag.ps1
Description:    Update mks.poll.headlessRates for improved VMware View
performance    and reduced lag when dragging Windows between
desktops in a multi-monitor PCoIP environment. See VMware
KB 2010359 for more info.
Warning: may increase CPU on View ESXi Hosts!!!
Last Updated:     July 26 2012
Created by:        Josh Townsend, https://vmtoday.com
Usage:            Update variables.  Run .Fix-ViewWindowDrag.ps1 in PowerCLI
===============================================================================
"@
$VMFolder = "View-Desktops"    # Folder where View VMs have been placed - Must be unique in VC
$vCServer = "vcenter.local"    # Name or IP of VMware vCenter Server hosting View Desktops

# Connect to VC
Connect-VIServer -Server $vCServer

# Get all VMs placed in $vmfolder
$vms = Get-VM -Location (Get-Folder $VMFolder)

$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

$extra = New-Object VMware.Vim.optionvalue
$extra.Key="mks.poll.headlessRates"
$extra.Value="1000 100 2"

$vmConfigSpec.extraconfig += $extra

foreach($vm in $vms){
$vm.ReconfigVM($vmConfigSpec)
}

# Disconnect from vCenter Server
Disconnect-VIServer -Confirm:$False

Here’s a copy of the PS1 file: Fix-ViewWindowDrag

I also have a copy of the script that will target just a single VM – good for testing or those few full VM pools you might have:

@"
===============================================================================
Title:            Fix-ViewWindowDrag-single.ps1
Description:    Update mks.poll.headlessRates for improved VMware View
performance    and reduced lag when dragging Windows between
desktops in a multi-monitor PCoIP environment. See VMware
KB 2010359 for more info.
Warning: may increase CPU on View ESXi Hosts!!!
Last Updated:     July 26 2012
Created by:        Josh Townsend, https://vmtoday.com
Usage:            Update variables.  Run .Fix-ViewWindowDrag.ps1 in PowerCLI
Note: This fixes a single VM
===============================================================================
"@
$VMFolder = "Windows7"    # Folder where View VMs have been placed - Must be unique in VC
$vCServer = "192.168.1.111"    # Name or IP of VMware vCenter Server hosting View Desktops
$VMName = "MyLaggyViewVM"      #Name of the View desktop that you want to tweak

# Connect to VC
Connect-VIServer -Server $vCServer

# Get all VMs placed in $vmfolder
$vms = Get-View -ViewType VirtualMachine -Filter @{"Name" = "$VMName"}

$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

$extra = New-Object VMware.Vim.optionvalue
$extra.Key="mks.poll.headlessRates"
$extra.Value="1000 100 2"

$vmConfigSpec.extraconfig += $extra

foreach($vm in $vms){
$vm.ReconfigVM($vmConfigSpec)
}

# Disconnect from vCenter Server
Disconnect-VIServer -Confirm:$False

And the single VM .ps1 script download: Fix-ViewWindowDrag-single

Leave a commend below if you have questions, problems, or have a better way of implementing this!
This article is cross-posted on the Clearpath Solutions Group blog – follow me and my fellow VMware, EMC, and Cisco experts for news, tips, and tutorials.

Clearpath Solutions Group Logo

VMware vSphere Resource Pools – Resource Allocation Revisited

by 14 Comments

I have found VMware vSphere Resource Pools to be an often misunderstood element, and incorrectly implemented by even seasoned VMware administrators.  I recently found an example of an incorrect implementation of Resource Pools being used to organize VM’s in vCenter’s Hosts and Clusters view, without considering the potential performance impact of this configuration.  I thought that I might try to explain the resource pool piece a bit better, particularly around Shares.

First, a few technical points to remember:

  • Resource Pools can only be used when VMware Distributed Resource Scheduler (DRS) is enabled on a cluster.  DRS is available in vSphere Enterprise and Enterprise Plus editions.
  • VMware Shares are like stock/shares in a company –> Shares are proportional.  The more  shares you issue (on your company or in your VMware environment), the less valuable the individual share is.  The more shares you have, the greater your voting power/ability to claim a right to resources is.
  • The vSphere Hosts Cluster itself is technically a Resource Pool too.
  • Shares only come into play when there is resource contention (Memory or CPU).  If the host/cluster has spare resources available for a guest it allocates availability of those resources based on the configured vRAM size and vCPU count, taking into consideration any limits (Mem/CPU) on the VM.
  • Resource Pools can be used to apply resource Shares to VM’s, as well as Reservations and Limits.  Shares, Reservations, and Limits can also be set on a per-VM basis.  The values on Resource Pools and individual VM’s combined can do funny things.

Now, consider the following screen shot of the environment running on my laptop lab:

  vSphere Resource Pool Shares CPU Resource Allocation

I have 16 VM’s total, 4 in ResourcePool1, and 1 VM in ResourcePool2.  The resource pools are set to ‘Normal Shares’ with no other configuration changes to the default.  The VM’s are all the same size as far as vRAM and vCPU are concerned.

If we look at the ‘Resource Allocation’ tab of the Cluster, we see that the VM’s in the Cluster resource pool each have 5% of the Shares for CPU, and each ‘Normal’ resource pool has 21% of the total shares of the cluster.  If we divide the 21% shares by the 4 VM’s in ResourcePool1, we find that each VM gets 5.25% of the cluster’s total shares.  That means that the VM’s in this ‘Normal’ pool actually get a bit more CPU time under resource contention than those in the parent cluster pool.  The single VM (MoveVM1) in ResourcePool2 gets 21% of the shares all to itself – much larger than the remaining VM’s in the Cluster, even though it may not be a VM of higher importance in your environment.

Now consider Memory resource allocation:

 vSphere Resource Pool Shares Memory Resource Allocation

The ‘Normal’ shares value for memory on a resource pool is 163840.  Compare this to the 640 shares each single VM gets in my current config (VM-* each have 64MB RAM, so a very low share value per VM).  We’ve effectively configured only a fraction of 1% (basically 0% shares) to the VM’s in the parent cluster pool, while the few VM’s in the ResourcePool1 and ResourcePool2 get to share 48% of cluster resources per pool.  This still leads to an imbalance for the VM’s in each pool as compared to VM’s in the sibling pool and to the VM’s in the parent pool.  I.E. – MoveVM1 gets all 48%; while the four VM’s in ResourcePool1 share 48% (48/4=12% each).

Now let’s say that ‘MoveVM1’ was just a VM that a well-intentioned administrator spun up for some testing, putting it in a resource pool for ease of locating it in an otherwise busy vCenter view.  This otherwise inconsequential testing VM now has the right to claim 48% of the memory (vRAM) in the cluster and 21% of CPU time when there is resource contention in the cluster.  Now if this lowly testing VM is the cause of resource contention on an otherwise healthy cluster, it’s getting its cake and eating it too.  When the going get’s tough, the least valuable workload gets the priority in this configuration!  We’ve created a case where the meek (VM) shall inherit the resources.  All your resources are belong to us.  You get the idea – a big fat oops!

Hopefully you can see how using Resource Pools as units of organization can negatively affect the performance of an environment under resource contention, even if those resource pools are configured with ‘default/normal’ values.  The risk becomes more severe if the values of those resource pools were to change (say, move to a ‘High’ value for shares on ResourcePool2 in my lab).

Best Practices

  • It is best to use folders the ‘VM’s and Templates View’ in the vSphere Client for organizing VM’s and only (sparingly) use resource pools to achieve specific performance goals under resource contention (shares) and/or resource allocation goals (limits & reservations) under normal operating circumstances with resource availability.
  • Use the Resource Allocation tab to review and tune your Resource Pool (and individual VM) Shares settings.  Export the resource allocation list from vCenter and use Excel if you want to experiment with values and do what-if calculations.
  • Restrict permissions to create and modify resource allocation on VM’s and on Resource Pools in vCenter to only VMware Administrators who understand the implications of these changes, and who are capable of doing the math to properly design a resource allocation model.  If you don’t have an expert on staff, engage a consultant to help (might I suggest Clearpath Solutions Group ;-)).
  • Check out VMware vCenter Operations Manager (vCenter Ops/ vCOPs) for advanced monitoring and capacity planning (the product formerly known as VMware CapIQ is included with vCenter Ops).  Good monitoring and forecasting can help to avoid resource contention in highly consolidated, oversubscribed, or performance-sensitive environments (and what environments are not all three of these today?).
  • Review the VMware vSphere Resource Management Guide to better understand the use and function of Resource Pools. The Guide can be found here: https://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-resource-management-guide.pdf.
  • Buy  Duncan Epping’s and Frank Denneman’s book, VMware vSphere 5 Clustering Technical Deepdive to really understand how VMware vSphere Distributed Resource Scheduler (DRS) (as well as VMware High Availability (HA) VMware Distributed Power Management (DPM), and VMware Storage DRS) function and are best configured.

vSphere 5 Networking Bug Affects Software iSCSI

by 91 Comments

Update 1: This issue was fixed as of 3/15/2012 in ESXi 5.0 Update 1 per the original knowledge base article: VMware KB 2008144. To download ESXi 5.0/vCenter Server 5.0 Update 1, see the VMware Download Center. 

Update 2 (Nov 7 2012): While VMware says the issue was fixed in vSphere 5.0 Update 1, I have not found this to be the case.  I’ve had several environments where the bug was still present, and in some cases this nasty bug still appeared when implementing the Failback = NO setting on the vSwitch.  Additionally, setting Failback = No may cause some other problems – see my update here for information from Chris Towles.

Update 3 (Nov 8 2012): I received an email from Nick Eggleston about this issue – Nick experienced this problem at a customer site and heard from VMware support that “There is a vswitch failback fix which may be relevant and has not yet shipped in ESXi 5.0 (it postdates 5.0 Update 1 and is currently targeted for fixing in Update 2).  However this fix *has* already shipped in the ESXi 5.1 GA release.”  Thanks, Nick!

Update 4 (Nov 9 2012): Once I knew what this bug looked like, I’ve found other symptoms that can help you diagnose the issue in your environment.  One of the most telling is that the switches (assuming Cisco) carrying your iSCSI traffic may show high Output Queue Drops (OQD) counters when you run a ‘Show Interface Summary’.  This is because the packets sent out the wrong vmnic have no business being on the switch they were routed too, so the switch drops the packet.  I’ve also noticed that when this bug is causing issues the MAC address table (show mac address table) loses all entries for the iSCSI vmknic’s and in some cases the storage array’s front end ports.

I recently stumbled on two vSphere 5 ESXi networking bugs that I thought I would share. The issues are very similar from a cursory level, but have different symptoms, troubleshooting steps, and implications for your architecture, so I’m going to split the issues into two separate posts. Because troubleshooting these issues was a real pain, I’ll provide some details on how to identify these issues in your environments and wrap up with a third post on what I believe to be some best practices to avoid these same problems and achieve greater redundancy and resiliency in your vSphere environments.

The Problem

Today, we’ll look at an ESXi 5 networking issue that caused massive iSCSI latency, lost iSCSI sessions, and lost network connectivity. I’ve been able to reproduce this issue in several environments, on different hardware configurations. Here’s the background information on how all this started: I upgraded an ESXi 4.1 host to ESXi 5 using vSphere Update Manager (VUM). Note that I did use the host upgrade image that contained the ESXi500-201109001 iSCSI fixes – if you are upgrading to vSphere 5 and have iSCSI in your environment, use this image. Here’s a quick look at how the networking was configured on this host:

The iSCSI networking was configured in a very typical setup, and per best practices, as outline in VMware’s documentation, as well as from many vendors (see EMC’s Chad Sakac’s ‘A Multivendor Post on using iSCSI with VMware vSphere’), with two vmnic uplinks, two vmknics, with one active adapter on the correct layer-2/layer-3 network, and the other unused.

vSwitch iSCSI vmknic override failover order with unused NICvSwitch iSCSI vmknic override failover order with unused NIC

After the upgrade, the standard vSwitch with two vmnics for uplinks (Broadcom NetXtreme II BCM5709 1000Base-T) and two vmknics that serviced the software iSCSI adapter failed to pass traffic (vmkping to the iSCSI targets failed) and could not mount ANY iSCSI LUN’s. VM network, management, and vMotion ports were not affected.

If I let the host sit long enough, it *might* find a couple paths to the storage, but even then performance was deteriorated per the vmkernel.log:

WARNING: ScsiDeviceIO: 1218: Device naa.60026b90003dcebb000003ca4af95792 performance has deteriorated. I/O latency increased from average value of 5619 microseconds to 495292 microseconds.

Troubleshooting

I’m going to dump a whole bunch of my troubleshooting steps on you – hopefully they not only help folks dealing with this particular bug, but help with general network and configuration troubleshooting in VMware vSphere. During troubleshooting, I removed the vmk binding for these two on the iSCSI adapter, removed the software iSCSI Adapter itself, removed the vmknics on the vSwitch, and removed the vSwitch itself. I then recreated the vSwitch, set vSwitch MTU to 9000, recreated two vmk ports, set 9000MTU, assigned IP, and set failover order for multipath iSCSI. I then re-created the software iSCSI adapter and bound the two vmk ports. I was able to pass vmk traffic and mount iSCSI LUN’s. Great – problem solved!?!?! Not so much – I rebooted the host and the problem returned.

Here are my next troubleshooting steps:

  • I repeated the procedure above and re-gained connectivity, but the problem returns on subsequent reboots. I can verifiably recreate the problem.
  • I verified end-to-end connectivity for other hosts on the same Layer 1, Layer 2, and Layer 3 network as the iSCSI initiator and iSCSI targets.
  • I verified the ESXi host’s networking configuration using the vSphere client, double-checking the vSwitch, vmnic uplinks, and vmknic configurations. Everything looked good so I canceled out.
  • I then reinstalled ESXi from scratch (maybe something was left over from 4.1 that a clean install would weed out), built up the same configuration, and was again able to re-create the problem.
  • I poured over logs (vmkernel.log, syslog.log and storagerm.log primarily). I could see an intermittent loss of storage connectivity, failure to log into the storage targets (duh – there is no connectivity, no vmkping) and high storage latency on hosts where I had rebuilt the iSCSI stack and run a few VM’s.
  • I switched out the Broadcom NIC for an Intel NIC (the Broadcom had hardware iSCSI capabilities – I wanted to be sure the hardware iSCSI was not interfering).
  • I verified TOE was enabled.

The ‘Ah-Ha’ Moment

Next, I verified the ESXi host’s networking configuration using the vSphere client one more time – the properties of the vSwitch, the properties of the vmkernel (vmk) ports, the manual NIC teaming overrides, IP addressing, etc. Everything looked correct – I MADE NO CHANGES – but when I clicked OK (last time I canceled) to close the vSwitch properties and was greeted with this warning:

changing an iscsi initiator port group warning

Wait a second… I didn’t change anything, why am I being prompted with a you’re ‘Changing an iSCSI Initiator Port Group’ warning? I like to live dangerously, and wanted to see what would happen, so I said ‘Yes’.

Much to my surprise, after only viewing and closing the vSwitch and iSCSI vmk port group settings, I was able to complete a vmkping on the iSCSI-bound vmk’s. And moreover, I completed a Rescan of all storage adapters and my iSCSI LUN’s were found, mounted, and ready for use. Problem solved? Nope. The same ugly issue re-appeared after a reboot.

While the problem wasn’t solved, I now had something to work with. My go-to troubleshooting question “What Changed?” could maybe be answered. Even though I didn’t change anything in the vSwitch Properties GUI, something changed. To see what changed in the background, I compared the output of the following ESXi Shell (or vCLI, or PowerCLI) commands before and after making ‘the change’ happen (by viewing the properties of the vSwitch/vmk ports), but found no changes.

  • esxcfg-vswitch -l
  • esxcfg-vmknic -l
  • esxcfg-nics -l

Then, I made backup copy of esx.conf

 cp /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

Then I caused ‘the change’ and then compared checksums using md5sum, but found no differences:

 md5sum /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

I compared the running .conf and the backup .conf, but found no differences:

 diff /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

Call in Air Support
At this point, I was out of ideas so I called for help: “Hello, 1-866-4VMWARE, option 4, option 2 – help!”

After repeating many of the same troubleshooting steps, the support engineer decided that I had hit on a known, and not yet patched, bug. The details of the bug are included in KB 2008144: Incorrect NIC failback occurs when an unused uplink is present. That’s right – my iSCSI traffic, vmkpings, etc were being sent down the wrong NIC – the UNUSED NIC. Ouch. The bug caused the networking stack to behave in a very unpredictable way, making my troubleshooting steps next to useless, and any other advanced troubleshooting ideas I had (sniffing, logs, etc.)

Once I knew what the issue was, I could see a bit of evidence in the logs:

WARNING: VMW_SATP_LSI: satp_lsi_pathIsUsingPreferredController:714:Failed to get volume access control data for path "vmhba33:C0:T0:L4": No connection

NMP: nmp_DeviceUpdatePathStates:547: Activated path "NULL" for NMP device "naa.60026b90003dcebb0000c7454d5cc946".

WARNING: ScsiPath: 3576: Path vmhba33:C0:T0:L4 is being removed

Notice the NULL path – the path can’t be interpreted correctly when being sent down the wrong (unsued) vmnic that is on a different subnet and VLAN. The gotcha on this issue is that I had followed best practices where applicable, and accepted default settings on the vSwitch and vmknics.

The Quick Fix
VMware KB 2008144 offers two workaround for this bug. The quick fix for the problem is to simply change the Failback setting on either the vSwitch running the software iSCSI vmknic’s to “No” (default is yes), or to change the setting on the vmknic itself if you have other port groups on the vSwitch (such as a VM Network port group to give your guest VM’s access to the iSCSI network).

Change vSwitch or Portgroup Failback

Changing Failback = No on the iSCSI vmknics and then rescanning the storage adapters fix the glitch immediately.

Architecture Changes
The second workaround from VMware is “Do not have any unused NICs present in the team.”. This translates to a slightly different architecture than that described in many documents. To achieve this workaround, the configuration would have to change to two vSwitches, each with a single vmnic uplink and a single vmk port, bound to the iSCSI adapter. This change does not impact redundancy or availability when compared with the single-vSwitch:two-vmk configuration that I was running with as one of the vmnics was set to unused anyway. This workaround does add a bit more complexity, as there are a few more elements to configure, monitor, manage, and document.

This problem seems to only present itself on vSphere Standard Switches (vSwitch), although I could not get confirmation of this (please post a comment if you know!). Assuming this is true, a vDistributed Switch (vDS) could be used for Software iSCSI traffic. Mike Foley has a write-up on how to migrate iSCSI from a vSwitch to a vDS on his blog here: https://www.yelof.com/?p=72.

A Couple More Notes
My troubleshooting fix of viewing the vSwitch settings and clicking ok seemed to temporarily resolve the issues because it triggered an up/down event on the vmk of the unused uplink. This caused the network stack to re-evaluate paths and start using the correct, Active, uplink.

Note that this problem can occur outside of my iSCSI use case – any vSwitch, Port Group, or VMKNIC with an unused adapter set in the NIC Teaming Failover Order are susceptible to this bug, so watch for it on redundant vMotion networks (vMotion randomly fails), VM Network networks (sudden loss of guest connectivity), or even your management network (hosts fall out of manageability from vCenter, and can’t be contacted via SSH, vSphere client, etc.
Leave a comment if you’ve experienced this bug – your notes on the problem may help others find and fix the issue until VMware releases a fix. I understand that a fix for this particular bug is not due out until at least vSphere 5 Update 1.

I’ll have another (shorter) writeup on the 2nd networking bug I found in ESXi 5 later in the week – check back here for a link once it is published.