One of the things I’ve really enjoyed since joining Clearpath Solutions Group is participating in our learning events.  Coming from the customer side, I didn’t get out to learn and interact with the community as often as I would have liked.  With the incredible rate of innovation in IT and paradigm shifting technologies like virtualization, cloud computing, & Everything-as-a-Service, it is increasingly important to take a break from the day-to-day and pause to re-imagine what you could and should be doing to prepare for the next wave of technology.  Events like the VMware User Group (VMUG), the VMware Virtualization Forum (coming to Washington, DC on June 19, 2012), and Clearpath’s Lunch & Learn’s are a great way to get out and get educated while having some fun.

Clearpath’s next big event will be in Baltimore, on Tuesday, May 29, 2012 at M&T Bank Stadium (Ravens Stadium).  Together with VMware, Cisco, and EMC, we’ll discuss private cloud technologies from EMC, VMware and Cisco, along with complimentary cloud computing services for disaster recovery, virtual desktops and infrastructure services. Learn what a hybrid cloud strategy can do for your IT operations and budget. You’ll learn about technologies that are available for immediate deployment and impact.  Expect a bit of hands on with gear from Cisco (Unified Computing System – UCS) and EMC (VNX arrays), as well as product demo’s from VMware (vSphere, View, and vCloud Director).

A light lunch will be provided followed later by an appetizer and cocktail hour towards to the conclusion of the event. PLUS - The event will include a tour of the stadium and special appearance with autograph signing from Ravens Wide Receiver, Torrey Smith. There will also be raffle prizes and giveaways including a trip to VMworld 2012 in San Francisco with hotel and airfare paid for!

Space is limited!  Sign up for this special event today!

Planning for the Cloud | Event Details:
Date: Tuesday, May 29th, 2012
Time: 1 – 7:30 pm
Location: M&T Bank Stadium
1101 Russell Street, Baltimore, MD

Register for the event at Clearpath’s website here: http://clearpathsg.com/planning-for-the-cloud

I recently presented at a couple of Clearpath Solutions Group Lunch & Learn style events that focused on VMware View and touched on many of the components of VMware’s growing End User Computing portfolio.  The presentation walked attendees through the concept of VDI, why it matters, and how IT groups can leverage desktop virtualization to transform from an or organization servicing devices (legacy desktops) to an organization that delivers services (apps & data) to users.

A few key points that I wanted to call out from the presentation are:

• Cloud is both a journey & a destination (and an overused buzz word).  Desktop virtualization is the beanstalk in the cloud-provided end user computing environment – it let’s us climb from an environment rooted in Windows to the cloud.
• Desktop virtualization acts as a decoupling agent between the OS and apps/data that users really want.
• Desktop virtualization provides a reset button for IT, allowing us to re-imagine, reclaim, and redeploy IT assets (people, equipment & budget).
• Desktop virtualization changes the economics of end-user computing from CapEx to OpEx, preparing way for consumption-based cloud models.

I’ve embedded the presentation below  (I originally built it in PowerPoint and converted it to SlideRocket – some of the slides and animations didn’t import well).  I’m happy to do a WebEx of the presentation if you want to learn more.

If you have questions or want to learn more about how View can fit in your environment feel free to reach out.  If you are local to Washington DC, Virgina, or Maryland watch for upcoming Clearpath events here: http://clearpathsg.com/calendar.

VMware released an update to VMware Workstation 8.0.3 Build 703057 yesterday (May 3, 2012).  The update is a security update addresses two vulnerabilities that would allow out-of-bounds memory writes on floppy and SCSI devices under certain conditions.  The vulnerabilities may allow specially crafted attacks to crash the VMX process (i.e. crash the running VM) or theoretically execute commands on the host.  The VMware Workstation 8.0.3 release notes can be found here: https://www.vmware.com/support/ws80/doc/releasenotes_workstation_803.html.

VMware’s security advisories for these vulnerabilities can be found here: http://www.vmware.com/security/advisories/VMSA-2012-0009.html.

Users of VMware Workstation should be prompted to update to the latest version when launching the application:

If you are not automatically prompted to update VMware Workstation, manually check for updates from the Help menu:

You can also download the full install package of VMware Workstation from http://www.vmware.com/products/workstation.

While not specifically mentioned in the release notes, I wonder if this update was prompted by the VMware source code leak described in the VMware Security Blog: http://blogs.vmware.com/security/2012/05/vmware-security-note.html.  If so, kudos to VMware’s security and engineering teams for quickly identifying, fixing, and releasing updates to correct flaws that may be exposed through this code leak.

I’ve had several people ask me about using CD/DVD drives on VMware View virtual desktops.  Specifically,

• Can I use a CD or DVD drive with VMware View Desktops?
• Why doesn’t the CD or DVD drive on my re-purposed PC with the VMware View Client pass through to the View desktop?
• How can I make a CD or DVD drive work on my View Desktop?
• Are there specific CD/DVD drives that are supported on VMware View?
• How can I present a CD image (.ISO) to all View Desktops?

CD/DVD Devices in a View Environment – What works, what doesn’t?

First, the bad news.  When using PC over IP (PCoIP) as the connection protocol, IDE, SCSI, or SATA attached CD or DVD (or BluRay, HD-DVD, LaserDisc) drives local to the VMware View Client are not passed through to the VMware View virtual desktop.  This quite simply means that the internal CD drive of your re-purposed desktop will not be mapped to the View desktop when using PCoIP.  That said, mapping local CD or DVD drives to a View virtual desktop is supported when using Microsoft Remote Display Protocol (RDP).  Unfortunately, user experience (video performance, audio quality) declines and low/latent bandwidth  can increase if you switch from PCoIP to RDP.

The better approach to presenting a local CD/DVD drive to a View Desktop is to use a USB-connected drive.  This approach will work with both re-purposed PC’s acting as your View client as well as with Thin Clients and Zero Clients.  USB-connected devices can be passed through from a View Client to a VMware View virtual desktop while using PCoIP.  Teradici (the creators of PCoIP and maker of most of the Tera chips in industry leading Thin and Zero clients) has published a list of tested USB CD/DVD drives on their KB here http://techsupport.teradici.com/ics/support/default.asp?deptID=15164&task=knowledge&questionID=566.  Note that if you use some of these drives with a Windows XP View desktop, you may experience a View disconnect when ejecting a disc from the drive.  Other USB CD and DVD drives may also work – I grabbed this drive (OWC SuperSlim USB 2.0 Enclosure) from a colleague and was able to pass it through to my Clearpath Cloud hosted Windows 7 View desktop with no problems.

CD/DVD Drive Performance in a View Environment

While a USB connection will get your CD or DVD drive connected to your View desktop, performance may not be amazing.  For many tasks like software installs, reading images, or copying files performance will be acceptable.  A Teradici client can support up to 15Mbps for USB connected devices.  On a low bandwidth connection, or a connection with high latency, a local USB connected CD/DVD may not work well, if at all.  On a fast LAN connection, I would expect that redirected CD/DVD performance would be acceptable for many tasks, including reading medical images (PACS, DICOM viewer, RIS, HIS).  Do not expect to stream media or watch movies from a USB connected CD/DVD drive – but because View is typically a business solution, and watching the latest Hollywood hit at work is not often a business requirement, this should present few problems in all but the smallest niche cases (so get back to work!).  Disc burning would work with this configuration too – better for burning small images than a mass disc duplication effort.

Alternative Methods and Niche Cases

I’ve run into a few situations over the past decade or so of IT work where a desktop needed to have a CD/DVD perpetually connected to support some crummy application (FoxPro comes to mind here).  If you are supporting an environment like this, you can configure your parent VM with a persistently connected ISO on a VMFS or NFS vSphere datastore. Using this method, you could present the same CD image (.ISO) to all the VM’s based on the parent VM.  The connected ISO setting carries over to View Composer replicas and linked clones.  If you wanted to target this configuration to just a specific pool of View desktops, create a new snapshot of the parent image with the virtual CD/DVD drive set to ‘Connect at power on’ and an ISO file selected in the ‘Datastore ISO File’ field, as pictured below:

Note that in the VMware View Optimization Guide for Windows 7, and many other View tuning guides, it is recommended to remove the virtual CD/DVD drive from the parent VM.  If you follow this recommendation, you could still leverage a virtual CD/DVD drive, such as Microsoft Virtual CD Control Tool or UltraISO can also be used inside the View guest operating system, with an ISO library established on a low cost shared storage tier network share to give users a self-service way to mount a CD or DVD image.

Restricting Access to CD/DVD Devices in a VMware View Environment

While VMware-empowered IT is all about giving users freedom and control, there may be times where you want to restrict access to USB CD/DVD devices (or any other USB devices) in your VMware View environment.  You can do this in several ways – a complete solution will likely use a combination of the following methods based on user requirements, administrative overhead, and security posture:

• Use VMware View Group Policy administrative templates (.adm) applied to the View Client to control either ‘Connect all USB devices to the desktop on launch’ or ‘Connect USB devices to the desktop when they are plugged in’.
• If using RDP, use the same View Client (vdm_client.adm) administrative template to disable the ‘Redirect supported plug-and-play devices’
• Use built-in Active Directory Group Policy Objects (GPO), applied to your View Desktops OU’s, such as

• Administrative Templates | System | Removable Storage Access
• Administrative Templates | Windows Components | Remote Desktop Services | Device and Resource Redirection
• Administrative Templates | System | Device Installation | Device Installation Restrictions
• Use your Thin Client or Zero Client vendor’s management software to restrict access to, or disable, USB devices on Thin Clients.

Serial port devices in a VMware View environment

I’ve also had a couple questions about serial device (DB9 COM port) connections to View desktops.  Most of the above applies to serial devices too.  If serial devices are required in your environment, pick up a USB to Serial adapter (something like this will do Cables To Go 26886 USB To DB9 Male Serial Adapter, Blue (0.45 Meters/1.5 Feet)) and use GPO’s to control access and redirection of the USB device.

Wrap-up

Hopefully this answers your questions on using CD and DVD drives in a VMware View virtual desktop environment.  A combination of VMware View settings, vSphere VM configuration settings, and Microsoft Group Policy Objects (GPO’s) give you the ability to use and control removable media devices in your View environment.

Have you struggled with any other devices in your VMware View environment – leave a comment with your questions or suggested work-arounds!

Note: This article is cross-posted at the Clearpath Solutions Group blog at http://clearpathsg.com/weblog.

This event has been postponed.  We have tentatively rescheduled for September 2012 (post VMworld).  Stay tuned to this site and http://dcvmug.com for more info and future events.

Potomac Regional VMUG is Coming to Washington D.C. – May 9th, 2012!

Join your fellow VMware users at this free, full-day event for in-depth technical sessions, demonstrations, exhibits, and networking opportunities. Come to the Nationals Stadium to meet with your peers and industry experts to discuss virtualization trends, best practices, and the latest technology!

Visit the Potomac Regional VMUG User Conference event page to reserve your place at this event now.

Take advantage of these educational and networking opportunities:

• 8 VMware educational sessions
• 12 partner education tracks
• Exhibitor area with VMware partners
• Complimentary breakfast and lunch
• Registration is complimentary, so register today!

Wednesday, May 9, 2012 8:00 a.m. – 4:30 p.m.

Location Nationals Park 1500 South Capitol Street Southeast Washington, DC 20003

Please note: You will need to log into the VMUG website to register for this meeting. If you have not already obtained your VMUG login credentials, please visit the Username and/or Password Help page to request your username and password.

Questions? Contact memberservices@vmug.com

I have found VMware vSphere Resource Pools to be an often misunderstood element, and incorrectly implemented by even seasoned VMware administrators.  I recently found an example of an incorrect implementation of Resource Pools being used to organize VM’s in vCenter’s Hosts and Clusters view, without considering the potential performance impact of this configuration.  I thought that I might try to explain the resource pool piece a bit better, particularly around Shares.

First, a few technical points to remember:

• Resource Pools can only be used when VMware Distributed Resource Scheduler (DRS) is enabled on a cluster.  DRS is available in vSphere Enterprise and Enterprise Plus editions.
• VMware Shares are like stock/shares in a company –> Shares are proportional.  The more  shares you issue (on your company or in your VMware environment), the less valuable the individual share is.  The more shares you have, the greater your voting power/ability to claim a right to resources is.
• The vSphere Hosts Cluster itself is technically a Resource Pool too.
• Shares only come into play when there is resource contention (Memory or CPU).  If the host/cluster has spare resources available for a guest it allocates availability of those resources based on the configured vRAM size and vCPU count, taking into consideration any limits (Mem/CPU) on the VM.
• Resource Pools can be used to apply resource Shares to VM’s, as well as Reservations and Limits.  Shares, Reservations, and Limits can also be set on a per-VM basis.  The values on Resource Pools and individual VM’s combined can do funny things.

Now, consider the following screen shot of the environment running on my laptop lab:

I have 16 VM’s total, 4 in ResourcePool1, and 1 VM in ResourcePool2.  The resource pools are set to ‘Normal Shares’ with no other configuration changes to the default.  The VM’s are all the same size as far as vRAM and vCPU are concerned.

If we look at the ‘Resource Allocation’ tab of the Cluster, we see that the VM’s in the Cluster resource pool each have 5% of the Shares for CPU, and each ‘Normal’ resource pool has 21% of the total shares of the cluster.  If we divide the 21% shares by the 4 VM’s in ResourcePool1, we find that each VM gets 5.25% of the cluster’s total shares.  That means that the VM’s in this ‘Normal’ pool actually get a bit more CPU time under resource contention than those in the parent cluster pool.  The single VM (MoveVM1) in ResourcePool2 gets 21% of the shares all to itself – much larger than the remaining VM’s in the Cluster, even though it may not be a VM of higher importance in your environment.

Now consider Memory resource allocation:

The ‘Normal’ shares value for memory on a resource pool is 163840.  Compare this to the 640 shares each single VM gets in my current config (VM-* each have 64MB RAM, so a very low share value per VM).  We’ve effectively configured only a fraction of 1% (basically 0% shares) to the VM’s in the parent cluster pool, while the few VM’s in the ResourcePool1 and ResourcePool2 get to share 48% of cluster resources per pool.  This still leads to an imbalance for the VM’s in each pool as compared to VM’s in the sibling pool and to the VM’s in the parent pool.  I.E. – MoveVM1 gets all 48%; while the four VM’s in ResourcePool1 share 48% (48/4=12% each).

Now let’s say that ‘MoveVM1′ was just a VM that a well-intentioned administrator spun up for some testing, putting it in a resource pool for ease of locating it in an otherwise busy vCenter view.  This otherwise inconsequential testing VM now has the right to claim 48% of the memory (vRAM) in the cluster and 21% of CPU time when there is resource contention in the cluster.  Now if this lowly testing VM is the cause of resource contention on an otherwise healthy cluster, it’s getting its cake and eating it too.  When the going get’s tough, the least valuable workload gets the priority in this configuration!  We’ve created a case where the meek (VM) shall inherit the resources.  All your resources are belong to us.  You get the idea – a big fat oops!

Hopefully you can see how using Resource Pools as units of organization can negatively affect the performance of an environment under resource contention, even if those resource pools are configured with ‘default/normal’ values.  The risk becomes more severe if the values of those resource pools were to change (say, move to a ‘High’ value for shares on ResourcePool2 in my lab).

Best Practices

• It is best to use folders the ‘VM’s and Templates View’ in the vSphere Client for organizing VM’s and only (sparingly) use resource pools to achieve specific performance goals under resource contention (shares) and/or resource allocation goals (limits & reservations) under normal operating circumstances with resource availability.
• Use the Resource Allocation tab to review and tune your Resource Pool (and individual VM) Shares settings.  Export the resource allocation list from vCenter and use Excel if you want to experiment with values and do what-if calculations.
• Restrict permissions to create and modify resource allocation on VM’s and on Resource Pools in vCenter to only VMware Administrators who understand the implications of these changes, and who are capable of doing the math to properly design a resource allocation model.  If you don’t have an expert on staff, engage a consultant to help (might I suggest Clearpath Solutions Group ).
• Check out VMware vCenter Operations Manager (vCenter Ops/ vCOPs) for advanced monitoring and capacity planning (the product formerly known as VMware CapIQ is included with vCenter Ops).  Good monitoring and forecasting can help to avoid resource contention in highly consolidated, oversubscribed, or performance-sensitive environments (and what environments are not all three of these today?).
• Review the VMware vSphere Resource Management Guide to better understand the use and function of Resource Pools. The Guide can be found here: http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-resource-management-guide.pdf.
• Buy  Duncan Epping’s and Frank Denneman’s book, VMware vSphere 5 Clustering Technical Deepdive to really understand how VMware vSphere Distributed Resource Scheduler (DRS) (as well as VMware High Availability (HA) VMware Distributed Power Management (DPM), and VMware Storage DRS) function and are best configured.

Now that I’ve settled in at Clearpath Solutions Group, I’ll be writing occasional blog posts on the company blog site.  My first article went live today – a fancy little piece on using Microsoft Key Management Services with VMware View to activate Windows 7.  Here’s a quick excerpt:

When deploying or recomposing a pool of Windows 7 (or Windows Vista) virtual desktops on VMware View, you may receive an error message stating: “composer agent initialization error (16): Failed to activate software license (waited 0 seconds).”  This error occurs when you use a Multiple Activation Key (MAK) to activate the parent VM.  View Composer QuickPrep, by default, uses Microsoft Key Management Services (KMS).  VMware KB Article 1026556 provides workarounds for activating MAK clients, or bypassing activation by QuickPrep so you can use a trial key.  The workarounds are appropriate for test/pilot cases, but will eventually cause problems such as MAK key exhaustion.  The proper way to license and activate a deployment of Windows 7 on VMware View is to use a KMS.

Read the rest of my post on how to setup and use a KMS for VMware View on the Clearpath Blog: http://clearpathsg.com/using-kms-vmware-view-windows-activation.

On Wednesday, I wrote about a VMware vSphere 5 networking bug that caused issues with iSCSI networking.  That bug, described in VMware KB 2008144 caused vmk traffic to be sent over the unused vmnic uplink in a team where there is an unused uplink and an explicit failover policy present.  See the diagram below to better understand what was going on there….

The second bug vSphere 5 networking bug I experienced was similar to the first: traffic was sent out of an unexpected interface after upgrading to ESXi 5.  This particular bug surfaced while troubleshooting my iSCSI bug (because why not have two unrelated bugs at the same time).   Many of the troubleshooting steps I used in the first networking bug were employed on this, so I won’t bore you again with the details.  I will, however, give you a quick overview of the network setup that this issue appeared in.

Configuration

Here’s layer 1 connectivity for ESXi host vmnics to the switching stack.

Here’s the ESXi network config:

The specific portion of the configuration that was impacted by this bug was vSwitch0, which contained vmnic0 & vmnic1, my Management Network vmknic and vMotion vmk port group.  The Management and vMotion port groups had a manually set failover order as pictured below:

This is all pretty standard network configuration for a VMware ESXi host with 6 physical network adapters, and follows best practice for management network redundancy for VMware HA (I highly recommend reading more on HA best practices in Duncan Epping and Frank Dennemon’s VMware vSphere 5 Clustering Technical Deepdive book).

The Problem

The problems that were manifested as a result of the bug were:

• ESXi hosts would intermittently fall out of manageability by vCenter, the vSphere Client, and SSH (which was enabled from the console of the hosts).  Management connectivity could be restored (most times) by restarting the ESXi Management Network from the console.  I could usually ping the management network IP address even though the host was not manageable.
• ESXi syslogs stopped being sent to the vCenter Syslog collector.
• vMotion between hosts in the cluster intermittently worked.  vMotion success was not always in sync with management network connectivity.  vMotion capabilities could be restored by restarting the ESXi Management Network from the console.
• As an added bonus, VMware High Availability (HA) would sometimes detect host failures and restart VM’s on the surviving HA nodes.

Notice my use of ‘intermittently, usually, and sometimes’ – this made for tough troubleshooting.  If you’re gonna fail, fail big.  None of this wimpy on-again, off-again nonsense.

The Resolution

Luckily, I had VMware support on the phone as this problem appeared.  The support tech seemed to know just what the problem was:

A known issue on ESXi 5 occurs when two or more vmkernel NIC’s (vmknic) are on the same standard vSwitch.  Under this configuration traffic may be sent out the incorrect vmknic.

As far as I am aware, there is no VMware Knowledgebase article for this issue yet (comment if you know of one), so details are based on my own conversation with the support engineers working the case.  From what I was able to infer, this bug appears:

• More often when ESXi hosts are under stress (my iSCSI involving network bug really stressed out the hosts – and me – when all paths were down)
• Seems to happen more on Broadcom NIC’s then Intel
• Triggered and/or fixed by a network up/down event (such as restarting the management network on the host).
• Does NOT happen with a Distributed Virtual Switch.
• Is scheduled to be patched with or after vSphere 5 Update 1.

Whereas my iSCSI bug involved a vmnic team with an unused uplink, and traffic being sent out the wrong vmnic, this second bug occurred with two vmnic’s (one active, one standby), two vmk ports on a standard switch, and traffic being sent out the wrong vmk port, which happened to have a different active vmnic than the correct vmk port had.  Here’s a diagram of the traffic flow gone wrong:

I still find it a bit odd that ICMP traffic continued to flow to the interface, but that the Management traffic took an alternate route out and landed on my non-routed vMotion VLAN (and different subnet).

Workaround

The workaround for this bug is simple – remove the second NIC and second VMkernel Port (vMotion for me) from the vSwitch and restart the ESXi Management Network.  Once this was done, management traffic flowed normally.

I then created a new vSwitch, attached the second vmnic to it, and then re-created the VMkernel port for vMotion.

While the work-around was great for getting my hosts back into manageability, it was not so great for the redundant architecture I had originally implemented.  After splitting the VMkernel ports onto two different vSwitches, I received warnings in vCenter that “Host currently has no management network redundancy.”  KB 1004700 addresses this message if you are looking for more info on it.  I could disable the warning, but that would be like slapping a fresh coat of paint on a jalopy.

Architecture Changes

The workaround for this bug kills redundancy.  Simply adding another two physical NIC’s and, in my case, binding one to the Management vSwitch and one to the vMotion vSwitch.  This change would require host downtime to install new hardware if your host only had 6 NIC’s like this environment did.

Alternatively, you could migrate your Management network and vMotion networks to a virtual Distributed Switch (vDS) as this bug does not appear to impact vDS – only standard vSwitches.  Side note: Check Duncan Epping’s post on using a virtual vCenter server connected to a vDS if that’s holding you back from going to a vDS.  Also read the new vDS Best Practices whitepaper from VMware.

Final Note

This bug could impact more configurations than the one I highlighted.  For example, I could see it causing issues with Multiple-NIC vMotion in Sphere 5.

Drop a comment if you have experienced this bug, know of a KB article, or can think of any other ways it might be manifested.

Update: This issue was fixed as of 3/15/2012 in ESXi 5.0 Update 1 per the original knowledge base article: VMware KB 2008144. To download ESXi 5.0/vCenter Server 5.0 Update 1, see the VMware Download Center. 

I recently stumbled on two vSphere 5 ESXi networking bugs that I thought I would share. The issues are very similar from a cursory level, but have different symptoms, troubleshooting steps, and implications for your architecture, so I’m going to split the issues into two separate posts. Because troubleshooting these issues was a real pain, I’ll provide some details on how to identify these issues in your environments and wrap up with a third post on what I believe to be some best practices to avoid these same problems and achieve greater redundancy and resiliency in your vSphere environments.

The Problem

Today, we’ll look at an ESXi 5 networking issue that caused massive iSCSI latency, lost iSCSI sessions, and lost network connectivity. I’ve been able to reproduce this issue in several environments, on different hardware configurations. Here’s the background information on how all this started: I upgraded an ESXi 4.1 host to ESXi 5 using vSphere Update Manager (VUM). Note that I did use the host upgrade image that contained the ESXi500-201109001 iSCSI fixes – if you are upgrading to vSphere 5 and have iSCSI in your environment, use this image. Here’s a quick look at how the networking was configured on this host:

The iSCSI networking was configured in a very typical setup, and per best practices, as outline in VMware’s documentation, as well as from many vendors (see EMC’s Chad Sakac’s ‘A Multivendor Post on using iSCSI with VMware vSphere’), with two vmnic uplinks, two vmknics, with one active adapter on the correct layer-2/layer-3 network, and the other unused.

After the upgrade, the standard vSwitch with two vmnics for uplinks (Broadcom NetXtreme II BCM5709 1000Base-T) and two vmknics that serviced the software iSCSI adapter failed to pass traffic (vmkping to the iSCSI targets failed) and could not mount ANY iSCSI LUN’s. VM network, management, and vMotion ports were not affected.

If I let the host sit long enough, it *might* find a couple paths to the storage, but even then performance was deteriorated per the vmkernel.log:

WARNING: ScsiDeviceIO: 1218: Device naa.60026b90003dcebb000003ca4af95792 performance has deteriorated. I/O latency increased from average value of 5619 microseconds to 495292 microseconds.

Troubleshooting

I’m going to dump a whole bunch of my troubleshooting steps on you – hopefully they not only help folks dealing with this particular bug, but help with general network and configuration troubleshooting in VMware vSphere. During troubleshooting, I removed the vmk binding for these two on the iSCSI adapter, removed the software iSCSI Adapter itself, removed the vmknics on the vSwitch, and removed the vSwitch itself. I then recreated the vSwitch, set vSwitch MTU to 9000, recreated two vmk ports, set 9000MTU, assigned IP, and set failover order for multipath iSCSI. I then re-created the software iSCSI adapter and bound the two vmk ports. I was able to pass vmk traffic and mount iSCSI LUN’s. Great – problem solved!?!?! Not so much – I rebooted the host and the problem returned.

Here are my next troubleshooting steps:

• I repeated the procedure above and re-gained connectivity, but the problem returns on subsequent reboots. I can verifiably recreate the problem.
• I verified end-to-end connectivity for other hosts on the same Layer 1, Layer 2, and Layer 3 network as the iSCSI initiator and iSCSI targets.
• I verified the ESXi host’s networking configuration using the vSphere client, double-checking the vSwitch, vmnic uplinks, and vmknic configurations. Everything looked good so I canceled out.
• I then reinstalled ESXi from scratch (maybe something was left over from 4.1 that a clean install would weed out), built up the same configuration, and was again able to re-create the problem.
• I poured over logs (vmkernel.log, syslog.log and storagerm.log primarily). I could see an intermittent loss of storage connectivity, failure to log into the storage targets (duh – there is no connectivity, no vmkping) and high storage latency on hosts where I had rebuilt the iSCSI stack and run a few VM’s.
• I switched out the Broadcom NIC for an Intel NIC (the Broadcom had hardware iSCSI capabilities – I wanted to be sure the hardware iSCSI was not interfering).
• I verified TOE was enabled.

The ‘Ah-Ha’ Moment

Next, I verified the ESXi host’s networking configuration using the vSphere client one more time – the properties of the vSwitch, the properties of the vmkernel (vmk) ports, the manual NIC teaming overrides, IP addressing, etc. Everything looked correct – I MADE NO CHANGES – but when I clicked OK (last time I canceled) to close the vSwitch properties and was greeted with this warning:

Wait a second… I didn’t change anything, why am I being prompted with a you’re ‘Changing an iSCSI Initiator Port Group’ warning? I like to live dangerously, and wanted to see what would happen, so I said ‘Yes’.

Much to my surprise, after only viewing and closing the vSwitch and iSCSI vmk port group settings, I was able to complete a vmkping on the iSCSI-bound vmk’s. And moreover, I completed a Rescan of all storage adapters and my iSCSI LUN’s were found, mounted, and ready for use. Problem solved? Nope. The same ugly issue re-appeared after a reboot.

While the problem wasn’t solved, I now had something to work with. My go-to troubleshooting question “What Changed?” could maybe be answered. Even though I didn’t change anything in the vSwitch Properties GUI, something changed. To see what changed in the background, I compared the output of the following ESXi Shell (or vCLI, or PowerCLI) commands before and after making ‘the change’ happen (by viewing the properties of the vSwitch/vmk ports), but found no changes.

• esxcfg-vswitch -l
• esxcfg-vmknic -l
• esxcfg-nics -l

Then, I made backup copy of esx.conf

 cp /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

Then I caused ‘the change’ and then compared checksums using md5sum, but found no differences:

 md5sum /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

I compared the running .conf and the backup .conf, but found no differences:

 diff /etc/vmware/esx.conf /etc/vmware/esx.conf.bak

Call in Air Support
At this point, I was out of ideas so I called for help: “Hello, 1-866-4VMWARE, option 4, option 2 – help!”

After repeating many of the same troubleshooting steps, the support engineer decided that I had hit on a known, and not yet patched, bug. The details of the bug are included in KB 2008144: Incorrect NIC failback occurs when an unused uplink is present. That’s right – my iSCSI traffic, vmkpings, etc were being sent down the wrong NIC – the UNUSED NIC. Ouch. The bug caused the networking stack to behave in a very unpredictable way, making my troubleshooting steps next to useless, and any other advanced troubleshooting ideas I had (sniffing, logs, etc.)

Once I knew what the issue was, I could see a bit of evidence in the logs:

WARNING: VMW_SATP_LSI: satp_lsi_pathIsUsingPreferredController:714:Failed to get volume access control data for path "vmhba33:C0:T0:L4": No connection

NMP: nmp_DeviceUpdatePathStates:547: Activated path "NULL" for NMP device "naa.60026b90003dcebb0000c7454d5cc946".

WARNING: ScsiPath: 3576: Path vmhba33:C0:T0:L4 is being removed

Notice the NULL path – the path can’t be interpreted correctly when being sent down the wrong (unsued) vmnic that is on a different subnet and VLAN. The gotcha on this issue is that I had followed best practices where applicable, and accepted default settings on the vSwitch and vmknics.

The Quick Fix
VMware KB 2008144 offers two workaround for this bug. The quick fix for the problem is to simply change the Failback setting on either the vSwitch running the software iSCSI vmknic’s to “No” (default is yes), or to change the setting on the vmknic itself if you have other port groups on the vSwitch (such as a VM Network port group to give your guest VM’s access to the iSCSI network).

Changing Failback = No on the iSCSI vmknics and then rescanning the storage adapters fix the glitch immediately.

Architecture Changes
The second workaround from VMware is “Do not have any unused NICs present in the team.”. This translates to a slightly different architecture than that described in many documents. To achieve this workaround, the configuration would have to change to two vSwitches, each with a single vmnic uplink and a single vmk port, bound to the iSCSI adapter. This change does not impact redundancy or availability when compared with the single-vSwitch:two-vmk configuration that I was running with as one of the vmnics was set to unused anyway. This workaround does add a bit more complexity, as there are a few more elements to configure, monitor, manage, and document.

This problem seems to only present itself on vSphere Standard Switches (vSwitch), although I could not get confirmation of this (please post a comment if you know!). Assuming this is true, a vDistributed Switch (vDS) could be used for Software iSCSI traffic. Mike Foley has a write-up on how to migrate iSCSI from a vSwitch to a vDS on his blog here: http://www.yelof.com/?p=72.

A Couple More Notes
My troubleshooting fix of viewing the vSwitch settings and clicking ok seemed to temporarily resolve the issues because it triggered an up/down event on the vmk of the unused uplink. This caused the network stack to re-evaluate paths and start using the correct, Active, uplink.

Note that this problem can occur outside of my iSCSI use case – any vSwitch, Port Group, or VMKNIC with an unused adapter set in the NIC Teaming Failover Order are susceptible to this bug, so watch for it on redundant vMotion networks (vMotion randomly fails), VM Network networks (sudden loss of guest connectivity), or even your management network (hosts fall out of manageability from vCenter, and can’t be contacted via SSH, vSphere client, etc.
Leave a comment if you’ve experienced this bug – your notes on the problem may help others find and fix the issue until VMware releases a fix. I understand that a fix for this particular bug is not due out until at least vSphere 5 Update 1.

I’ll have another (shorter) writeup on the 2nd networking bug I found in ESXi 5 later in the week – check back here for a link once it is published.

Time to make it official!  I have moved to Clearpath Solutions Group where I’ll take on the role of Virtualization Practice Manager, focusing on delivering VMware solutions and services to our VMware, Cisco, and EMC customers.  I’m looking forward to joining a high energy company where I can focus on the VMware technologies that I am passionate about.  I’m going to hit the ground running with VMware View implementations, Site Recovery Manager (SRM) work, and vSphere upgrades on EMC storage and Cisco UCS servers.

A few perks that I am looking forward to in this new job:

• With the changing roles, I’ll be ramping up my writing on VMtoday, as well as on Clearpath’s blog.  A consistent focus on virtualization ought to help provide fodder for my blogging.
• My commute will shorten by about 33% (new commute will be a whopping 4 miles each way on side roads – unheard of in Northern Virginia / Washington, DC!).
• More active in social media – @ClearpathSG gets it (+Clearpath Solutions Group, http://www.facebook.com/ClearpathSolutions, http://www.linkedin.com/company/100452)
• VMware Partner Exchange (PEX) – I look forward to seeing all my virtualization friends at PEX next week!
• New colleagues – there’s some great folks at Clearpath, and we’re still hiring.  Check it out and join the team!

Leaving my role as IT Manager at Tiber Creek Consulting was not an easy decision, but it is time to move from being a constantly distracted, generalist IT Manager to something with a bit more focused. Tiber Creek has been amazingly flexible and encouraging over the past three years as Stephanie Townsend and I have worked though her health issues and the stress it imposes on our family (with Stephanie having improved over the past few months after a new procedure to patch her cerebral spinal fluid (CSF) leaks, I feel some freedom to go after great new things).   Tiber Creek is a great group of people doing great work for our armed forces, and I’ll miss the great team that I have been a part of for the past three years.  Thanks for everything, Tiber Creek!

Here’s to an awesome new role, at an awesome new company.  Thumbs up – let’s do this thing!

This is a re-post from http://dcvmug.com/dc-vmug-january-17th-2011/.  Please follow the DC VMUG site for updates!

The Washington DC VMUG invites you to the first meeting of the DC VMUG in 2012. The event will be held on Tuesday, January 17th, 2011 at Nationals Park.

Our sponsors, Tintri and Veeam, will have some great give-aways for attendees. We’ll have a tour of the new Nationals Park at the end of the VMUG meeting.

Register Now!

Agenda

7:30am – 8:00am: Registration & Breakfast
8:00am – 8:45: Tintri Presentation
8:45 – 9:00am: Break
9:00 – 9:45am: Apps in the Enterprise : Horizon App Manager vs. Citrix Xenapp; Michael Letschin, vExpert, Convergence Technology Consulting
9:45 – 10:00am: Break
10:00 – 10:45am: Veeam Presentation
10:45 – 11:15am: Ask the Experts and Closing Remarks
11:30am: Nationals Park Tour

Location

Washington Nationals Park

1500 South Capitol St., SE
Washington, DC 20003

Free Parking in Lot C, or Metro to Navy Yard (Green Line) and walk 0.3 mi SW to Nationals Park

Proceed to Red Porch Restaurant

I’ve been asked several times recently to recommend training resources for VMware, so I thought I might write my responses up in a blog post to help out folks in the community who are looking for the best resources to gain VMware knowledge, prepare for their VCP and other certifications, and continue on their journey to becoming a virtualization rockstar.

I’ve picked up a bunch of certifications over the past 10 years.  For me, certification is not the means to an end, but the end of some long, intensive studying and lab work, then doing some deep dive studying and doing.  By the time I get to the test, passing should be a forgone conclusion.  I’ll save details of my lab for a future post and focus on the books and other learning resources that I use.  When getting into a new or updated technology, I start out my studying with a good overall survey of the technology I want to learn.  I like a good book that hits all of the major components, provides background information to help explain why the technology, component, or module really matters and how it fits into the big picture.  Then I get into technology specific books – deep dives, command line references, and architecture books.

Books

Mastering VMware vSphere 5, by Scott Lowe

My go-to book for VMware vSphere has been Scott Lowe’s Mastering VMware vSphere 4.  Scott’s updated book, Mastering VMware vSphere 5 started shipping yesterday.  Scott covers everything from the basics of what a hypervisor is to VMware vSphere best practices.  This is a great book to accompany lab work as it includes licensing, planning and installation, setting up virtual networking, storage basics, security, resource allocation, HA, DRS, and even some automation with the CLI and PowerCLI (PowerShell).  The book is well written, taking you methodically through vSphere, while providing plenty of helpful hints along the way.  Do yourself a favor and click the picture to the left to order it from Amazon now (paperback or Kindle format).  This book is a great way to get started with studying for your VCP certification.

VMware vSphere 5 Clustering Technical Deepdive

Once I have the basics down, I get into the deep dive work. The first deep-dive book for VMware vSphere 5 is VMware vSphere 5 Clustering Technical Deepdive, by Duncan Epping and Frank Denneman.  This is Duncan and Frank’s second book that focuses on the clustering and high availability technologies available in VMware vSphere.  Readers of Duncan and Frank’s first book, VMware vSphere 4.1 HA and DRS Technical deepdive (Volume 1), got an incredibly deep look at how to configure VMware HA and DRS.  The new vSphere 5 Clustering Technical Deepdive includes Storage DRS as well.  I’ve talked to several readers of both these books and Duncan and Frank’s blogs who have remarked that 1.) I’ve been doing it wrong all along, 2.) I totally understand how HA and DRS work after reading this, and 3.) My environment really is resilient and reliable thanks to this book.

Pearson and VMware teamed up earlier this year to create VMware Press.  There are several books coming from VMware Press, as well as other authors/publishers that are now available for pre-order from Amazon.com.  These include:

There are not many vSphere 5 specific books out yet, but many of the vSphere 4 resources are still very useful.  My library includes these:

Video Training

If you are not a big reader or you are looking for additional topics, check out TrainSignal’s VMware Training Videos.  TrainSignal offers a whole slew of courses (many taught by VMware vExperts), including:

• vSphere 5 Training
• VMware View Administration Training
• vSphere Troubleshooting
• vSphere Performance Monitoring
• vSphere Security Design
• vSphere PowerCLI.

I have a couple of TrainSignal DVD’s and found them to be good quality with deep technical content.

Blogs and BrownBags

Once you are comfortable with the material, you can start to study for your VCP.  Several bloggers have published collections of materials to help you prepare for the VCP, VCAP, and even the VCDX.  I recommend Simon Long’s collection here: http://www.simonlong.co.uk/blog/vcp-vsphere-upgrade-study-notes/ and Cody Bunch’s VCP4 Resource Page and BrownBag sessions.

Instructor Led & Certification

Finally, once you are all read up, head to a VMware Education instructor led class.  You need to take a VMware Authorized Training course to qualify to sit for the VMware Certified Professional (VCP) certification exam.  VMware also offers a nice catalog of eLearning courses.  If you want to get a discount on eLearning, Instructor-Led training, and certification exams from VMware, check out the VMUG Advantage program.

Did you miss VMworld this year, but want a chance to get a peek at the exciting new technologies on display at the show?  The VMworld 2011 Snapshot Series is your chance to catch the excitement of VMworld in a city near you.

Please join VMware for this complimentary half-day event to get a taste of VMworld 2011 and learn how VMware and our partners are enabling organizations like yours to understand the true value of cloud computing and business agility. Cloud computing is an organizations strategic weapon-not just for IT, but to enable full business transformation, and will eventually change how the business operates. Attend, participate and learn from our Subject Matter Experts on how to modernize your IT operations. Improve, simplify and save with VMware. Topics to be discussed:

• VMware Keynote Review / General Session Highlights
• Build and Own Your Cloud: vSphere 5.0, vCloud Director 1.5 vCenter SRM 5 and vShield 5
• Desktop and End User Computing Strategies

This is a unique opportunity to learn and understand more about infrastructure, applications and end user transformation that is happening today making organizations more automated, competitive and agile. Hear from our subject matter experts, see live demos as well as network with your peers. This is your VMworld Snapshot. This is your Cloud. Own it.

For the folks here in Washington, DC with me, the VMworld 2011 Snapshot will be on October 18th.

Join your fellow virtualization professionals on Thursday, September 15th from 6:00pm to 11:00pm for an independent networking event. vBeers is designed to be vendor agnostic and independent and allow IT pros to get together and network and talk without the influence of the vendors we all get pushed. Shortly after the VMworld conference we should have plenty to talk about. Old Dominion has a standard happy hour until 7 PM with discounts on food and drink along with some specials running all night. We have a central area reserved just for the group. Just one block from the Mt. Vernon Square /7th St-Convention Center metro stop, you can take the green or yellow line right to us.

Sign up here: http://dcvbeers.eventbrite.com

More info on the venue here: http://olddominionbrewhouse.com/

Here’s the SlideRocket presentation for tomorrow’s DC VMUG.  My first real time playing with VMware’s Slide Rocket. It took me a few crashes of PowerPoint to move to SlideRocket (even after I had included it in the recent VMware Acquisitions slide).  Overall a pretty good experience!

This presentation is subject to change (I’m running out of steam tonight).  If/when I do update it on SlideRocket, this embedded deck will just magically update – the magic of the cloud!

Feel free to comment with changes or suggestions for other free utilities and scripts!  I  added a Scripts & Free Tools page to http://vmtoday.com today with some embedded Google Docs spreadsheets to keep a running list of the cool VMware scripts and tools that I come across. I’ve found and promptly forgotten so many over the years – hopefully this helps me as much as it helps you in finding and saving some great community resources!

This is your last chance to register for tomorrow’s (July 12, 2011) big VMware online event, “Raising the Bar, Part V”.  During the event, VMware CEO Paul Maritz and CTO Steve Herrod will announce the next generation of VMware cloud infrastructure offerings.  The event will be broadcasted live beginning at 9:00 AM PDT

The agenda looks like this:

• 9:00-9:45 Maritz and Herrod present
• 10:00-12:00 Deep dive breakout sessions
• 10:00-12:00 Live Q&A with VMwarevExperts

While I don’t have any inside knowledge on the event, I expect new product and service offerings, along with details on recent acquisition announcements on how VMware is bringing to bear its vision for a seamless cloud infrastructure – public, private, hybrid – that is simple to manage, easy to scale, and with all the dynamism that one could imagine.   And if that weren’t enough, one lucky attendee for this online event will win a ticket to VMworld 2011.  This event promises to be a great intro to what is shaping up to be a great VMworld 2011 in Las Vegas!

The Washington DC VMUG Invites You to a Joint Event with VMware on vCenter™ Management Solutions.
Join us in the morning and learn from VMware subject matter experts how VMware vCenter™ Operations are fundamentally simplifying how IT is managed:

• Automate operations with intelligence and visibility
• Interpret workload, health and capacity metrics
• Identify and resolve workload stress and population problems
• Correlate change events and performance patterns

Stick around for the VMUG lunch and meeting:

• Networking Lunch
• VMware Updates and What’s New with VMUG
• VMworld 2011 Preview
• Community Scripts and Free Tools to Manage and Monitor Your VMware Environment. *Attendees are invited to bring your favorite scripts for monitoring, managing, and reporting on your VMware environments to share with the group.*

This is a unique opportunity to learn from best practices, see a demo and to speak with vCenter™ Operation experts directly.

Space is limited, so register today!
Wednesday, July 20, 2011
9:00 a.m. – 2:30 p.m.

Location
JW Marriott Washington DC
1331 Pennsylvania Avenue NW
Washington, DC 20004
Driving directions

Agenda
8:30 a.m. – Breakfast & Registration
9:00 a.m. - VMware Presentation: Master Your Virtualized Infrastructure
10:15 a.m. - Partner Presentation
11:00 a.m. – VMware Product Demo
11:30 a.m. – Morning Wrap up
12:15 p.m. - VMUG Lunch, Updates, and Announcements
01:00 p.m. - VMware Security and Compliance, Rob Babb, VMware

01:30 p.m. - Community Scripts & Free Tools to Manage & Monitor Your VMware Environment

Sponsored by

We’ll kick off a normal VMUG meeting after the VMware event. Please make special note – *Attendees are invited to bring your favorite scripts for monitoring, managing, and reporting on your VMware environments to share with the group.* Prizes for the best scripts shared!

This one is for all you in the DC area (I”m looking at you, DC VMUG members) looking to network with fellow VMware folks….

For all the guys out there that have talked for weeks and months about our choice of beer and bourbon, the time has come.  We will be having a vBeers Happy Hour at one of the newest and hottest places in DC, Jack Rose Dining Salon.

For those of you who don’t know it was featured in the Washingtonian Magazine this month and is just opening with one of the largest Scotch selection ever assembled from a private collector in DC.  They also should have 20+ beers on tap.

Heres the details, hope you all can make it and if we find the date I picked is really bad… don’t fret this will not be the only time we do this..

Location: Jack Rose Dining Saloon
Address: 2007 18th Street Northwest, Washington D.C., District of Columbia 20009
Date: July 13, 2011
Time: 6PM – 9PM ish

Get yourself to the vBeers website to register:  Direct link to the event here: http://www.vbeers.org/2011/06/08/vbeers-and-vbourbon-dcnovamd-weds-13th-july-2011/

As both an IT Manager and storage & virtualization consultant, I have shepherded SMB’s through SAN purchases in support of VMware environments.  This may not hold true for the bigger IT shops, but as I take these smaller companies through the SAN purchase process, a common set of questions is often asked – Do we really need a SAN, and why is it so darn expensive!?!? I quickly learned that my (cheeky) answer “because it’s enterprise class, baby” just didn’t cut it.  So how do you move beyond “I can get a 2TB hard drive for $100 at Best Buy, why are you selling me on 300GB hard drives for $500 each, and you want me to buy how many?”

For me, the answer boils down to a handful of key factors: performance, reliability, availability, supportability, data integrity, and ease of administration.  When we’re talking business-class workloads, your 2TB USB disk  might have the ease of administration thing going for it (if you can’t plug-n-play that you probably shouldn’t be talking about storage…), but probably falls far short in the other areas.

The Disks

Disks are the basic building block of storage arrays - usually spinning disks, so we’ll start with looking at the difference between consumer-grade and enterprise grade disks.  (Note: I talked disks back in the early posts in this series if you want to do some catch up reading).  I/O activity on disks make the pretty little lights go blink blink – whether we’re talking the hard drive in my mom’s 10-year old PC or the dozens of FC disks in that newfangled EMC array in my office.  The similarities stop there.

There are substantial differences between desktop class hard drives and enterprise class hard drives.  Enterprise class drives are designed for longer duty cycles, higher throughput, and greater reliability.  Some key differences include:

• Enterprise class drives are built with bigger, better, and faster components – from stronger actuator magnets and faster spindle speeds to increased platter counts, enterprise class drives are more capable of delivering more IOPS than desktop class drives.  This addresses the performance factor.
• Enterprise class drives are rated for a much heavier duty cycle, capable of serving more data, more often.  This addresses the reliability and availability factors.
• Enterprise class drives such carry longer warranties from the manufacturer and, better yet, are supported by your equipment vendor (EMC, NetApp, Dell, HP, etc.).  Supportability: check.
• Enterprise class drives have more ‘intelligent’ firmware that is capable of identifying and recovering from corruption and other errors.  I’ve got your data integrity right here, buddy.
• Enterprise class arrays running enterprise class drives have automatic RAID rebuild, proactive hot-sparing, and monitoring/alerting capabilities.  These all ease administrative burden – especially where the stuff hits the fan.

If you want more formal documentation, I recommend reading this whitepaper from Intel that covers some of the differences in enterprise and desktop class drives: http://download.intel.com/support/motherboards/server/sb/enterprise_class_versus_desktop_class_hard_drives_.pdf.  There are always exceptions – I’ve had brand new enterprise class drives show up dead on arrival, and my mom’s poor old 4.3 GB IDE drive is still crunching away.  Check out Google’s 2007 Hard Drive Study results if you want some real life metrics: http://labs.google.com/papers/disk_failures.pdf.  Interestingly enough, Google’s study specifically states that they are using consumer-grade hard drives….

Magnetic spinning media is not the only disk in the game – Solid State Disks’s are rapidly gaining in popularity in both consumer and enterprise applications.  Just like with traditional hard drives, the SSD space has differences between Enterprise Flash Drives (EFD) and consumer SSD’s. Multi Layer Cell (MLC) SSD’s are cheaper, slower, and more prone to errors, but are sufficient for consumer use.  Single Layer Cell SSD’s are typically more performant, reliable, and are often built with improved error-detection/correction abilities.  EFD’s are increasingly working their way into the mid-range storage appliances from several vendors.

The Array

If we think beyond the single disk to the array, the same enterprise vs. consumer class questions arise.  I’ll talk about best practices in choosing a storage array for your virtualization project in a later post, but there are a few items that compliment the consumer vs. enterprise class drive discussion that I want to present now.

First, if you’re building a bet-the-business solution, make it complete solution.  While that Drobo, Qnap, or Synology may be on the VMware HCL and is sporting more features than you can shake a stick at, those disks you got for cheap on Tiger Direct aren’t enterprise class, have a short manufacturer warranty, and won’t be supported by the array vendor when you need that support the most (usually 2am on a Saturday).  Nothing against Drobo, Qnap or Synology mind you – I’d love to have one of their NAS devices for my home lab – just suggesting that you think through your decisions and their impact on performance, reliability, availability, supportability, data integrity, and ease of administration.

Second, an array is more than a bunch of disks that go blink blink.  From advanced caching and vSphere integration, to seemingly simple things like rack mount design and vibration dampening, enterprise class storage arrays complex!  Don’t believe me?  Check out this video on shouting in the data center and the subsequent effects of the hard drive vibrations caused by the sound waves.

httpv://www.youtube.com/watch?v=tDacjrSCeq4

Extra Credit

Speaking of hard drive vibrations and reliability, there are several storage vendors getting creative.  Check out this write up on how Xiotech is innovating: http://www.virtualinsanity.com/index.php/2010/10/18/the-spinning-disks-stay-of-execution/

Need some extra ammunition to argue for that new SAN?  Check out this Hitachi Data Systems (HDS) whitepaper “Is DAS really cheaper than SAN? Storage Economic Methods help to differentiate Price and Cost“.

Bottom Line

If you serious about building a virtualization platform or private cloud environment for your organization, choose components that are built to withstand your workload’s performance and availability requirements.

VMware has been investing in the VMware User Group (VMUG) program over the past year, adding new benefits and opportunities to an already great program.  In addition to local meetings where you can interact with fellow VMware professionals, the VMUG program offers other great benefits – the latest is the VMUG Advantage program. The VMUG Advantage program offers the following benefits:

• $100 Discount to VMworld
• 20% Discount on VMware Instructor Led Training
• 20% Discount on VMware Certification
• 1-Year Free Subscription: All Access eLearning Courses ($750 Value)
• 30% Discount on Workstation and Fusion Software html Licenses

The VMUG Advantage program has an introductory price of $170 for individuals (with corporate/multi-user discounts available) until May 21.  Not a bad deal if you plan to attend VMworld in Las Vegas (Aug 29 – Sept 1), complete a certification (VCP, VCAP, VCDX, VCA-DT), or take some instructor led training through VMware.  Heck – the eLearning subscription alone more than pays for program subscription cost!

As a VMUG member be sure to login to the new www.myvmug.org and update your profile or add an picture or avatar. Over the next month you’ll see exciting new benefits added to the website, including VMUG Community Pages, Event Listings, and Education Resources.

If you’re not already a VMUG member, go ahead and sign up at www.myvmug.org to learn about local/regional events and to get access to the other great benefits like the VMUG Advantage.