As both an IT Manager and storage & virtualization consultant, I have shepherded SMB’s through SAN purchases in support of VMware environments.  This may not hold true for the bigger IT shops, but as I take these smaller companies through the SAN purchase process, a common set of questions is often asked – Do we really need a SAN, and why is it so darn expensive!?!? I quickly learned that my (cheeky) answer “because it’s enterprise class, baby” just didn’t cut it.  So how do you move beyond “I can get a 2TB hard drive for $100 at Best Buy, why are you selling me on 300GB hard drives for $500 each, and you want me to buy how many?”

For me, the answer boils down to a handful of key factors: performance, reliability, availability, supportability, data integrity, and ease of administration.  When we’re talking business-class workloads, your 2TB USB disk  might have the ease of administration thing going for it (if you can’t plug-n-play that you probably shouldn’t be talking about storage…), but probably falls far short in the other areas.

The Disks

Disks are the basic building block of storage arrays - usually spinning disks, so we’ll start with looking at the difference between consumer-grade and enterprise grade disks.  (Note: I talked disks back in the early posts in this series if you want to do some catch up reading).  I/O activity on disks make the pretty little lights go blink blink – whether we’re talking the hard drive in my mom’s 10-year old PC or the dozens of FC disks in that newfangled EMC array in my office.  The similarities stop there.

There are substantial differences between desktop class hard drives and enterprise class hard drives.  Enterprise class drives are designed for longer duty cycles, higher throughput, and greater reliability.  Some key differences include:

• Enterprise class drives are built with bigger, better, and faster components – from stronger actuator magnets and faster spindle speeds to increased platter counts, enterprise class drives are more capable of delivering more IOPS than desktop class drives.  This addresses the performance factor.
• Enterprise class drives are rated for a much heavier duty cycle, capable of serving more data, more often.  This addresses the reliability and availability factors.
• Enterprise class drives such carry longer warranties from the manufacturer and, better yet, are supported by your equipment vendor (EMC, NetApp, Dell, HP, etc.).  Supportability: check.
• Enterprise class drives have more ‘intelligent’ firmware that is capable of identifying and recovering from corruption and other errors.  I’ve got your data integrity right here, buddy.
• Enterprise class arrays running enterprise class drives have automatic RAID rebuild, proactive hot-sparing, and monitoring/alerting capabilities.  These all ease administrative burden – especially where the stuff hits the fan.

If you want more formal documentation, I recommend reading this whitepaper from Intel that covers some of the differences in enterprise and desktop class drives: http://download.intel.com/support/motherboards/server/sb/enterprise_class_versus_desktop_class_hard_drives_.pdf.  There are always exceptions – I’ve had brand new enterprise class drives show up dead on arrival, and my mom’s poor old 4.3 GB IDE drive is still crunching away.  Check out Google’s 2007 Hard Drive Study results if you want some real life metrics: http://labs.google.com/papers/disk_failures.pdf.  Interestingly enough, Google’s study specifically states that they are using consumer-grade hard drives….

Magnetic spinning media is not the only disk in the game – Solid State Disks’s are rapidly gaining in popularity in both consumer and enterprise applications.  Just like with traditional hard drives, the SSD space has differences between Enterprise Flash Drives (EFD) and consumer SSD’s. Multi Layer Cell (MLC) SSD’s are cheaper, slower, and more prone to errors, but are sufficient for consumer use.  Single Layer Cell SSD’s are typically more performant, reliable, and are often built with improved error-detection/correction abilities.  EFD’s are increasingly working their way into the mid-range storage appliances from several vendors.

The Array

If we think beyond the single disk to the array, the same enterprise vs. consumer class questions arise.  I’ll talk about best practices in choosing a storage array for your virtualization project in a later post, but there are a few items that compliment the consumer vs. enterprise class drive discussion that I want to present now.

First, if you’re building a bet-the-business solution, make it complete solution.  While that Drobo, Qnap, or Synology may be on the VMware HCL and is sporting more features than you can shake a stick at, those disks you got for cheap on Tiger Direct aren’t enterprise class, have a short manufacturer warranty, and won’t be supported by the array vendor when you need that support the most (usually 2am on a Saturday).  Nothing against Drobo, Qnap or Synology mind you – I’d love to have one of their NAS devices for my home lab – just suggesting that you think through your decisions and their impact on performance, reliability, availability, supportability, data integrity, and ease of administration.

Second, an array is more than a bunch of disks that go blink blink.  From advanced caching and vSphere integration, to seemingly simple things like rack mount design and vibration dampening, enterprise class storage arrays complex!  Don’t believe me?  Check out this video on shouting in the data center and the subsequent effects of the hard drive vibrations caused by the sound waves.

httpv://www.youtube.com/watch?v=tDacjrSCeq4

Extra Credit

Speaking of hard drive vibrations and reliability, there are several storage vendors getting creative.  Check out this write up on how Xiotech is innovating: http://www.virtualinsanity.com/index.php/2010/10/18/the-spinning-disks-stay-of-execution/

Need some extra ammunition to argue for that new SAN?  Check out this Hitachi Data Systems (HDS) whitepaper “Is DAS really cheaper than SAN? Storage Economic Methods help to differentiate Price and Cost“.

Bottom Line

If you serious about building a virtualization platform or private cloud environment for your organization, choose components that are built to withstand your workload’s performance and availability requirements.

I ran into an issue with a customer today where a VM was performing terribly.  From within the guest OS (a Windows 2003 application server running .NET in IIS which I will call BigBadServer) things appeared sluggish and CPU time was high.  The amount of time being spent on the kernel was notably high.  The VM in question had 4 vCPU’s and a good helping of memory.

I don’t have access to the VMware client at this particular site – just some of the guests, so I was flying blind.  Gut feeling told me that I was dealing with a resource contention issue.  I had the VMstats provider running in the guest (http://vpivot.com/2009/09/17/using-perfmon-for-accurate-esx-performance-counters/) showed me that there was no ballooning or swapping going on, and that the vCPU’s were not limited and the CPU share value seemed to be at the default.

I strongly suspected that the physical server running VMware ESX was oversubscribed on physical CPU (pCPU) resources.  Essentially, the guest VM’s that are sharing the resources of the physical machine are demanding more resources than the machine can handle.  To verify this theory, I had the client check the ‘CPU Ready’ metric on BigBadServer and bingo!

CPU Ready is a measure of the amount of time that the guest VM is ready to run against the pCPU, but the VMware CPU Scheduler cannot find time to run the VM because other VM’s are competing for the same resources.

From the stats the customer provided on our phone call, the CPU Ready for any one of the 4 vCPU’s on the BigBadServer was on average 3723ms (min: 1269ms, max:8491ms).  (Update 8/25/2010 to clarify summation stat) The summation for the entire VM was around 12,000ms on average and peaked around 35,000.  The stats came from the real-time performance  graph/table in the vSphere client. The real-time stats in the vSphere Client update every 20 seconds, so the CPU Ready summation value should be divided by 20,000 to get a percentage of CPU ready for the 20 second time slice.  If I take the worst case scenario of 8491ms per vCPU, this VM spent nearly 43% (8491/20,000) of the 20 second time slice waiting for CPU resources.

The CPU Ready summation in milliseconds counter in the vCenter Client is not always the most accurate or easy to interpret stat – to better quantify the problem it might be best to go to the ESX command line and run ESXTOP.  CPU Ready over 5% could be a sign of trouble, over 10% and there is a problem.  Running ESXTOP in batch mode and then analyzing the output using Windows Perfmon or Excel might be a good way to go on this to get a view over several hours rather than the realtime stats we were looking at.  I wrote a post a while back with more info on ESXTOP batch mode: http://vmtoday.com/2009/09/esxtop-batch-mode-windows-perfmon/

To help quantify the problem a bit more, the BigBadServer is on an ESX 4.0 server with about 10 other servers.  The physical blade has two dual-core CPU’s (AMD Opteron 2218HE’s which are not hyperthreaded).  The other VM’s on the blade have different vCPU and vMemory configurations.  3 VM’s (including BigBadServer) have 4 vCPU’s.  A couple have 2 vCPU’s, and the remainder are configured with 1 vCPU.  In ESX 4.x, the VMware console OS actually runs as a hidden VM, pegged to pCPU #1.

I generally recommend a pCPU:vCPU ration of 1:4 for mid-sized VMware deployments of single vCPU VM’s.  The blade we are running on is a 1:5 with several multi-vCPU VM’s.  The multi-vCPU’s start to skew the ratio recommendation and require some advanced design decisions.  VMware’s scheduler requires that all the vCPU’s on a VM run concurrently (even if the Guest OS is trying to execute a single thread).  Also, the VMware CPU Scheduler prefers to have all the vCPU’s from a VM run on the same pCPU.  As workloads are bounced around between pCPU’s, the benefits of CPU cache are lost.  This is one of those ‘more-is-less’ situations that you run into on virtualized environments.

What this CPU Scheduler nonsense means in this case is that the 4 vCPU’s on BigBadServer have to wait until all logical pCPU’s on the box are idle (including the one that runs ESX itself) before it can run.  If ESX can’t accomplish that (we are experiencing resource contention) it starts prioritizing workloads according to what it can best run.  It is much easier to schedule the smaller VM’s, so it tends to run those on pCPU more frequently.  The larger VM’s tend to suffer a bit more than the smaller ones.  We are competing with 2 other VM’s with 4 vCPU’s that use up all of the logical pCPU’s when they need to run, as well as with the smaller VM’s.

I suggested a few ways to fix this issue for the BigBadServer web server:

1. Using Shares and/or Reservations on the VM.  This probably won’t work in our situation as the physical server is too over-subscribed.  We might see a slight improvement in BigBadServer (or we might not see any change), but possibly at the extreme expense of the other VM’s sharing the blade.
2. Reduce the number of vCPU’s on BigBadServer AND the other multi-vCPU VM’s on the same physical server.  This would reduce resource contention and open up a whole bunch of scheduling options for the VMware CPU Scheduler.  This is the quickest/cheapest fix, but will not work if the VM’s really do need 4 vCPU’s.  A little workload analysis should determine which can be made smaller (the vCenter server graphs/stats should be enough for this).  For what it’s worth, by our analysis BigBadServer seems to be happier with 4 vCPU assuming we can run with a low CPU Ready on those 4.
3. Move the BigBadServer VM to a physical ESX server with fewer multi-vCPU VM’s so there is less contention.
4. Move the BigBadServer VM to a physical ESX server with quad-core pCPU’s (ideally two quad-cores or bigger).  This would give a lot more flexibility to the VMware CPU Scheduler and allow it to run quad-vCPU VM’s on the same pCPU for greater efficiency.
5. Split BigBadServer into 2 smaller VM’s – The server currently runs a couple sites.  We could split them onto two servers – one for Project1 and one for Proejct2.  This configuration would take some design, testing, and time but could scale out better, give more flexibility and availability in the long run.

I’m not sure which way the customer will go on this one yet, but I feel good having armed them with enough knowledge and options to make an informed decision.

To avoid problems like this in the future, I recommend these rules of thumb:

• Design your hosts for your guests.  Taking your Guest VM sizes into account when designing your environment and choosing physical hardware is crucial if you need bigger VM’s.
• Don’t make your VM’s bigger than you have to.  It is always easier to add resources than take them away.  Hot Add of CPU and Memory in vSphere make adding incredibly easy.
• Monitor your environment for CPU Ready, Swapping, and other metrics that can indicate an inefficient design.
• Call for help when you can’t figure out what is going on (I’m happy to help!).  VMware is super powerful, but some things can be downright backwards when it comes to resource allocation on a fixed set of hardware.

If you are looking for some resources to help explain CPU Scheduling a bit more, I recommend:

• VMware’s Official documentation of CPU Scheduler in vSphere 4.1 – http://www.vmware.com/files/pdf/techpaper/VMW_vSphere41_cpu_schedule_ESX.pdf.
• A nice summary of co-scheduling from VMware’s Performance Blog: http://blogs.vmware.com/performance/2008/06/esx-scheduler-s.html
• Description and stats on Ready Time metrics for VI3: http://www.vmware.com/pdf/esx3_ready_time.pdf
• Understanding Virtual Center Performance Statistics: http://communities.vmware.com/docs/DOC-5230.pdf

(Updated 8/25/2010 to include a few additional reference links and corrected summation divided by time slice to get accurate values)

I wrote some time back about networking problems with a clean install of ESX 3.5 U3 on a HP DL380 G3 server in a lab environment.  A simple downgrade to ESX 3.5 RTM corrected the issue and I didn’t think much about it.  One of the servers in the lab died and I went about the business of rebuilding it.  Having learned my lesson, I started with an ESX 3.5 RTM install and then patched to Update 3 plus other applicable updates.  Much to my chagrin, the server began crapping out on me randomly.  Some reboots, some networking issues, and other assorted not so good things.  Now the DL380 G3 is not the spring chicken it used to be, so I assumed some faulty hardware was probably to blame.  Some diagnostics and log reviews yielded no hardware issues.

On a whim, I decided to check the VMware HCL to see if the DL380 G3 was still on the list of compatible servers for ESX.  Now, I had checked, or rather ‘remembered’ checking, the HCL before that first problematic install, but a recheck never hurts.  When I arrived at the VMware HCL page I saw the same old trusty PDF link with a slightly newer revision date than my previous visit.  I was pleasantly surprised when I clicked the PDF link to find that I was redirected to a searchable, filterable forms-based version of the HCL.  Nice!  Let’s do this thing….

I’m a little lazy, so I simply used a keyword search to look up ‘DL380 G3′.  Presto-chango: I’ve got results, and I like what I see:

Search Results for DL380 G3 on the VMware HCL

My eyes jump right to ESX 3.5 – Supported, on my platform, no further questions your honor.  Close the old browser window and move on with my life, my life being troubleshooting this darn server.

A few hours later I am still struggling with the server and turn to Ebay for salvation.  “If you can’t beat em, cheat em,” my grandfather used to say.  I’ll find new hardware for my lab.  I identified some other hunk of junk that just might work and decided to check the HCL for it.  That’s when it jumped out at me: there are Update versions included in the HCL and I had been to quick to see it on my DL380 G3 search.  Back to the HCL.

This time I just do a search for ‘DL380′, leaving off the Generational notation and get the following:

Search Results for DL380 from the VMware HCL

The ProLiant DL380 G5 with Quad-core Intel Xeon processors lists ESX 3.5 U3, ESX 3.5 U2, and ESX 3.5 U1 as supported releases, along with the RTM ESX 3.5.  The Update versions are not listed for the G3 or G4.  After some self-deprecating curses and a reinstall of ESX 3.5 Update-nada, stability returned.

The lesson learned, double-check the HCL (or if you are a little slow like me, a triple-check doesn’t hurt).  The HCL is major version and Update-revision sensitive.  And, not all models are treated equally.  You’ll notice in the picture to the left that the DL380 G5 has different supported releases depending on the CPU Model.

Also, keep in mind that you need to verify that all components of your VMware infrastructure are on the HCL from Servers and Systems to IO Devices, and Storage/SAN.  The VMware HCL site offers some basic tips for searching here: http://www.vmware.com/resources/compatibility/help.php.

Here’s the real take-away: The VMware HCL is there for a reason.  Sure, you might be able to get something that is not on the HCL to work, but you may experience instability along the way.  In the event that you are running a non-HCL system you may also find that VMware Support may be limited in what they can do for you.

Between budget cuts and New Year’s resolutions, improving your security posture is probably near the top of your to-do list.  Much has been made of security concerns in a virtual environment, but it is always good to re-visit your configurations and make sure they are still on par with recommended best practices.  I began re-reviewing VI security best practices after reading at post by Bob Plankers at The Lone SysAdmin (Bob has been on my reading list for years – he has a great style and always brings fresh insights) on why you would want a second super-user account on your ESX servers.

We certainly all have our own opinions and operations procedures when it comes to configuring and hardening our environments, but I decided to take a look at what the experts had to say on this particular subject and other basic build and hardening recommendations.  Here is what I found:

VMware Security Resources

VMware Security Utilities

VI3.5 Security Hardening Whitepaper

Defense Informaion Systems Agency (DISA) ESX Server Security Technical Implementation Guide

DISA ESX Server Checklist

As a side note, DISA publishes many STIG’s at http://iase.disa.mil/stigs/.  Your tax dollars paid for these, so you might as well check them out.

NSA VMware ESX  Server 3 Configuration Guide

There are also numerous tips and scripts for locking down your virtual infrastructure in the VMware Community Forums (Start here: http://communities.vmware.com/message/941372).

So back to the question of second super user accounts: It seems that best practices are to create a second user account with sufficient access to the console, granting that user SUDO privledges, and then disabling the default root account.