VMtoday

VMworld Lines, Long Lines and Damn Long Lines

Joshua Townsend — Mon, 30 Aug 2010 22:47:14 +0000

I’ve heard some grumbling about the long lines popping up at VMworld with folks blaming the new first-come format for admission into the various sessions. A few folks have asked me my take on it – here’s what I’ve got:

- Yes, the lines are long but they seem to move fast. Whether that’s because people are bailing out or because the staff here is efficiently moving people into session rooms as they open up I don’t know. I suspect a bit of both.
- I anticipate lines getting shorter over the next few days as the Solutions Exchange opens up a ton of space and activities for VMworld participants.
- As people get a feel for the flow of the event, things will start moving along better. Rather than jam-packing as many sessions into their schedule, people will start to balance vendor time on the Solution Exchange floor, Hands-On Labs, and other activities.

What has your experience been so far? Any suggestions for improving the VMworld experience? I’d love to hear your ideas in the comments!

VMworld Content Catalog Released The VMworld 2009 Content Catalog was released on Friday night...
VMworld 2010 Labs Preview I was fortunate enough to be offered a sneak peak...
VMworld Here I Come Today is the last day for VMworld early-bird registration, and...

VMworld T-Shirt Count

Joshua Townsend — Sun, 29 Aug 2010 23:55:06 +0000

Way back in the day (we’re talking way back in high school here) I worked my summers away at a Six Flags park in Western New York. Every fall the park hosted an International Food Festival. Let me tell you – the perogies, sausage and gyro’s slid down like nobody’s business. But the real prize was the Italian bakery’s cannoli’s. The 5 or so folks that shared an office with me decided we should track our cannoli consumption. We did (like you do a drug or spin up VM’s just cause you can) something like 126 cannoli’s in one weekend. We called it the Cannoli Count – kept a tally o a whiteboard in the office. Sickening, right? So, what does this have to do with virtualization, you ask. Well, not much really, but here is where I am going with it. T-shirts are handed out like nobody’s business at VMworld, and they accumulate like a pack-rat’s pile of newspapers in my dresser drawers (I still have a high-school wrestling t-shirt that dates back to 1994 in the rotation). My VMware Widow wife hates them, so I figure I’ll see how many more I can collect this week. With two already in my hands after VMworld check-in, enter the:

Josh’s VMworld T-Shirt Count

I’ll keep the running tally going throughout VMworld – let’s see where this ends up (besides me cleaning out my dresser to avoid sleeping on the couch when I get home).

VMworld Here I Come Today is the last day for VMworld early-bird registration, and...
VMworld Content Catalog Released The VMworld 2009 Content Catalog was released on Friday night...
VMworld 2009 Sponsor and Exhibitor List The VMware ecosystem is rich and thriving, with partners, vendors,...

VMworld 2010 Labs Preview

Joshua Townsend — Sun, 29 Aug 2010 20:56:31 +0000

I was fortunate enough to be offered a sneak peak at the VMworld 2010 Hands-on Labs setup this morning, and let me tell you – I am impressed. A lot of hard work has gone into planning, architecting and deploying the Labs environment, promising to make it the most user-friendly VMworld Labs setup yet. Here is what you need to know:

Location: The Labs will be held at Moscone West, on the corner of 5th & Howard St. This is a change from last year.

Format: There will be two types of hands-on labs – instructor led (they’re calling these Advanced Lab Tutorials) and self-paced.

The instructor labs are more of a tutorial for those who want to be walked through the lab manual by a subject matter expert in an open discussion format. The Advanced Lab Tutorial sessions support 250 seats. The Advanced Lab Tutorials will be useful for preparing for the associated self-paced labs. Take the Advanced Lab Tutorial first, then head downstairs to the Self-Paced lab.
The self paced labs are designed with a ton of flexibility, allowing you to choose what and when you work through the material. For an overview of the Lab topics, check out the VMworld 2010 Program Guide.
When you arrive at the Self-Paced Labs area, you will register for the lab you want and head to a nice waiting area if no seats are available. When your number is called, you will be lead to your seat and will fire up your lab. You’ll have an hour to work through the lab. If you need more time, ask.

Technical Specs: The VMware Core Team has obviously put an enormous amount of thought and time into improving the lab experience. For those who attended VMworld 2009, the lab experience folks a bit disappointed due to some technical glitches and scheduling issues. This year’s Labs are built with a ton of redundancy and allow for a much smoother, user-directed schedule. The scale and scope of the labs is astonishing to say the least. Here are some stats I gleaned on the lab setup:

There are 30 self-paced lab topics, each demanding their own unique environment.
There are 480 seats available for the self-paced labs, in a stadium seating configuration. This allows a huge number of people to flow through the lab environment efficiently, with minimal wait time. The lab schedule has some 40 hours of time for you to get in and work over the next several days. This equates to more than 20,000 lab-seat hours (up from about 5000 hours last year).
The labs run from one of three data centers: Miami, FL (Terremark); Ashburn, VA (Verizon); and locally in the Moscone Center. This provides a great deal of redundancy and positions the labs as a cloud offering to fit the theme of this year’s VMworld. The Miami and Ashburn sites have been running for a while, and will be reused for VMworld Europe next month. This is a change from last year where the gear was fork-lifted in for the show (remember all the racks at the bottom of the escalators?). This has given the team more time to work on the setup and iron out any problems.
The self-paced labs are based on VMware’s Cloud Lab infrastructure, purpose built for VMworld Labs. Cloud Lab provides a slick interface for provisioning labs to participants while doing some really smart things in the background to enhance performance and flexibility.
It is estimated that more than 100,000 VM’s will be provisioned in Labs this week – more than 5000 VM’s built and destroyed per lab hour! <- Read that again. Astonishing, no?
The gear driving the labs is provided by HP, Dell, EMC, NetApp, Cisco, and Xsigo. Xangati is used for monitoring performance of PCoIP to the Wyse thin clients at each seat.
There are 4 racks of compute power and 2 racks of storage per datacenter.
The storage environment is mostly 10GbE. EMC FastCache and NetApp Dedupe are both in use. Storage is mostly NFS-based.
The memory footprint required to run the labs is some 36TB.
Labs are running a few levels deep – ESX nested inside of ESX with VM’s running inside.
Host Profiles are heavily leveraged to ensure a consistent environment.
Twin DS3′s provide Internet connectivity for the Labs.
In true cloud fashion, the Lab Cloud product dynamically pre-populates lab environments based on demand. As some labs rise in popularity, the Lab Cloud will stage up environments based on that demand. This will reduce wait time for the lab environment to be readied. In years past, students would wait 5-7 minutes for their custom lab environments to be readied (building, deploying and booting a unique Active Directory, vCenter, ESXi, nested VM’s and associated products takes some time). No guarantees that there won’t be some wait time, but this is a huge step in the right direction.
There will be some 150 moderators ready to help with Self-Paced labs. Moderators are subject matter experts. If you request help through the Lab Cloud interface, a moderator who is a SME in your topic will be dispatched to help you.

A few more things to note:

There will be prize drawings for those who do the most labs, as well as those who complete the labs the fastest. Prizes will include a full pass to VMworld 2011 in Las Vegas.
Lab manuals will be made available after the show.
Some of the labs look really cool. You can find a list in the VMworld 2010 Program Guide. I am excited to see the VMware vSphere Sandbox lab – an everything-but-the-kitchen-sink setup of as many products as they could cram in. This provides a playground for you to see all of the VMware products working together, where you can create, destroy and otherwise play as you wish.
I would love to see this environment be made available for other uses after VMworld. I think VMUG’s could really benefit, as could VMware’s partner community.

Special thanks to Adam Zipman who leads the team putting this together, Dan Anderson (Dan is the lead architect behind this massive operation) and Curtis Pope who led development of the Cloud Lab interface. Also, thanks to John Troyer for setting up this morning’s breifing. I appreciate your time today, guys.

I hope you all are as excited about the labs this year as I am. I am planning to spend a good chunk of time working through the lab environments.

VMworld Content Catalog Released The VMworld 2009 Content Catalog was released on Friday night...
VMworld Here I Come Today is the last day for VMworld early-bird registration, and...
VMworld Lines, Long Lines and Damn Long Lines I’ve heard some grumbling about the long lines popping up...

High CPU Ready, Poor Performance

Joshua Townsend — Wed, 25 Aug 2010 19:52:07 +0000

I ran into an issue with a customer today where a VM was performing terribly. From within the guest OS (a Windows 2003 application server running .NET in IIS which I will call BigBadServer) things appeared sluggish and CPU time was high. The amount of time being spent on the kernel was notably high. The VM in question had 4 vCPU’s and a good helping of memory.

I don’t have access to the VMware client at this particular site – just some of the guests, so I was flying bling. Gut feeling told me that I was dealing with a resource contention issue. I had the VMstats provider running in the guest (http://vpivot.com/2009/09/17/using-perfmon-for-accurate-esx-performance-counters/) showed me that there was no ballooning or swapping going on, and that the vCPU’s were not limited and the CPU share value seemed to be at the default.

I strongly suspected that the physical server running VMware ESX was oversubscribed on physical CPU (pCPU) resources. Essentially, the guest VM’s that are sharing the resources of the physical machine are demanding more resources than the machine can handle. To verify this theory, I had the client check the ‘CPU Ready’ metric on BigBadServer and bingo!

CPU Ready is a measure of the amount of time that the guest VM is ready to run against the pCPU, but the VMware CPU Scheduler cannot find time to run the VM because other VM’s are competing for the same resources.

From the stats the customer provided on our phone call, the CPU Ready for any one of the 4 vCPU’s on the BigBadServer was on average 3723ms (min: 1269ms, max:8491ms). (Update 8/25/2010 to clarify summation stat) The summation for the entire VM was around 12,000ms on average and peaked around 35,000. The stats came from the real-time performance graph/table in the vSphere client. The real-time stats in the vSphere Client update every 20 seconds, so the CPU Ready summation value should be divided by 20,000 to get a percentage of CPU ready for the 20 second time slice. If I take the worst case scenario of 8491ms per vCPU, this VM spent nearly 43% (8491/20,000) of the 20 second time slice waiting for CPU resources.

The CPU Ready summation in milliseconds counter in the vCenter Client is not always the most accurate or easy to interpret stat – to better quantify the problem it might be best to go to the ESX command line and run ESXTOP. CPU Ready over 5% could be a sign of trouble, over 10% and there is a problem. Running ESXTOP in batch mode and then analyzing the output using Windows Perfmon or Excel might be a good way to go on this to get a view over several hours rather than the realtime stats we were looking at. I wrote a post a while back with more info on ESXTOP batch mode: http://vmtoday.com/2009/09/esxtop-batch-mode-windows-perfmon/

To help quantify the problem a bit more, the BigBadServer is on an ESX 4.0 server with about 10 other servers. The physical blade has two dual-core CPU’s (AMD Opteron 2218HE’s which are not hyperthreaded). The other VM’s on the blade have different vCPU and vMemory configurations. 3 VM’s (including BigBadServer) have 4 vCPU’s. A couple have 2 vCPU’s, and the remainder are configured with 1 vCPU. In ESX 4.x, the VMware console OS actually runs as a hidden VM, pegged to pCPU #1.

I generally recommend a pCPU:vCPU ration of 1:4 for mid-sized VMware deployments of single vCPU VM’s. The blade we are running on is a 1:5 with several multi-vCPU VM’s. The multi-vCPU’s start to skew the ratio recommendation and require some advanced design decisions. VMware’s scheduler requires that all the vCPU’s on a VM run concurrently (even if the Guest OS is trying to execute a single thread). Also, the VMware CPU Scheduler prefers to have all the vCPU’s from a VM run on the same pCPU. As workloads are bounced around between pCPU’s, the benefits of CPU cache are lost. This is one of those ‘more-is-less’ situations that you run into on virtualized environments.

What this CPU Scheduler nonsense means in this case is that the 4 vCPU’s on BigBadServer have to wait until all logical pCPU’s on the box are idle (including the one that runs ESX itself) before it can run. If ESX can’t accomplish that (we are experiencing resource contention) it starts prioritizing workloads according to what it can best run. It is much easier to schedule the smaller VM’s, so it tends to run those on pCPU more frequently. The larger VM’s tend to suffer a bit more than the smaller ones. We are competing with 2 other VM’s with 4 vCPU’s that use up all of the logical pCPU’s when they need to run, as well as with the smaller VM’s.

I suggested a few ways to fix this issue for the BigBadServer web server:

Using Shares and/or Reservations on the VM. This probably won’t work in our situation as the physical server is too over-subscribed. We might see a slight improvement in BigBadServer (or we might not see any change), but possibly at the extreme expense of the other VM’s sharing the blade.
Reduce the number of vCPU’s on BigBadServer AND the other multi-vCPU VM’s on the same physical server. This would reduce resource contention and open up a whole bunch of scheduling options for the VMware CPU Scheduler. This is the quickest/cheapest fix, but will not work if the VM’s really do need 4 vCPU’s. A little workload analysis should determine which can be made smaller (the vCenter server graphs/stats should be enough for this). For what it’s worth, by our analysis BigBadServer seems to be happier with 4 vCPU assuming we can run with a low CPU Ready on those 4.
Move the BigBadServer VM to a physical ESX server with fewer multi-vCPU VM’s so there is less contention.
Move the BigBadServer VM to a physical ESX server with quad-core pCPU’s (ideally two quad-cores or bigger). This would give a lot more flexibility to the VMware CPU Scheduler and allow it to run quad-vCPU VM’s on the same pCPU for greater efficiency.
Split BigBadServer into 2 smaller VM’s – The server currently runs a couple sites. We could split them onto two servers – one for Project1 and one for Proejct2. This configuration would take some design, testing, and time but could scale out better, give more flexibility and availability in the long run.

I’m not sure which way the customer will go on this one yet, but I feel good having armed them with enough knowledge and options to make an informed decision.

To avoid problems like this in the future, I recommend these rules of thumb:

Design your hosts for your guests. Taking your Guest VM sizes into account when designing your environment and choosing physical hardware is crucial if you need bigger VM’s.
Don’t make your VM’s bigger than you have to. It is always easier to add resources than take them away. Hot Add of CPU and Memory in vSphere make adding incredibly easy.
Monitor your environment for CPU Ready, Swapping, and other metrics that can indicate an inefficient design.
Call for help when you can’t figure out what is going on (I’m happy to help!). VMware is super powerful, but some things can be downright backwards when it comes to resource allocation on a fixed set of hardware.

If you are looking for some resources to help explain CPU Scheduling a bit more, I recommend:

VMware’s Official documentation of CPU Scheduler in vSphere 4.1 – http://www.vmware.com/files/pdf/techpaper/VMW_vSphere41_cpu_schedule_ESX.pdf.
A nice summary of co-scheduling from VMware’s Performance Blog: http://blogs.vmware.com/performance/2008/06/esx-scheduler-s.html
Description and stats on Ready Time metrics for VI3: http://www.vmware.com/pdf/esx3_ready_time.pdf
Understanding Virtual Center Performance Statistics: http://communities.vmware.com/docs/DOC-5230.pdf

(Updated 8/25/2010 to include a few additional reference links and corrected summation divided by time slice to get accurate values)

IBM DS3300 iSCSI Write Performance Solved I have been pulling my hair out with a small...
Balloon Driver Problems with SQL I have been meaning to write this up for a...
The Skinny on ESXTOP A reader named Mark contacted me today and asked if...

VMware vExpert

Joshua Townsend — Tue, 06 Jul 2010 17:21:43 +0000

I’ve held off on posting this as I don’t like to toot my own vuvuzela too loudly, but in a fit of shameless self promotion and a desire to see the program grow, here goes nothin:

I am a VMware vExpert!

VMware notifed recipients of the designation last month. The 300 or so awardees are recognized for their “advocacy of VMware solutions, your contributions to the community of VMware users, and your willingness to share your expertise with others.” I find it an absolute honor to be included amongst the vExpert class of 2010 – a group full of folks who are on the cutting edge of virtualization and cloud computing, many of whom have helped me through their books, blogs, and contributions to the VMware Community. Arnim van Lieshout has compiled a nice list of vExperts here: http://www.van-lieshout.com/vexpert-2010/.

It’s an exciting time to be a virtualization professional – in terms of the technology, people and seemingly endless opportunities. I look forward to continuing my contributions to the community through this here blog, my leadership of the Washington, DC Metro Area VMware User Group, and through my work as a implementer of VMware solutions! Special thanks to John Troyer for sponsoring the program and doing so much to organize the VMware community through the weekly VMware Community Roundtable, Social Media and the many other programs he touches.

NetApp Extends 50% Virtualization Guarantee NetApp has extended their 50% Virtualization Guarantee to include Citrix...
About VMtoday.com is managed by Joshua Townsend. Josh is an IT...
Washington DC VMware User Group – April 29th The next Washington, DC VMware User Group event will be...

EMC Virtual Storage Integrator Update

Joshua Townsend — Tue, 06 Jul 2010 16:49:32 +0000

I upgraded my in-house VMware vSphere environment to 4.0 Update 2 last week. After upgrading my vSphere Client to the Update 2 version I was greeted with a series of 7 ugly error messages stating: Method not found: ‘VMware.CustomControls.LabelEx VpxClient.Common.Util.Helper.AddLabel(System.Windows.Froms.Control,Int32, int32, System.String, System.Drawing.FontStyle, Boolean)”.I assumed a plug-in had caused the error message. I started my troubleshooting by disabling the 3rd-party plug-ins in the environment, beginning with the free EMC Storage Viewer. Upon disabling the EMC Storage Viewer 2.x plug-in the problem was resolved. I went out to EMC PowerLink to see if an update was available for the plug-in and was surprised to find that I had missed a major update/rebranding of the plug-in. EMC now calls the plug-in the ‘EMC Virtual Storage Integrator’. A hotfix was released on July 2nd to bring Update 2 support to the plug-in (version 3.0.0.32).

I updated the Solution Enabler installation (I installed Solution Enabler on my vCenter server, but it is also available as a SUSE based virtual appliance), and then updated the plug-in.

The update appeared to install without any problems. The vSphere Client launched like a champ after the update – no errors, but no EMC Storage plug-in either. Odd. The problem occurred on both my vCenter Server’s vSphere client and my workstation, so seems to be more than an isolated issue.

I did an uninstall of the plug-in using Add/Remove Programs, and then reinstalled it. After the reinstall the EMC Storage plug-in icon appeared in my vSphere Client as pictured below.

The EMC Virtual Storage Integrator (VSI) plug-in is free – installing it is a no-brainer for anyone running Clariion or Celerra storage arrays. The VSI simplifies the job of mapping vSphere Datastores to LUN’s and NFS shares on your EMC storage, and helps pinpoint the location of VM’s and RDM’s on your array. This visibility for the VMware administrator into the storage layer can go a long way in helping troubleshoot storage performance issues and simplifying communication between server, storage, and virtulization teams.

EMC actually offers three different free vSphere plug-ins, including the VSI. The EMC Unified Block plug-in and the EMC Unified NAS plug-in round out the trio. EMC’s Virtual Geek, Chad Sakac, covers all three in his blog post here: http://virtualgeek.typepad.com/virtual_geek/2010/06/update-on-emc-vsphere-plugins.html.

Installing PowerPath/VE using VMware Update Manager I am finishing up an installation of an EMC Clariion...
Networking Problems with ESX 3.5 Update 3 on the DL380 G3 I began building up a new lab environment with a...
Virtualization Bookmarks for August 28th Here are some bookmarks for resources that I have recently...

Storage Basics – Part VII: Storage Alignment

Joshua Townsend — Wed, 23 Jun 2010 13:32:35 +0000

At the risk of beating a dead horse, it’s time to resurrect my Storage Basics series. I’ve recently had some great feedback on the series and figured I should round out a few of the concepts before I wrap it up. I want to cover a topic often discussed amongst virtualization professionals, but one I often find general practitioners and server admins not understanding: storage alignment. Storage alignment, or the lack of alignment, is not a new issue and is not unique to VMware or virtualization in general. However, the effects of misaligned storage can be more greatly felt in terms of reduced performance and strain on a storage system in shared, oversubscribed or high I/O environments. Many others in the virtualization and storage communities have already covered partition alignment (see Duncan Epping, Vaughn Stewart, and most recently Chad Sakac), but I feel it is an important enough topic for me to re-hash as part of this series.

What is Storage Alignment?

Let’s start with a quick overview of what storage alignment means. Quite simply, storage alignment refers to the positioning (starting offset) of the various pieces of a systems storage components – the physical disk sectors or array’s chunks, the VMware File System (VMFS) in a VMware environment, and the guest file system’s clusters within a partition – in relation to the layer directly under the element in question. A quick graphic often makes quick work of explaining this (I often whiteboard this concept for colleagues and clients):

As you can see, the starting offset of the VMFS partition does not correspond to the physical segmentation of the underlying disks (in this case, the chunks on a SAN – but could be conceptually replaced with the sectors of a single disk). Furthermore, the clusters (or blocks) of the guest VM are not aligned to the VMFS partition nor to the underlying storage. For traditional (physical) systems or VMware RDM’s, the VMFS layer could be abstracted, but the result would be the same – the clusters of a partition would be misaligned to the underlying disk.

What Does it Mean?

Quite simply, misaligned storage (both VMFS partitions and Guest File Systems) can lead to poor performance under certain conditions. How badly performance is impacted depends on the degree of I/O strain your server and storage are under, the caching mechanisms in your environment, and the architecture of your SAN. Again, a visual can help explain how misaligned storage can hurt you. For simplicity let’s leave out the VMFS layer as we consider the following diagram (pardon my hasty Visio visualization):

What we see is that the target data in a tiny 16kb read request spans two 64kb chunks on our storage array. Any reads of that piece of data will result in twice the amount of data as would be minimally necessary being transferred to the host’s storage stack. The net effect is an increase in the work the storage array must do – gobbling up IOPS that would otherwise be available for the real work of reading data, reducing throughput on the interface, and messing with cache algorithms and dedupe mechanisms on some arrays. In short, misaligned storage is an efficiency killer. Now add in the VMFS layer back in and you’ll see how things get complicated.

If (and we’re talking a big IF here) every bit of data you wanted to read spanned a chunk or sector boundary, you could experience half the expected performance due to misalignment. In reality, depending on your workload and storage technology your performance increase from properly aligning your storage will probably be somewhere between 10-30%.

Want to dig deeper?

There have been some great resources published on this issue over the past few years on storage alignment. Major vendors have all begun pushing information on the problem – here are some of the best that I have found:

Microsoft has a Knowledge Base article (http://support.microsoft.com/kb/929491) that describes the problem and symptoms of misaligned partitions, how to determine if your partition is aligned, and the use of diskpart to create aligned partitions.

Microsoft also has an in-depth article on MSDN, including some performance numbers at http://msdn.microsoft.com/en-us/library/dd758814.aspx. Also check out Jimmy May’s series Partition (Sector) alignment for SQL Server here: http://blogs.msdn.com/b/jimmymay/archive/2008/10/14/disk-partition-alignment-for-sql-server-slide-deck.aspx. One of the best descriptions of the complexities of the problem can be found in Jimmy’s blog series.

VMware has an article here: http://www.vmware.com/pdf/esx3_partition_align.pdf. Be aware that this article is for Virtual Infrastructure 3, not vSphere 4.0. Some of the information is now a bit dated.

Netapp has a few documents to check out: http://media.netapp.com/documents/tr-3428.pdf (VI3), and http://media.netapp.com/documents/tr-3749.pdf (vSphere)

EMC covers alignment in their TechBooks for Clariion, Celerra, and Symmetrix.

Tools to Align Partitions:

Ok – so you’ve bought into this whole partition alignment thing as being a real issue. How to you fix it? Here are some tools:

MSInfo32.exe, wmic, and dmdiag will show you misaligned partitions on Windows machines (check the Microsoft links above for usage info).
Diskpart.exe (or diskpar.exe on versions of Windows previous to 2003) creates aligned partitions on Windows systems. Diskpart cannot be used to realign a previously created partition, only to create new correctly aligned partitions.
MBRScan/MBRAlign from NetApp can report on and realign existing virtual disks on a VMware ESX server. Also a nifty PowerShell script from NetApp to find if your partitions are aligned: http://communities.netapp.com/docs/DOC-6175
vOptimizer from Vizioncore can report on and realign existing virtual disks.
GParted can be used to create aligned partitions on both Windows and Linux machines, and to realign some existing partitions.
VMware vCenter – VMFS datastores created using vCenter are aligned automatically. Note – Guest VMDK’s are not aligned automatically by vCenter – you must manually create aligned partitions on your VMDK’s or use a Guest OS that creates properly aligned partitions (Windows 2008 and later).

Best Practices:

Before I wrap this installment up, here are some best practices for storage alignment in your environment:

Create aligned partitions in your VMware templates. Do it once, do it right – every machine you deploy from the template will be aligned.
Use caution with tools like Symantec Ghost. Ghost can take images of aligned partitions and misalign them when laying down on a new system.
Use caution when performing P2V’s using VMware vCenter Converter – it does not align guest disks on import. You might consider using Converter to perform a P2V of the system disk only, then create new VMDK’s on the converted guest. Use Diskpart, gparted, or another tool to create aligned partitions on the new VMDK’s and finally copy the data over to the newly virtualized server using a tool like Robocopy, RichCopy, or rsync.
SSD’s are particularly sensitive to misalignment, leading to poor performance and excessive wear.
Local VMFS volumes created by the ESX installer are not aligned. If you are using an installer-created local VMFS for anything where performance matters, you might consider re-creating it through vCenter.
Watch out when attaching a data disk from an older VM to a new VM. For example, you are upgrading your SQL servers to Windows 2008 R2 from 2003. You decide to do a side-by-side upgrade, using the detach/attach method. You install (or better yet, deploy from template) a new Windows 2008 R2 VM, detach your databases from the old server, move your SQL data and log virtual disks from your 2003 VM to the new VM and attach the SQL DB’s on the new server. Those old VMDK’s may be misaligned! Consider using Robocopy, RichCopy or rsync to ensure an aligned disk.
Check your storage vendors best practices for your particular environment (OS, workload, SAN, etc.).
There is some debate on whether or not it is advised to align your OS partitions. There is no clear-cut answer on this as it depends so much on your environment and particular needs. For help in deciding if you should align your Guest OS drives, see the comments in the blogs by Duncan Epping, Vaughn Stewart, and Chad Sakac.
While working the VMware User Group booth at the Washington, DC Virtualization Forum 2010 I had a user ask me if rules and procedures for alignment on 4k sector disks are different. I forgot to research it until just now, so I honestly don’t know (please comment if you do know!). Check with your storage vendor if this is an issue for you.
Finally, you can’t realign partitions using tools like mbralign or vOptimizer in ESXi -Aaaron Delp explains the problem here: http://blog.aarondelp.com/2010/06/my-1-issue-with-vmware-esxi-today.html.

I hope this is helpful for you in understanding the problem of storage alignment and how it can impact your environment. Comments or questions are welcomed!

Storage Basics – Part VI: Storage Workload Characterization Most of what I covered in Storage Basics Parts 1...
Storage Basics – Part V: Controllers, Cache and Coalescing My Storage Basics series has been neglected for some time...
Storage Basics – Part III: RAID This is the third in a multi-part series on storage...

Free SAN Monitor for DS3300, MD3000i and others

Joshua Townsend — Fri, 07 May 2010 21:54:58 +0000

One of my most popular posts to date had been IBM DS3300 Write Performance Problem Solved. I am pleased to have upgrade my internal environment to an EMC Clariion CX4 array, but still have customers using the DS3300 and the MD3000i from Dell. For that reason, I keep an eye out for news on these arrays. You’ll recall that these arrays are based on the same LSI kit. Truth be told, properly designed and configured, these arrays are a technically proficient and cost efficient solution for SMB’s looking to get their VMware environments going.

While reviewing my email today I found an email from SolarWinds about their new Free SAN Monitor. I have used several products from SolarWinds in the past, and so thought I would give this a quick try. This tool can monitor the LSI-based arrays from Dell (including the MD3000 series), IBM (DS3000, DS4000, and DS5000), and SunStorageTek 2000 and 60000. Installation and configuration were both fast and easy. A wizard asked me for the array’s IP address, monitoring interval, and some thresholds (which I left blank). Once I clicked Finish the app launched on my desktop and immediately began to display data.

This screen-shot is of the app running on my Windows 7 x64 workstation. You’ll notice that the array is reporting performance, in terms of IOPS, and response time for individual LUNS. The health of my array is also displayed. You’ll see that my ‘Controller 2′ shows a failure. I do not have a dual-controller configuration (hence my wildly popular post on performance being in the tank due to a disabled write cache). The SAN Monitor incorrectly reported that I had two controllers, and that the 2nd is failed.

This is not bad for a free tool, in my opinion. I would love to see SolarWinds add some info on write cache status for the array (cache hit %, cache enabled per LUN, etc.) as well as LUN utilization in terms of free space remaining. For a free desktop GUI, it’s worth the 5 minutes to download and setup.

Reporting on I/O Response time (in milliseconds) is helpful in determining if your array is stressed. If response times are consistently above 15ms, you ought to look at adding some spindles. If you are over 25ms on a regluar basis, you’re likely feeling some pain on the servers generating the workload against your array.

If you are into the whole free tool thing, SolarWinds also has a free VM Monitor that is capable of monitoring a single ESX server via SNMP and it’s associated VM’s. It too runs as a desktop application, requiring minimal setup and configuration. While it won’t monitoring your whole VMware Essentials (3 ESX servers + vCenter) environment, it’s a start.

You might also check out Nick Weaver’s vSphere Mini Monitor for basic real-time alerts and monitors, as well as NetWrix’s Change Reporter for VMware. A final free (and nothing short of awesome) tool is Alan Renouf’s vCheck Daily Report PowerShell Script.

Do you have free tools, scripts or utilities for your VMware environment? I’d love to hear more about them in the comments section!

IBM DS3300 iSCSI Write Performance Solved I have been pulling my hair out with a small...
Guest Free Disk Space Revisited I wrote about a method for determining guest free disk...
Obtaining VMware Guest Disk Free Space for NFS Sizing I am often asked about sizing storage vis-à-vis how much...

Washington DC VMware User Group – April 29th

Joshua Townsend — Fri, 16 Apr 2010 14:21:02 +0000

It was great seeing so many VMware User Group (VMUG) members at this week’s Virtualization Forum in Washington, DC. There are several more Virtualization Forum‘s scheduled for North America, Europe, and Asia – these events are a great chance to see live demo’s, connect with experts, and catch a peek of some exciting technologies in the virtualization space. I hope to see everyone I met at the VMware booth at the next Washington, DC VMUG.

Our next event will be on Thursday, April 29th at the Sheraton Crystal City Hotel in Arlington, VA where we will be discussing best practices when Virtualizing Tier 1 Applications.

Here is the agenda for the meeting:

11:30 a.m.   Registration
11:50 a.m.   Opening Remarks
12:00 p.m.   Riverbed Presentation by Jim Borda
12:45 p.m.   Lunch
01:30 p.m.   What’s New by VMUG Leader Jason Langone
01:45 p.m.   Double-Take Presentation: “The Secret to Disaster Recovery for Any Application, Anywhere” by James Senecal
02:30 p.m.   Beverage Break
02:45 p.m.   Virtualization of Tier 1 Apps
03:30 p.m.   Wrap-up

This is a great opportunity to meet with your peers to discuss virtualization trends, best practices and the latest technology. Register today to join us for this free informative event. Space is limited, so respond as soon as possible to reserve your seat.

The official registration form can be found here: http://campaign.vmware.com/usergroup/invites/WashingtonDC_4-29-10.html.

Washington, DC VMUG – March 10, 2010 Please join us for the upcoming Washington DC VMware® User Group...
VMworld Content Catalog Released The VMworld 2009 Content Catalog was released on Friday night...
VMware vExpert I’ve held off on posting this as I don’t like...

Storage Basics – Part VI: Storage Workload Characterization

Joshua Townsend — Thu, 08 Apr 2010 16:34:44 +0000

Most of what I covered in Storage Basics Parts 1 through 5 was at a very elementary level. The math I used to do IOPS calculations, for example, is only true under very certain conditions. RAID controllers implement caching and other techniques that skew the simple math that I provided. I mentioned that the type of interface that you ought to use on your storage array should not be randomly chosen. In fact, choosing the right array with the appropriate components and characteristics can only be done when you enlighten your decision with a characterization of workloads it will be running.

The character of your storage workload can be broken down into several traits – random vs. sequential I/O, large vs. small I/O request size, read vs. write ratio, and degree of parallelism. The traits of your particular workload dictate how it interacts with the components of your storage system and ultimately determine the performance of your environment under a given configuration. There is an excellent whitepaper available from VMware entitled “Easy and Efficient Disk I/O Workload Characterization inVMware ESX Server” that is authoritative on this subject. If you want to get down and dirty with the topic, it’s a good read. I’m aiming for something a bit less academic. With that said, let’s break down workload characterization a bit so as to better understand how it will impact your real-world systems.

Random vs. Sequential Access

In Part II of this series we looked at the formula for calculating IOPS capabilities for a single disk. That formula goes something like this:

IOPS = 1000/(Seek Latency + Rotational Latency)

You’ll recall that we divide into 1000 to remove milliseconds from the equation, leaving (Seek Latency + Rotational Latency) as the important part of the equation. Rotational latency is based on the spindle speed of the disk – 7.2k, 10k, or 15k RPM for standard server or SAN disks. If we consider the same Seagate Cheetah 15k drive from Part II, we see that rotational latency is 2.0ms. The only way to change rotational latency is to buy faster (or slower) disks. This essentially leaves seek latency as the only variable that we can “adjust”. You’ll also recall that seek latency was the larger of the latencies (3.4ms for read seeks, and 3.9ms for write seeks) and counts more against IOPS capability than does rotational latency. Seeking is the most expensive operation in terms of performance.

It is next to impossible to adjust seek latency on a disk because it is determined by the speed of the servos that move the heads across the platter. We can, however, send workloads with different degrees of randomness to the platter. The more sequential a workload is, the less time that will be spent in seek operations. A high degree of sequentiality ultimately leads to faster disk response and higher throughput rates. Sequential workloads may be candidates for slower disks or RAID levels. Conversely, workloads that are highly randomized ought to be placed on fast spindles in fast RAID configurations.

You’ll notice that I said it was next to impossible to adjust seek latency on a disk. While not common, some storage administrators employ a method know as ‘short stroking’ when configuring storage. Short stroking uses less than the full capacity of the disk by placing data at the beginning of the disk where access is faster, and not placing data at the end of the disk where seeks times are greater. This results in a smaller area on the disk platter for heads to travel over, effectively reducing seek time at the expense of capacity.

While not applicable to all workloads, storage arrays, or file systems, fragmentation can cause higher degrees of randomness leading to degraded performance. This is the prime reason some vendors recommend that you regularly defragment your file system. It should be noted that a VMware VMFS file system is resilient against the forces of fragmentation. Whereas a Windows NTFS parition may hold hundreds, thousands or tens of thousands of files of different sizes, accessed randomly throughout the system’s cycle of operations, a VMFS datastore typically holds no more than a couple hundred files. Additionally, most of the files on a VMFS datastore are created contiguously if you are using thick-provisioned virtual disks (VMDK). Thin-provisioned VMDK’s are slightly more susceptible to fragmentation, but do not typically suffer a high enough degree of fragmentation to register a performance impact. See this VMware whitepaper for more on VMFS fragmentation: Performance Study of VMware vStorage Thin Provisioning.

Examples of sequential workloads include backup-to-disk operations and the writing of SQL transaction log files. Random workloads may include collective reads from Exchange Information Stores or OLTP database access. Workloads are often a mix of random and sequential access, as is the case with most VMware vSphere implmentations. The degree to which they are random or sequential dictates the type of tuning you should perform to obtain the best possible performance for your environment.

I/O Request Size

I/O request size is another important factor in workload characterization. Generally speaking, larger reads/writes are more efficient than smaller I/O to a certain point. The use of larger I/O requests (64KB instead of 2KB, for example) can result in faster throughput and reduced processor time. Most workloads do not allow you to adjust your I/O request size. However, knowing your I/O request size can help with appropriate configuration of certain parameters such as array stripe size and file system cluster size. Check with your storage vendor for more information as it pertains to your specific configuration.

If you are in a Windows shop, you can use perfmon counters such as Avg. Disk Bytes/Read to determine average I/O size. If you are running a VMware-virtualized workload, you can take advantage of a great tool – vscsiStats – to identify your I/O request size. More on vscsiStats later in this article.

Read vs. Write

Every workload will display a differing amount of read and write activity. Sometimes a specific workload, say Microsoft Exchange, can be broken down into sub-workloads for logging (write-heavy) and reading the database (read-heavy). Understanding the read-to-write ratio may help with designing the underlying storage system. For example, a write-heavy workload may perform better on a RAID10 LUN than a RAID5 array due to the write penalty associated with RAID5. The ratio of read:write may also dictate caching strategies. The read:write ratio, when combined with a degree of randomness measure, can be quite useful in architecting your storage strategy for a given application or workload.

Parallelism/Outstanding I/O’s

Some workloads are capable of performing multi-threaded I/O. These types of workloads can place a higher amount of stress on the storage system and should be understood when designing storage, both in terms of IOPS and throughput. Multipathing may help with multi-threaded I/O workloads. A typical VMware vSphere environment is a good example of a workload capable of queuing up outstanding I/O.

Measuring the Characteristics of Your Workload

So how do we actually characterize storage workloads? Start with the application vendor – many have published studies that can shed light on specific storage workloads in a standard implementation. If you are interested in measuring your own for planning/architecture reasons, or performance troubleshooting reasons, read on…. There are several tools to measure storage characteristics, depending on your operating system and storage environment. Standard OS performance counters, such as Windows Performance Monitor (perfmon) can reveal some of the characteristics. Array based tools such as NaviAnalyzer on EMC gear can also reveal statistics on the storage end of the equation.

One of the most exciting tools for storage workload characterization comes from VMware in the form of vscsiStats. vscsiStats is a tool that has been included in VMware ESX server since version 3.5. Because all I/O commands pass through the Virtual Machine Monitor (VMM), the hypervisor can inspect and report on the I/O characteristics of a particular workload, down to a unique VM running on an ESX host. There is a ton of great information on using vscsiStats, so I won’t re-hash it all here. I recommend starting with Using vscsiStats for Storage Performance Analysis as it contains an overview and usage instructions. If you want to dig a bit deeper into vscsiStats, read both Storage Workload Characterization and Consolidation in Virtualized Environments and vscsiStats: Fast and Easy Disk Workload Characterization on VMware ESX Server.

vscsiStats can generate an enormous amount of data which is best viewed as a histogram. If you’re a glutton for punishment, the data can be reviewed manually on the COS. To extract vscsiStat output data, use the -c option to export to a .csv file. From there you can analyze the data and create histograms using Excel. Paul Dunn has a nifty Excel macro for analyzing and reporting on vscsiStats output here. Gabrie van Zanten more detailed instructions for using Paul’s macro here. Here are a couple histogram examples that I just generated from a test VM.

vscsiStats is only included with ESX, not ESXi. However, Scott Drummond was kind enough to post a download of vscsiStats for ESXi on his Virtual Pivot blog: http://vpivot.com/2009/10/21/vscsistats-for-esxi/. Using vscsiStats on ESXi requires dropping into Tech Support Mode (unsupported) and enabling ESXi for scp to transfer the binary to the ESXi server.

VMware esxtop can display some information but is limited in scope and does not currently support NFS. A community-supported python script called nfstop can parse vscsiStats data and display esxtop-like data per VM on screen.

Experiment

If you are interested in generating workloads with various characteristics, check out Iometer and Bonnie++. These tools will allow you to generate I/O that you can monitor with the tools I covered in this article.

Put it to Use

If you are provisioning a new workload or expanding an existing, invest some time in understanding your storage workload characteristics and convey those characteristics to your storage team. A request for storage that includes the workload characteristics I discussed here, as well as expected IOPS requirements, will go much further in ensuring performance for your applications – physical or virtual – than simply asking for a certain capacity of disk.

Storage Basics – Part V: Controllers, Cache and Coalescing My Storage Basics series has been neglected for some time...
Storage Basics – Part II: IOPS In Part I of this series, I discussed the important...
Storage Basics – Part VII: Storage Alignment Storage alignment, or the lack of alignment, is not a...

Update: SVGA Drivers on Windows 2008 R2 and Windows 7

Joshua Townsend — Mon, 29 Mar 2010 00:23:43 +0000

I posted an article in December on how the SVGA driver included with VMware Tools caused the guest VM to freeze. I referenced VMware’s KB Article 1011709, which directed you to not use the SVGA drivers included with VMware Tools. KB1011709 has since been updated (as of February 25, 2010) to indicate that the VMware Tools package included with ESX 4.0 Update 1 includes a new WDDM driver that is fully supported. If you have updated to Update 1, you should upgrade VMware Tools to take advantage of the new driver.

If you followed the KB1011709′s original advice and did a custom install of VMware Tools (leaving out the SVGA driver through a custom install), you may have to do a re-install of VMware Tools before the new driver is available. Once you get VMware Tools upgraded, the new driver can be found in the guest VM at C:\Program Files\Common Files\VMware\Drivers\wddm_video. These drivers are not automatically installed, so you’ll have to update your guest’s video adapter driver in Device Manager.

It’s a bummer that the WDDM SVGA drivers are not automatically installed. You could probably copy these drivers to other VM’s and use Windows Device Manager to replace the standard driver with the newer WDDM driver without having to do the uninstall, reboot, reinstall of VMware tools on all of your VM’s.

Just as I was about to publish this, I saw a TweetDeck pop-up from @jasonboche saying that he had published pretty much the same update here: http://www.boche.net/blog/index.php/2010/03/28/windows-2008-r2-and-windows-7-on-vsphere/. Not only does he have pretty pictures to go with his post, but also points out that VMware Tools installs/upgrades executed with VMware Update Manager (VUM) will not install the upgraded SVGA driver. He also recommends updating templates to include the upgraded drivers. Great points, Jason.

Windows Server 2008 R2 & Windows 7 Freeze When Using SVGA Drivers I recently ran into an issue when installing my first...
Networking Problems with ESX 3.5 Update 3 on the DL380 G3 I began building up a new lab environment with a...
Installing PowerPath/VE using VMware Update Manager I am finishing up an installation of an EMC Clariion...

Storage Basics – Part V: Controllers, Cache and Coalescing

Joshua Townsend — Tue, 23 Mar 2010 19:37:13 +0000

My Storage Basics series has been neglected for some time (sick kids, snow storms, VMware Upgrades, SAN implementations and some Cisco switch upgrades took all my free time), so let’s jump right in to Part V – Cache, Controllers, and Coalescing. Between the alliteration and fancy words, it might seem like I am about to tell a tale of international espionage. Unfortunately, my introductory treatment of these aspects of a storage system will probably not keep you on the edge of your seat – but I’ll try to keep it interesting.

Throughout this series, we’ve been working our way from the basic building block of any storage system – the disks – outwards towards the brains of the operation – the controller. You’ll recall that in Part II I introduced IOPS and the math that goes into calculating the IOPS capacity of a disk array. In Part III we considered a RAID implementation’s impact on performance and availability. And most recently in Part IV we looked at the common interface types when dealing with storage arrays. If we put the previous parts together we still don’t have a functional storage system. The missing piece is the controller. Simply put, the storage controller is the hardware adapter between the disks and the servers that connect to the storage. The controller has a specific ‘interface‘ type, is responsible for RAID operations, and handles advanced storage functionality. A controller can be as simple as the Dell PERC or HP Smart Array add-in card on your server, or as complex as the Storage Processor in an enterprise class Storage Area Network (SAN) such as an EMC CLARiiON or NetApp FAS.

Controllers

As we look at controllers and the advanced features they provide we’ll see that some of the earlier performance equations start to break down. The simplest controllers take disk read/write commands from the operating system and send commands down to the disk(s) attached to be read or written. This gets data onto the disk, but often does not do so in an efficient or reliable manner. RAID-capable controllers take on the added responsibility of configuring disks in the desired RAID level, calculating & writing parity data, and writing the data in disk-spanning stripes or mirrors depending on the RAID level.

Cache

To increase performance and improve reliability, storage vendors implement a caching system on their controllers. Cache is memory that acts as a buffer for disk I/O, and is usually battery-backed to prevent data loss in the event of a power failure. Because of the exponentially greater speed of RAM over spinning magnetic disks, cache can improve performance by orders of magnitude. Cache can operate on both reads and writes to disk.

When dealing with writes, the controller cache is typically used in one of two ways: write-through or write-back. In write-through mode, data is written to volatile cache and then to disk, and only acknowledged as written once the data resides on the non-volatile disk. Write-back mode allows the controller to acknowledge the data as having been written as soon as it is held in cache. This allows the cache to buffer writes quickly and then write them to the slower disk when the disk has cycles to accept I/O. The greater your cache size, the more data that can be buffered, ultimately resulting in better performance as measured in both IOPS and throughput. This graph from my article on troubleshooting write performance on an IBM DS3300 iSCSI array shows how throughput increased and latency decreased when enabling write cache. The extent to which cache increases performance is highly dependent on the workload characteristics (I/O size, randomness, and ratio of reads:writes).

Read-cache acts as a buffer for reads in a couple ways. First, some controllers attempt to ‘read-ahead’, anticipating future read requests from the operating system and buffering what it expects to be the next blocks of desired data. Some entry-level controllers simply buffer the next physical chunk of data and fill cache memory with it, while more advanced controllers may attempt to predict the right block of data based on previous requests (you just asked for 3 blocks in a row, I’m guessing you’ll come asking for the 4th next so I’ll just buffer it in fast cache for you now). Secondly, read cache holds data that has been previously read, regardless of any pre-fetching the controller may have done. This allows for much faster subsequent access of the same data because it is held in the faster cache, eliminating the need for the controller to go to disk for the data again. Just like with write cache, the extent to which cache increases performance is highly dependent on the workload characteristics.

A given storage array controller only has so much cache to work with. A Dell PERC5/E, for example, has 256MB of cache that can be used for both read and write. While this may be enough for a direct-attached storage array, SAN’s serving multiple systems demand more cache. In contrast, an EMC CLARiiON CX4-960 has 32GB. Some storage vendors, such as NetApp, are getting creative with cache. NetApp’s Performance Acceleration Module (PAM) is an add-in card that provides up to a whopping 512GB of Layer 2 cache to the storage system.

Caching mechanisms can dramatically influence performance under the right conditions. With healthy cache in place, IOPS calculations become skewed. However, cache can be exhausted or may not hold the data you are interested in. If cache is insufficient to satisfy read requests, or has reached its high-water mark for writes, performance can drop off. When cache is exhausted, the backing disk must be able to satisfy the I/O workload or performance will be unacceptable. This is where the IOPS calculations kick in, and where having the right disk type and configuration really matters.

Queuing & Coalescing

Advanced storage systems introduce additional features to reduce I/O contention and improve cache utilization. I won’t go into all of the features here because they vary by storage vendor. However, I will point out two common techniques – queuing and coalescing.

Queuing refers to the ability of a storage system to queue storage commands for later processing. Queuing can take place at various points in your storage environment, from the HBA to the storage processor/controller. A little queuing may be OK depending on your workload, but too many outstanding I/Os can negatively impact performance (this is measured in latency). Queue depths can be adjusted on many components in your storage and VMware landscape, but check with your vendor’s support group before you make changes to these settings.

Coalescing is performed by some storage systems to modify the character of the workload. To better understand coalescing, picture a bunch of random write activity. Without cache in place, the disk heads will be bouncing all over the platters trying to get the data on to disk. A little write cache will allow the storage array to acknowledge the write for the OS, but the array still needs to de-stage that data from cache to disk quickly to prevent cache exhaustion. The back-end disks will still be doing the chicken dance, bouncing around trying to write the random workload…. Now picture an intelligent system that re-orders the random writes that are held in cache and writes them to the disk in nice sequential stripes. The disk heads will be less prone to jumping around the platter and the behavior will start to look more like a nice waltz than the funky chicken dance. Coalescing is used for writes, not reads, so not all workloads benefit.

Wrap-up

With this article on Controllers, Cache, and Coalescing we’ll end our look at the basic building blocks of a storage array. Before we end the Storage Basic series I am planning a few more articles on Storage Workload Characterization (which has been mentioned, but not directly addressed in this and previous articles), Identifying a Stressed Storage System, and Best Practices for Storage Performance in a VMware Environment.

If you are interested in more reading on Controllers, Cache, and Coalescing, I recommend the following:

Additional Reading:

Impact of cache on the performance of the HP StorageWorks XP12000 Disk Array white paper
Performance impact of controller cache: SQL Server read only workloads
IOps? – Dig into the article’s comments for some great dialog between some people who really know their stuff!
Storage Performance for SQL Server
Storage Caching 101 – Chuck Hollis (EMC)
Improving Performance with Interrupt Coalescing for Virtual Machine Disk IO in VMware ESX Server

Storage Basics – Part VI: Storage Workload Characterization Most of what I covered in Storage Basics Parts 1...
Storage Basics – Part III: RAID This is the third in a multi-part series on storage...
Storage Basics – Part IV: Interface In parts I, II, and III of the Storage Basics...

Washington, DC VMUG – March 10, 2010

Joshua Townsend — Tue, 02 Mar 2010 02:43:52 +0000

Please join us for the upcoming Washington DC VMware^® User Group meeting on Tuesday, March 10th, 10:30 a.m. – 3:00 p.m.

This is a great opportunity to meet with your peers to discuss virtualization trends, best practices and the latest technology.

Agenda
10:30 a.m. Check-In
10:50 a.m. Opening Remarks
11:00 a.m. Compellent Presentation
11:45 a.m. Lunch
12:30 p.m. VMware ThinApp Presentation by Jason Langone
01:30 p.m. Beverage Break
01:45 p.m. Veeam Presentation
02:30 p.m. Wrap-up session

Register today to join us for this free informative event. Space is limited, so respond as soon as possible to reserve your seat.

Location:
Westin City Center
National Ballroom A
1400 M Street NW
Washington, DC 20005
Map Here

Installing PowerPath/VE using VMware Update Manager

Joshua Townsend — Fri, 05 Feb 2010 19:17:07 +0000

I am finishing up an installation of an EMC Clariion CX4 SAN. One of the final steps of the installation is to configure PowerPath/VE on the ESXi hosts. PowerPath/VE is EMC’s multipathing extension module for VMware (and Hyper-V), designed to replace the Native Multipathing Plugin (NMP) for increased I/O performance and failover management. To simplify and automate the installation of PowerPath/VE, I decided to use VMware Update Manager (VUM) to push the extension to the ESXi 4.x hosts in the environment.

The process of setting up an additional VUM patch repository to host PowerPath/VE (and other 3rd party extensions such as the Cisco Nexus 1000v) is pretty straight forward. 3rd party extensions are supported in VUM beginning with vSphere 4.0 Update 1. Chad Sakac has posted a great video guide on YouTube that covers the setup:

I opted to use the tomcat installation on the environment’s vCenter server to host the PowerPath/VE repository. To accomplish this, I simply created a new directory in the tomcat root directory. The default path for the root directory on a vSphere vCenter Server is “C:\Program Files\VMware\Infrastructure\tomcat\webapps” (or C:\Program Files (x86)\VMware\Infrastructure\tomcat\webapps on a 64-bit installation).

I created a directory named ‘depot’ and within that directory created a PowerPathVE folder. I extracted the contents of the VUM folder from the PowerPath .zip file that I downloaded from http://powerlink.emc.com. A screenshot of the directory is below:

PowerPath/VE Depot Directory Tree

After creating the directory for the patch repository, I simply added an Extension Repository to VMware Update Manager as Chad shows in his video. I would like to call out one caveat – Because vCenter may not listen on standard HTTP/HTTPS ports, I used https://vcenter.domain.local:8443/depot/PowerPathVE/index.xml as the path to the source.

VUM Patch Source

Once PowerPath was added to an Extension Baseline in VUM, I simply had to scan my hosts for updates and remediate. Installation of PowerPath/VE requires the host to be in Maintenance Mode and concludes with a reboot. Pretty simple.

Then all you have to do is fight through an overly-complex licensing setup (seriously, a 112 page PDF on how to install licenses???), a bit of configuration, and you are multi-pathing with the best of them. If you are interested in learning more about PowerPath/VE, start with this whitepaper: EMC PowerPath/VE for VMware vSphere Best Practices Planning. For a bit of real-world insight into the performance increase you might see with PowerPath/VE, check out this blog post from Eric Sloof: Massive I/O power increase using EMC PowerPath/VE.

Update – 3/27/09: VMware published a Knowledge Base article on this procedure a few weeks after I wrote this post. You can find it in article 1018740.

EMC Virtual Storage Integrator Update I upgraded my in-house VMware vSphere environment to 4.0 Update...
Networking Problems with ESX 3.5 Update 3 on the DL380 G3 I began building up a new lab environment with a...
Virtualization Bookmarks for August 28th Here are some bookmarks for resources that I have recently...

Storage Basics – Part IV: Interface

Joshua Townsend — Wed, 27 Jan 2010 02:38:59 +0000

In parts I, II, and III of the Storage Basics series we looked at the basic building blocks of modern storage systems: hard disk drives. Specifically, we looked at the performance characteristics of disks in terms of IOPS and the impact of combining disks into RAID sets to improve performance and resiliency. Today we will have a quick look at another piece of the puzzle that impacts storage performance: the interface. The interface, for lack of a better term, can describe several things in a storage conversation. It can be let me break it down for you (remember, we’re keeping it simple here).

At the most basic level (assume a direct-attached setup), ‘interface’ can be used to describe the physical connections required to connect a hard drive to a system (motherboard/controller/array). The ‘interface’ extends beyond the disk itself, and includes the controller, cabling, and disk electronics necessary to facility communications between the processing unit and the storage device. Perhaps a better term for this would be ‘intra-connect’ as this is all relative to the storage bus. Common interfaces include IDE, SATA, SCSI, SAS, and FC. Before data reaches the disk platter (where it is bound by IOPS), it must pass through the interface. The standards bodies that define these interfaces go beyond the simple physical form factor; they also define the speed and capabilities of the interface, and this is where we find another measure of storage performance: throughput. The speed of the interface is the maximum sustained throughput (transfer speed) of the interface and is often measured in Gbps or MBps.

Here are the interface speeds for the most common storage interfaces:

Interface	Speed
IDE	100MBps or 133MBps
SATA	1.5Gbps or 3.0Gbps (6.0Gbps is coming)
SCSI	160MBps (Ultra-160) and 320MBps (Ultra-320)
SAS	1.5Gbps or 3.0Gbps (6.0Gbps is coming)
FC	1Gb, 2Gb, 4Gb, or 8Gb (Duplex throughput rates are 200MBps, 400MBps, 800MBps, and 1600MBps respectively)

If we take these speeds at face value, we see that a 320MBps SCSI and a 2Gbps FC are not too different. If you dig a bit deeper you will soon find that simple speed ratings are not the end of the story. For example, FC throughput can be impacted by the length and type of cable (fiber channel can use twisted pair copper in addition to fiber optic cables). Also, topologies can limit speeds – serial connected topologies are more efficient than parallel on the SCSI side, and arbitrated loops can incur a penalty on the FC side. The specifications of each interface type also define capabilities such as the protocol that can be used, the number of devices allowed on a bus, and the command set that can be used in communications on a storage system. For example, SATA native command queuing (NCQ) can offer a performance increase over parallel ATA’s tagged command queuing with other factors held constant. Because of this, you might also see some performance implications of connecting a SATA drive to a SAS backplane, as the SAS backplane translates SAS commands to SATA.

If we move away from the direct-connect model, and into a shared storage environment that you might use in a VMware-virtualized environment, the ‘interface’ takes on an additional meaning. You certainly still have the bus ‘interface’ that connects your disks to a backplane. Modern arrays typically use SAS or FC backplanes. If you have multiple disk enclosures, you also have an interface that connects each disk shelf to the controller/head/storage processor, or to an adjacent tray of disks. For example, EMC Clariion’s use a copper fiber channel cable in a switched fabric to connect disk enclosures to the back-end of the storage processors.

If we move to the front-end of the storage system, ‘interface’ describes the medium and protocol used by initiating systems (servers) when connecting to the target SAN. Typical front-end interface mediums on a SAN are Fiber Channel (FC) and Ethernet. Front-end FC interfaces come in the standard 2Gb, 4Gb, or 8Gb speeds, while Ethernet is 1Gbps or 10Gbps. Many storage arrays support multiple front-end ports which can be aggregated for increased bandwidth, or targeted by connecting systems using multi-pathing software for increased concurrency and failover.

Various protocols can be sent over these mediums. VMware currently supports Fiber Channel Protocol (FCP) on FC, and iSCSI and NFS on Ethernet. FC and iSCSI are block-based protocols that utilize encapsulated SCSI commands. NFS is a NAS protocol. Fiber Channel over Ethernet (FCoE) is also available on several storage arrays, sending FCP packets across Ethernet.

Determining which interface to use on both the front-end and back-end of your storage environment requires an understanding of your workload and your desired performance levels. A post on workload characterization is coming in this series, so I won’t get too deep now. I will, however, provide a few rules of thumb. First, capture performance statistics: using Windows Perfmon, look at Physical Disk|Disk Read Bytes/sec or Disk Write Bytes/sec), or check out stats in your vSphere Client if you are already virtualized.

If you require low latency, use fiber channel.
If your throughput is regularly over 60MBps, you should consider fiber channel connected hosts.
iSCSI or NFS are often a good fit for general VMware deployments.

There is a ton of guidance and performance numbers available when it comes to choosing the right interconnect for a VMWare deployment, and a ton of variables that impact performance. Start with this whitepaper from VMware: http://www.vmware.com/resources/techresources/10034. For follow up reading, check out Duncan Epping’s post with a link to a NetApp comparison of FC, iSCSI, and NFS: http://www.yellow-bricks.com/2010/01/07/fc-vs-nfs-vs-iscsi/. If you are going through a SAN purchase process, ask your vendor to assist you in collecting statistics for proper sizing of your environment. Storage vendors (and their resellers) have a few cool tools for collecting and analyzing statistics – don’t be afraid to ask questions on how they use those tools to recommend a configuration for you.

I’ve kept this series fairly simple. Next up in this series is a look at cache, controllers and coalescing. With the next post we’ll start to get a bit more complex and more specific to VMware and Tier 1 workloads, both virtual and physical. Thanks for reading!

Storage Basics – Part III: RAID This is the third in a multi-part series on storage...
Storage Basics – Part V: Controllers, Cache and Coalescing My Storage Basics series has been neglected for some time...
Storage Basics – Part VI: Storage Workload Characterization Most of what I covered in Storage Basics Parts 1...

Right-sizing Your Power and Cooling

Joshua Townsend — Thu, 21 Jan 2010 23:30:01 +0000

We all know that virtualization allows us to do more with less. Fewer servers and space-saving storage (talk about an oxymoron) help us put some green in the datacenter and back in the budget. But with tight budgets demanding greater efficiency, virtualization pushing per-U-space utilization higher, and increasingly rack-dense equipment, proper planning of your physical plant remains an essential part of IT. I argue that right-sizing your power, cooling, and floor-space is more critical now than it has ever been, and is a knowing how to do it is a darn good skill for a virtualization engineer to possess.

So along those lines… I was just doing some site-prep work for a new Clariion installation and noticed that the EMC Power Calculator has been updated. It is now a pretty slick little web app that can be found on the PowerLink site (login required) here: https://powerlink.emc.com/nsepn/webapps/powercalculator/Main.aspx.

While I am at it, here are some links to other power consumption calculators. Let me know if you have others and I will update this post:

Dell: http://www.dell.com/calc
IBM: http://www-03.ibm.com/systems/bladecenter/resources/powerconfig/index.html
NetApp: Storage Efficiency Calculator here - http://www.secalc.com – it doesn’t calculate your consumption, just what you might save over a competitor’s offering.
HP: http://h30099.www3.hp.com/configurator/powercalcs.asp
Sun: http://www.sun.com/solutions/eco_innovation/powercalculators.jsp
Hitachi/HDS: http://www.byhitachi.com/se/go/weight-and-power-calculator/
APC: http://www.apc.com/prod_docs/results.cfm?DocType=Trade-Off%20Tool&Query_Type=10 and http://www.apcc.com/products/runtime_for_extendedruntime.cfm?upsfamily=165
Emerson: Efficiency Calculator: http://www.emerson.com/edc/Calculator/default.aspx
VMware ROI Calculator: http://vmware.com/go/calculator
This site has a bunch of links to other calculators and resources: http://thegreenandvirtualdatacenter.com/calculator.html

There’s some fun and timely chatter happening right now on Twitter around power consumption and sizing – join in by following me at http://twitter.com/joshuatownsend/!

Obtaining VMware Guest Disk Free Space for NFS Sizing I am often asked about sizing storage vis-à-vis how much...
NetApp Extends 50% Virtualization Guarantee NetApp has extended their 50% Virtualization Guarantee to include Citrix...
EMC Virtual Storage Integrator Update I upgraded my in-house VMware vSphere environment to 4.0 Update...

Storage Basics – Part III: RAID

Joshua Townsend — Wed, 06 Jan 2010 23:24:26 +0000

This is the third in a multi-part series on storage basics. I’ve had some good feedback from folks in the SMB space saying that the first couple posts in this series have been beneficial, so we’ll be sticking with some basic concepts for another post or two before we dive into some nitty-gritty details and practical applications of these concepts in a VMware environment. In the second post of this series I introduced the concept of IOPS and explained how the physical characteristics of a hard disk drive determine the theoretical IOPS capability of a disk. I then noted that you can aggregate disks to achieve a greater number of IOPS for a particular storage environment. Today, we will look at just how you combine multiple disks and the performance impact of doing so. Remember that we are keeping this simple; the concepts I present here may not apply to that fancy new SAN you just purchased with your end-of-year money or the cheap little SATA controller on your desktop’s motherboard (not that there’s anything wrong with it) – we’re more in the middle ground of direct attached storage (DAS) as we firm up concepts.

Enterprise servers and storage systems have the ability to combine multiple disks into a group using Redundant Array of Independent Disks (RAID) technology. We’ll assume a hardware RAID controller is responsible for configuring and driving storage IO to the connected disks. RAID controllers typically have battery-backed cache (we’ll talk cache in a future post), an interconnect where the drives plug in, such as SCSI or SAS (we’ll talk about these too in a future post), and hold the configuration of the RAID set including stripe size and RAID level. The controller also does the basic work of reading and writing on RAID set – mirroring, striping, and parity calculations. There are several different types of RAID level – rather than rehash the details of them, read the Wikipedia entry on RAID and then come back here….

Ok, great. So you now know that RAID is implemented to increase performance through the aggregation of multiple disks, and to increase reliability though mirroring and parity. Now let’s consider the performance implications of some basic RAID levels. As with many things in the IT industry, there are trade-offs: security vs. usability, brains vs. brawn, and now performance vs. reliability. As we increase reliability in a RAID array through mirroring and parity, performance can be impacted. This is where the more disks = more IOPS bit starts to fall apart. The exact impact depends on the RAID type. Here are some examples of how RAID impact the maximum theoretical IOPS using the most common RAID levels, where:

I = Total IOPS for Array (note that I show Read and Write separately)

i = IOPS per disk in array (based on spindle speed averages from Part II: IOPS)

n = Number of disks in array

r = Percentage of read IOPS (calculated from the Average Disk Reads/Sec divided by total Average Disk Transfers/Sec in your Windows Perfmon)

w = Percentage of write IOPS (calculated from the Average Disk Writes/Sec divided by total Average Disk Transfers/Sec in your Windows Perfmon)

RAID0 (striping, no redundancy)

This is basic aggregation with no redundancy. A single drive error/failure could render your data useless and as such it is not recommended for production use. It does allow for some simple math:

I =n*i

Because there is no mirroring or parity overhead, theoretical maximum Read and Write IOPS are the same.

RAID 1 & RAID10 (mirroring technologies):

Because data is mirrored to multiple disks

Read I = n*i

For example, if we have six 15k disks in a RAID10 config, we should expect a theoretical maximum number of IOPS for our array to be 6*180 = 1080 IOPS

Write I = (n*i)/2

RAID5 (striping with a single parity disk)

Read I = (n-1)*i

Example: Five 15k disks in a RAID 5 (4 + 1) will yield a maximum IOPS of (5-1)*180 = 720 READ IOPS. We subtract 1 because one of the disks holds parity bits, not data.

Write I = (n*i)/4

Example: Five disks in a RAID 5 (4 + 1) will yield a maximum IOPS of (5*180)/4 = 225 WRITE IOPS

Again, these formulas are very basic and have little practical value. Furthermore, it is seldom that you will find a system that is doing only reads or only writes. More often, as is the case with typical VMware environments, reads and writes are mixed. An understanding of your workload is key to accurately sizing your storage environment for performance. One of the workload characteristics (we’ll explore some more in the future) that you should consider in your sizing is the percentage of read IOPS vs. the percentage of write IOPS. A formula like this gets you close if you want to do the math for a mixed read/write environment in a RAID5 set:

I = (n*i)/(r+4 *w)

Example: a 60% read/40% write workload with five 15k disks in a RAID5 would provide (5*180)/(.6+4*.4) = 409 IOPS.

The previous examples have all been from the perspective of the storage system. If we take a look at this from the server/OS/application side, something interesting shows up. Let’s say you fired up Windows perfmon and collected Physical Disk Transfers/sec counters every 15 seconds for 24 hours and analyzed the data in Excel to find the 95th Percentile for total average IOPS (this is a pretty standard exercise if you are buying enterprise storage array or SAN). Let’s say that you find that the server in question was asking for 1000 IOPS at the 95th Percentile (let’s stick with our 60% read/40% write workload). And finally, let’s say we put this workload on a RAID5 array. That’s saying a lot of stuff, but what does it all mean? Because RAID5 has a write penalty factor of 4 (again, Duncan Epping’s posted a great article here which I referenced in Part II that describes this in a slightly different way) we can tweak the previous formula to show the IO’s to the backend array given a specific workload.

I = Target workload IOPS

f = IO penalty

r = % Read

w = % Write

IO = (I * r) + (I * w) * f

Our example then looks like this (remember work inside parenthesis first, and then My Dear Aunt Sally):

(1000 * .6) + ((1000 * .4) * 4) = 2200

Simply stated, this means that for every 1000 IOPS that our workload requests from our storage system, the backing array perform 2200 IO’s, and it better do it quickly or you will start to see latency and queuing (we call this performance degradation, boys and girls!). Again, this is a very simplistic approach neglecting factors like cache, randomness of the workload, stripe size, IO size, and partition alignment which can all impact requirements on the backend. I’ll cover some of those later.

As you can hopefully see, the laws of physics combined with some simple math can provide some pretty useful numbers. A basic understanding of your array configuration against your workload requirements can go a long way in preventing storage bottlenecks. You may also find that as you consider the cost per disk against various spindle speeds, capacities and RAID levels that you are better off buying smaller, faster, fewer, more, slower…. disks depending on your requirements. The geekier amongst us could even take these formulas and some costs per disk and hit up Excel Goal Seek to find the optimal level, but that’s more than this little blog can do for you today.

Before I wrap up this post, I want to leave you with a few more links that I have bookmarked around the topics of IOPS and RAID over the past several years:

DB sizing for Microsoft Operations Manger, includes a nice chart with formulas similar to the ones I provided in this article: http://blogs.technet.com/jonathanalmquist/archive/2009/04/06/how-can-i-gauge-operations-manager-database-performance.aspx
An Experts Exchange post with some good info in the last entry on the page (subscription required) http://www.experts-exchange.com/Storage/Storage_Technology/Q_22669077.html
A Microsoft TechNet article with storage sizing for Exchange – a bit dated but still applicable: http://technet.microsoft.com/en-us/library/aa997052(EXCHG.65).aspx
A simple whitepaper from Dell on their MD1000 DAS array – easy language to help the less technical along: http://support.dell.com/support/edocs/systems/md1120/multlang/whitepaper/SAS%20MD1xxx.pdf
A great post that uses some math to show performance and cost trade-offs of RAID level, disk type, and spindle speed. http://www.yonahruss.com/architecture/raid-10-vs-raid-5-performance-cost-space-and-ha.html
Another nifty post that looks at cost vs. performance vs capacities of various disks speeds in an array http://blogs.zdnet.com/Ou/?p=322

Storage Basics – Part V: Controllers, Cache and Coalescing My Storage Basics series has been neglected for some time...
Storage Basics – Part IV: Interface In parts I, II, and III of the Storage Basics...
Storage Basics – Part VI: Storage Workload Characterization Most of what I covered in Storage Basics Parts 1...

Vote Now for Top Virtualization Bloggers

Joshua Townsend — Mon, 04 Jan 2010 18:48:38 +0000

Happy New Year to everyone! 2010 is shaping up to be quite a good year, both personally and professionally. Between two little boys at home and a bunch of new projects at work I should stay busy. I will also continue to co-lead the Washington, DC VMware User Group (VMUG). In my spare time I will continue to write what I hope are technically sound, practical, and timely articles on VMtoday.com.

And speaking of VMtoday.com… Eric Siebert has opened up voting for the top 25 virtualization bloggers on his vSphere-Land.com site. I am very honored to be included in the ballot list of 55 of our industry’s top bloggers. Please take a few minutes to vote – a couple lucky voters will win a copy of TrainSignal’s VMware vSphere DVD training course.

Thanks for reading and best wishes for a blessed and productive 2010!

~Josh

NetApp Extends 50% Virtualization Guarantee NetApp has extended their 50% Virtualization Guarantee to include Citrix...
VMtoday’s One Year Anniversary Today marks the one year anniversary of my first post...
Landing a Virtualization Job As the New Year rolls around you may find yourself...

Storage Basics – Part II: IOPS

Joshua Townsend — Tue, 29 Dec 2009 12:50:59 +0000

In Part I of this series, I discussed the important of storage performance in a virtual environment (really any environment, virtual or not, where you want acceptable performance), and introduced some of the basic measures of a storage environment. In Part II, we will look more closely at what may be the most important storage design consideration in a VMware server-consolidation enviornments, many SQL environments, and VDI environments to name a few: IOPS.

If we stick with a single-disk-centric approach as we did in Part I, IOPS is quite simply a measure of how many read and write commands a disk can complete in a second. IOPS is an important measure of performance in a shared storage environment (such as VMware) and in high-transaction-rate workloads like SQL. Because hard drives are forced to abide by the laws of physics, the IOPS capabilities of a disk are consistent and predictable given a specific configuration. The formula for calculating IOPS for a given disk is pretty straight forward (please show your work):

IOPS = 1000/(Seek Latency + Rotational Latency)

Exact latencies vary by disk type, quality, number of platters, etc. You can look up the tech specs for most drives on the market. As an example, I have randomly chosen the technical specifications of the Seagate Cheatah 15k.7 SAS drive. This particular drive has the following performance characteristics:

- Average (rotational) latency: 2.0msec

- Average read seek (latency): 3.4msec

- Average write seek (latency): 3.9msec

Using the read latency number, the math works out like this:

1000
———- = 185 maximum read IOPS
2.0+3.4

The maximum write IOPS will be a bit less (~169IOPS) because of the higher write seek latency. Writing is more ‘expensive’ than reading and therefore slower.

Fortunately, there are some widely accepted ‘working’ numbers, so you do not have to use this formula for each and every disk you might consider using. Because rotational latency is based on the rotational speed, we can use the published Rotations Per Minute (RPM) rating of the drive to guess-timate the IOPS capabilities. Typical spindle speeds (measured in RPM) and their equivalent IOPS are in the table below.

RPM………IOPS

7,200          80

10,000       130

15,000       180

SSD 2500 – 6000

While not a traditional spinning disk, I have also included Solid State Disks (SSD’s) for reference as SSD’s are starting to see increased market adoption. I have seen a wide range of sizing IOPS for SSD depending on the technology, type (SLC, MLC, etc.) Check out http://en.wikipedia.org/wiki/Solid-state_drive for an introduction, and ask your vendors for more in-depth technical information.

If you are brand-new to this (and you are still reading, congrats!), you can see how many IOPS your Windows computer is asking for by opening Performance Monitor and looking at the ‘Disk Transfers/sec’ counter under Physical Disk. This is a sum of the ‘Disk Reads/sec’ and ‘Disk Writes/sec’ counters as you can see in the screenshot below:

If you are after some stats for your VMware ESX environment, check out esxtop and looking for CMDS/s in the output. I published a couple articles on using esxtop here and here. The numbers from PerfMon and esxtop get you pretty close but can be skewed by a few things we’ll discuss in later posts.

Now that was fun and all, but let’s get real: Single-disk configurations are uncommon in servers. As such, we’ll part ways with our Simple Jack single disk approach to storage and begin to look at more real-world multi-disk enterprise-class storage configurations. A discussion of IOPS in a multi-disk array is a great way to start. From a very elementary perspective, you can combine multiple hard drives together to aggregate their performance capabilities. For example, two 15k RPM disks working together to server a workload could provide a theoretical 360 IOPS (180 + 180). This also scales out so ten 15k RPM disks could provide 1800 IOPS, and 100 15k RPM disks could provide 18,000 IOPS.

Designing your environment so that your storage can deliver sufficient IOPS to the requesting workload is of utmost importance. If you are working on a storage design, arm yourself with data from perfmon, top, iostat, esxtop, and vscsiStats. I typically gather at least 24 hours of performance data from systems under normal conditions (a few days to a week may be good if you have varying business cycles) and take the 95th percentile as a starting point. So from a very simple approach, if your data and calculations show a 1800 IOPS demand at the 95th percentile, you ought to have at least ten 15k RPM disks (or twenty-three 7.2k RPM SATA disks) to achieve performance goals. It’s amazing how some simple data and a pretty little Excel spreadsheet can help you understand and justify the right hardware for the job.

Now before you go and start filling out that PO form for a nice new storage system based on these numbers there are a few more things we ought to discuss. RAID, cache, and advanced storage technologies will skew these numbers and need to be understood. Stay tuned to future articles in this series for more on those topics and more.

Finally, there has been a bunch of activity in the VMware ecosystem of vendors, bloggers, and twittering-type-folks around storage performance. As this here post sat in my drafts folder, Duncan Epping posted this gem of an article that pretty much included all of the content of this article, as well as future ones in my series: http://www.yellow-bricks.com/2009/12/23/iops/. Do yourself a favor and read his post and the comments from his readers – both are filled with a ton of great information, including some vendor-specific implementations.
I was led to Duncan’s article by a post by Chad Sakac on his blog: http://virtualgeek.typepad.com/virtual_geek/2009/12/whats-what-in-vmware-view-and-vdi-land.html. This is also a great read that covers some of the same information with a focus on VMware View/VDI and is also worth a few minutes of your time. Also check out http://vpivot.com/2009/09/18/storage-is-the-problem/ for a rubber-meets-the-road post from Scott Drummonds on the importance of storage performance vis-a-vis IOPS in a VMware-virtualized SQL environment.

Storage Basics – Part VI: Storage Workload Characterization Most of what I covered in Storage Basics Parts 1...
Storage Basics – Part I: An Introduction I am increasingly finding that both my SMB and Enterprise...
Storage Basics – Part V: Controllers, Cache and Coalescing My Storage Basics series has been neglected for some time...

Storage Basics – Part I: An Introduction

Joshua Townsend — Tue, 29 Dec 2009 01:55:15 +0000

I am increasingly finding that both my SMB and Enterprise customers are uneducated on the fundamentals of storage sizing and performance. As a result, storage is often overlooked as a performance bottleneck despite it being a vital component to consider in a virtualization implementation. Storage will only increase in importance as hosts are getting bigger, data volumes increase, and more workloads are virtualized. For some reason, most people can grasp the importance of CPU and memory performance constraints but storage performance is often overlooked and can be hard to explain to business users or executives.

Case in point – I have recently been called into some environments that were not performing well – these environments happened to be running Microsoft SQL, but could just have well been running any application or collection of virtual machines. Fingers were being pointed in all directions: at applications, at the virtualization layer, at a lack of memory, and DBA’s were insisting that there were too few CPU’s. The situation was getting political and emotional when I walked into it. A few minutes with Windows Perfmon was all I needed to identify storage performance as the root cause of the firestorm that had been ignited. Using a bit of data, I was able to turn the discussion from an emotional fight to a simple problem of physics and mathematics (and a bit of simple math could have avoided the problem in the first place).

I have seen this play out a few too many times and so decided to write-up this multi-part series on the basics of storage with a focus on storage performance. That said, a little math and physics is where we will start as we look at the basic building block of a storage environment: a hard disk drive. Wikipedia defines a hard disk drive as “a non-volatile storage device that stores digitally encoded data on rapidly rotating platters with magnetic surfaces.” Your computer, server, or VMware cluster uses hard drives to read and write data. Wikipedia also covers the history and atomic structure of a hard drive pretty well. For our purposes, the take away is that hard drives are physical objects, and as such, follow the laws of physics (duh) in the following measurable ways:

1.) Capacity, which is measured in bits or bytes and exponents there of (MB, GB, TB, PB). This is how much data will fit on your disk, from simple text files to virtual disks, and everything in between. For example, if you have a 500GB SQL database, you darn well better have a hard drive that has a capacity of at least 500GB. This is a pretty simple concept, so I’ll leave it there for now.

2.) Performance, which is measured in a couple ways:

- at the disk itself in Input-Output Per Second (IOPS) – a measure of how many read and write commands a disk can complete in a second

- interface throughput, measured in MBps or Gbps – a measure of the peak rate that a volume of data can be read from or written to disk

- latency – the amount of time between when you ask a disk (or storage system if you want to read ahead) to do something and when it can actually do it, very closely related to IOPS as you’ll read in a forthcoming article in this series.

Each disk, array, and storage system has its own fixed set of measurements given a specific configuration. Knowing the physical capabilities of your storage system as measured in the above ways, and your systems storage requirements will go a long way towards a successful design and implementation of your storage environment. The remaining parts of this series will take a look at these performance characteristics a bit more in-depth and explain what happens as you introduce factors like RAID, cache, data reduction techniques such as snapshots and deduplication, and varying workloads.

Please keep in mind that while I have designed and implemented a variety of DAS, NAS, and SAN technologies from a host of vendors including Dell, EMC, IBM, and NetApp, I am by no means a storage expert. The information I will provide is generalized, over-simplified, and does not consider varying approaches from different storage vendors. Nonetheless, I hope you find this useful information if you are designing a solution, troubleshooting a performance issue or preparing to make a storage purchase.

Keep Reading:

Storage Basics – Part II: IOPS

Storage Basics – Part II: IOPS In Part I of this series, I discussed the important...
Storage Basics – Part V: Controllers, Cache and Coalescing My Storage Basics series has been neglected for some time...
Storage Basics – Part VI: Storage Workload Characterization Most of what I covered in Storage Basics Parts 1...