VMtoday

VMware News, Views, and How-To's from vExpert Josh Townsend

You are here: Home / Archives for SAN

SAN

Configuring VMware VASA for EMC VNX

by 6 Comments

vSphere Storage APIs for Storage Awareness (VASA) are one of several VMware vSphere Storage APIs.  VASA, new in vSphere 5.0, provides vCenter with a way of interrogating storage array LUNs and associated datastores to gain visibility into the underlying hardware and configuration of the storage layer.  Storage capabilities, such as RAID level, thin or thick LUN provisioning, replication state, caching mechanisms, and auto-tiering are presented through VASA to vCenter (a unidirectional read operation by vCenter against the array).  With VASA, vCenter can identify which datastores possess certain capabilities.  By associating a VM – or specific virtual disks within a VM – to storage profiles, we can begin to take advantage of VMware’s Profile Driven Storage capabilities.  With VASA helping to guide VM placement, IT can deliver a higher quality of service to match SLA’s.VMware vSphere Storage API's

A few examples of how using VASA can help IT guarantee SLAs are:

  • A user-defined storage profile defined for ‘High Speed Sequential Write’ could be associated with a VMDK used for database logging.  This same profile would be assigned to VMFS datastores based on RAID10, with ample write cache.
  • VM’s running critical applications could be associated with a storage profile for ‘Synchronous Replication’.  Datastores protected by a SAN-based replication package (such as EMC SRDF or EMC RecoverPoint) would be assigned this profile to guarantee replication of VM’s on the datastore.  VMware SRM would then be used to guarantee crash and application consistency, and automated failover/back capabilities.
  • Test/Dev VM’s could be associated with a storage profile for lower tiered disk without a flash based caching mechanism (i.e. EMC FAST Cache) to keep low priority machines from consuming expensive disk and cache.
  • A cloud provider configures multiple tiers of storage in a gold/silver/bronze fashion and assigns appropriate storage profiles to the datastores.  Customers choose which tier they want (based on cost vs. performance) and have VM’s automatically provisioned on the correct storage tier.  This can be done in vCenter or in vCloud Director!

vSphere Storage Profiles Gold Silver Bronze

VASA-enabled profile driven storage can be combined with vSphere Storage DRS for automated capacity and performance (IOPS) load balancing of like-datastores.  Greater degrees of automation decrease risk while improving SLA’s.  Taken one step further, VMware’s forthcoming vVols technology will basically create a bidirectional VASA capability, where a VM can tell the underlying storage what performance, features, and capabilities it requires and the storage array will automatically create a VMDK on itself to match the demands from the VM.

EMC VNX fully supports the current version of VASA in vSphere 5.1.  To give you an idea of what data can be seen through VASA, here are the storage capabilities exposed [Read more…] about Configuring VMware VASA for EMC VNX

Storage Basics – Part IX: Alternate IOPS Formula

by 12 Comments

StorageBasics9It’s been a long time since I published an article in my Storage Basics series – the series has been some of the top content on my site to date.  I sat for a beta version of the VMware Certified Advanced Professional – Desktop Design (VCAP-DTD) test today and was reminded by the test of a post I had in waiting about an alternate way to calculate IOPS (hint hint – memorize these formulas).  I have several more articles sitting in draft form, but a new job, crazy kids, home improvement projects, and a wife with chronic cerebral spinal fluid (CSF) leaks (it sounds bad – and it is – but look on the bright side: not many guys can get away with saying that their wife’s brain leaks!) all take time.  But enough excuses and back to storage….

I wrote in Storage Basics – Part II: IOPS on how to calculate theoretical IOPS for a given disk type and speed, and then followed it up in Storage Basics – Part III: RAID with some information on  how different RAID configurations impact IOPS.  I.E. More disks equals more IOPS, but with a write penalty assigned for different RAID types.  I then moved into a discussion of interconnects in Storage Basics – Part IV: Interface, but didn’t do a good job there of tying the interconnect speed to performance and IOPS.  Finally, in Storage Basics – Part VI: Storage Workload Characterization, I hinted briefly on how I/O request size can impact storage performance.  Let’s dig into this topic a little deeper!

The Original IOPS Forumula

Calculating IOPS at the disk/RAID level is fun and all (and I suggest you brush up on it if you are sitting for the VCAP exams), but does not paint a full picture of the storage subsystem.  The basic formula to get a single disk IOPS (I covered this in Part II) looks like this:

\frac{\text{Seek Latency} + \text{Rotational Latency}}{1000} = IOPS

(The LaTeX formulas keep getting messed up – sorry if they don’t look right. Let me know and I’ll fix them, again).
For simplicity sake, let’s just assume that this formula yields 180 IOPS for a standard 15k RPM SAS disk (the industry standard number of IOPS for this speed and type of disk)

The formula for RAID IOS is in Part III – here’s a sample for RAID5, where:

p = IOPS required, f = write IO penalty factor, r = % Read, w = % Write

{(p \times r)} + {((p \times w) \times f)} = \text{Back-end Spindle IOPS}

Let’s put it to use: let’s say I have a workload that I’ve measured as needing 4000 IOPS from the OS/application perspective, of which 70% are write (not unusual in VMware View Linked Clones): here’s how you figure out how many disks you need from an IOPS perspective for those linked clones:

{(4000 \times 30\%)} + {((4000 \times 70\%) \times 4)} = 12,400\ast

*12,400 Back-end IOPS needed on my spindles in a RAID5 array

Then, convert the number of back-end IOPS needed to a number of disks needed given our RAID5 assuming the 15k SAS disk:

\frac{12,400}{180} = 69

That is, I need 69 disks (rounded up) required on the back-end to meet our 4000 front-end IOPS, ignoreing things like best practices when putting a bunch of disks in a RAID group/storage pool (4+1 RAID5 sets) and hotspares (1 in 30 for SAS)

A quick run at the math suggests that you should probably not be using RAID5 for a write-heavy workload (run the math yourself with a write penalty of 2 for RAID0+1 to  see how many fewer disks you would need).  Also, this particular example leaves out any calculation for cache impact, including EMC FAST Cache or the View Storage Accelerator (CBRC).  If you have those technologies in the mix, then a very rough rule of thumb would be to size your back-end IOPS based on your front-end IOPS calculation (4000 in the example above, instead of 12,400) – I say very rough because this really depends on your cache size and cache hit ratio, and the rule pertains a bit more to read-heavy workloads than write heavy.  If you have CBRC in the mix, you *might* be able to greatly reduce and maybe eliminate the Read IOPS from the equation.  But before you go taking my word as gospel, test, test, and test some more using your particular workloads.

The Alternative IOPS Formula

But this is all back-end spindle stuff and me having fun with \text{LaTeX} (LaTeX is the language I used to make the cool math equations – if the forumlas don’t look right drop a comment so I can fix them. WordPress keeps eating the latex syntax…).  Hopefully your SAN admin already knows this, so go ahead and give her your front-end IOPS count and read:write ratio and you’ll get some LUN’s presented.  But you, the VMware admin, might be responsible for server hardware, including buying the interconnect card (HBA, iSCSI NIC).  How do IOPS inform your selection?  This is where the alternate formula for calculating IOPS comes in.  Here’s the forumla:

\frac{\text{Throughput (MB per second)} \times 1024 \text{(convert KB to MB)}}{\text{Block Size (KB/IO)}} = IOPS

[Read more…] about Storage Basics – Part IX: Alternate IOPS Formula

Storage Basics – Part VII: Storage Alignment

by 19 Comments

StorageBasics7At the risk of beating a dead horse, it’s time to resurrect my Storage Basics series.  I’ve recently had some great feedback on the series and figured I should round out a few of the concepts before I wrap it up.  I want to cover a topic often discussed amongst virtualization professionals, but one I often find general practitioners and server admins not understanding: storage alignment.  Storage alignment, or the lack of alignment, is not a new issue and is not unique to VMware or virtualization in general.  However, the effects of misaligned storage can be more greatly felt in terms of reduced performance and strain on a storage system in shared, oversubscribed or high I/O environments.  Many others in the virtualization and storage communities have already covered partition alignment (see Duncan Epping, Vaughn Stewart, and most recently Chad Sakac), but I feel it is an important enough topic for me to re-hash as part of this series.

What is Storage Alignment?

Let’s start with a quick overview of what storage alignment means.  Quite simply, storage alignment refers to the positioning (starting offset) of the various pieces of a systems storage components – the physical disk sectors or array’s chunks, the VMware File System (VMFS) in a VMware environment, and the guest file system’s clusters within a partition – in relation to the layer directly under the element in question.  A quick graphic often makes quick work of explaining this (I often whiteboard this concept for colleagues and clients):

partition alignment conceptual diagram

As you can see, the starting offset of the VMFS partition does not correspond to the physical segmentation of the underlying disks (in this case, the chunks on a SAN – but could be conceptually replaced with the sectors of a single disk).  Furthermore, the clusters (or blocks) of the guest VM are not aligned to the VMFS partition nor to the underlying storage.  For traditional (physical) systems or VMware RDM’s, the VMFS layer could be abstracted, but the result would be the same – the clusters of a partition would be misaligned to the underlying disk.

What Does it Mean?

Quite simply, misaligned storage (both VMFS partitions and Guest File Systems) can lead to poor performance under certain conditions.  How badly performance is impacted depends on the degree of I/O strain your server and storage are under, the caching mechanisms in your environment, and the architecture of your SAN.  Again, a visual can help explain how misaligned storage can hurt you. For simplicity let’s leave out the VMFS layer as we consider the following diagram (pardon my hasty Visio visualization):

What we see is that the target data in a tiny 16kb read request spans two 64kb chunks on our storage array.  Any reads of that piece of data will result in twice the amount of data as would be minimally necessary being transferred to the host’s storage stack.  The net effect is an increase in the work the storage array must do – gobbling up IOPS that would otherwise be available for the real work of reading data, reducing throughput on the interface, and messing with cache algorithms and dedupe mechanisms on some arrays.  In short, misaligned storage is an efficiency killer.  Now add in the VMFS layer back in and you’ll see how things get complicated.

If (and we’re talking a big IF here) every bit of data you wanted to read spanned a chunk or sector boundary, you could experience half the expected performance due to misalignment.  In reality, depending on your workload and storage technology your performance increase from properly aligning your storage will probably be somewhere between 10-30%.

Want to dig deeper?

There have been some great resources published on this issue over the past few years on storage alignment.  Major vendors have all begun pushing information on the problem – here are some of the best that I have found:

Microsoft has a Knowledge Base article (https://support.microsoft.com/kb/929491) that describes the problem and symptoms of misaligned partitions, how to determine if your partition is aligned, and the use of diskpart to create aligned partitions.

Microsoft also has an in-depth article on MSDN, including some performance numbers at https://msdn.microsoft.com/en-us/library/dd758814.aspx.  Also check out Jimmy May’s series Partition (Sector) alignment for SQL Server here: https://blogs.msdn.com/b/jimmymay/archive/2008/10/14/disk-partition-alignment-for-sql-server-slide-deck.aspx.  One of the best descriptions of the complexities of the problem can be found in Jimmy’s blog series.  Microsoft has a KB article that describes performance problems when disk partitions are misaligned here: https://support.microsoft.com/kb/929491, including the formulas to calculate if your existing partitions are correctly aligned.

VMware has an article here: https://www.vmware.com/pdf/esx3_partition_align.pdf.  Be aware that this article is for Virtual Infrastructure 3, not vSphere 4.0.  Some of the information is now a bit dated.

Netapp has a few documents to check out: https://media.netapp.com/documents/tr-3428.pdf (VI3), and https://media.netapp.com/documents/tr-3749.pdf (vSphere)

EMC covers alignment in their TechBooks for Clariion, Celerra, and Symmetrix.

Tools to Align Partitions:

Ok – so you’ve bought into this whole partition alignment thing as being a real issue.  How to you fix it?  Here are some tools:

  1. MSInfo32.exe, wmic, and dmdiag will show you misaligned partitions on Windows machines (check the Microsoft links above for usage info).
  2. Diskpart.exe (or diskpar.exe on versions of Windows previous to 2003) creates aligned partitions on Windows systems.  Diskpart cannot be used to realign a previously created partition, only to create new correctly aligned partitions.
  3. MBRScan/MBRAlign from NetApp can report on and realign existing virtual disks on a VMware ESX server.  Also a nifty PowerShell script from NetApp to find if your partitions are aligned: https://communities.netapp.com/docs/DOC-6175
  4. vOptimizer from Vizioncore can report on and realign existing virtual disks.
  5. GParted can be used to create aligned partitions on both Windows and Linux machines, and to realign some existing partitions.
  6. VMware vCenter – VMFS datastores created using vCenter are aligned automatically. Note – Guest VMDK’s are not aligned automatically by vCenter – you must manually create aligned partitions on your VMDK’s or use a Guest OS that creates properly aligned partitions (Windows 2008 and later).

Best Practices:

Before I wrap this installment up, here are some best practices for storage alignment in your environment:

  • Create aligned partitions in your VMware templates. Do it once, do it right – every machine you deploy from the template will be aligned.
  • Use caution with tools like Symantec Ghost.  Ghost can take images of aligned partitions and misalign them when laying down on a new system.
  • Use caution when performing P2V’s using VMware vCenter Converter – it does not align guest disks on import.  You might consider using Converter to perform a P2V of the system disk only, then create new VMDK’s on the converted guest.  Use Diskpart, gparted, or another tool to create aligned partitions on the new VMDK’s and finally copy the data over to the newly virtualized server using a tool like Robocopy, RichCopy, or rsync.
  • SSD’s are particularly sensitive to misalignment, leading to poor performance and excessive wear.
  • Local VMFS volumes created by the ESX installer are not aligned.  If you are using an installer-created local VMFS for anything where performance matters, you might consider re-creating it through vCenter.
  • Watch out when attaching a data disk from an older VM to a new VM.  For example, you are upgrading your SQL servers to Windows 2008 R2 from 2003.  You decide to do a side-by-side upgrade, using the detach/attach method.  You install (or better yet, deploy from template) a new Windows 2008 R2 VM, detach your databases from the old server, move your SQL data and log virtual disks from your 2003 VM to the new VM and attach the SQL DB’s on the new server.  Those old VMDK’s may be misaligned!  Consider using Robocopy, RichCopy or rsync to ensure an aligned disk.
  • Check your storage vendors best practices for your particular environment (OS, workload, SAN, etc.).
  • There is some debate on whether or not it is advised to align your OS partitions.  There is no clear-cut answer on this as it depends so much on your environment and particular needs.  For help in deciding if you should align your Guest OS drives, see the comments in the blogs by Duncan Epping, Vaughn Stewart, and Chad Sakac.
  • While working the VMware User Group booth at the Washington, DC Virtualization Forum 2010 I had a user ask me if rules and procedures for alignment on 4k sector disks are different. I forgot to research it until just now, so I honestly don’t know (please comment if you do know!).  Check with your storage vendor if this is an issue for you.
  • Finally, you can’t realign partitions using tools like mbralign or vOptimizer in ESXi -Aaaron Delp explains the problem here: https://blog.aarondelp.com/2010/06/my-1-issue-with-vmware-esxi-today.html.

I hope this is helpful for you in understanding the problem of storage alignment and how it can impact your environment.  Comments or questions are welcomed!

Keep Reading:

Free SAN Monitor for DS3300, MD3000i and others

by 4 Comments

One of my most popular posts to date had been IBM DS3300 Write Performance Problem Solved.  I am pleased to have upgrade my internal environment to an EMC Clariion CX4 array, but still have customers using the DS3300 and the MD3000i from Dell.  For that reason, I keep an eye out for news on these arrays.  You’ll recall that these arrays are based on the same LSI kit.  Truth be told, properly designed and configured, these arrays are a technically proficient and cost efficient solution for SMB’s looking to get their VMware environments going.

WhileSolarWinds Free San Monitor reviewing my email today I found an email from SolarWinds about their new Free SAN Monitor.  I have used several products from SolarWinds in the past, and so thought I would give this a quick try.  This tool can monitor the LSI-based arrays from Dell (including the MD3000 series), IBM (DS3000, DS4000, and DS5000), and SunStorageTek 2000 and 60000.  Installation and configuration were both fast and easy.  A wizard asked me for the array’s IP address, monitoring interval, and some thresholds (which I left blank).  Once I clicked Finish the app launched on my desktop and immediately began to display data.

This screen-shot is of the app running on my Windows 7 x64 workstation.  You’ll notice that the array is reporting performance, in terms of IOPS, and response time for individual LUNS.  The health of my array is also displayed.  You’ll see that my ‘Controller 2’ shows a failure.  I do not have a dual-controller configuration (hence my wildly popular post on performance being in the tank due to a disabled write cache).  The SAN Monitor incorrectly reported that I had two controllers, and that the 2nd is failed.

This is not bad for a free tool, in my opinion.  I would love to see SolarWinds add some info on write cache status for the array (cache hit %, cache enabled per LUN, etc.) as well as LUN utilization in terms of free space remaining.  For a free desktop GUI, it’s worth the 5 minutes to download and setup.

Reporting on I/O Response time (in milliseconds) is helpful in determining if your array is stressed.  If response times are consistently above 15ms, you ought to look at adding some spindles.  If you are over 25ms on a regluar basis, you’re likely feeling some pain on the servers generating the workload against your array.

If you are into the whole free tool thing, SolarWinds also has a free VM Monitor that is capable of monitoring a single ESX server via SNMP and it’s associated VM’s.  It too runs as a desktop application, requiring minimal setup and configuration.  While it won’t monitoring your whole VMware Essentials (3 ESX servers + vCenter) environment, it’s a start.

You might also check out Nick Weaver’s vSphere Mini Monitor for basic real-time alerts and monitors, as well as NetWrix’s Change Reporter for VMware.  A final free (and nothing short of awesome) tool is Alan Renouf’s vCheck Daily Report PowerShell Script.

Do you have free tools, scripts or utilities for your VMware environment?  I’d love to hear more about them in the comments section!

Storage Basics – Part VI: Storage Workload Characterization

by 21 Comments

StorageBasics6Most of what I covered in Storage Basics Parts 1 through 5 was at a very elementary level.  The math I used to do IOPS calculations, for example, is only true under very certain conditions.  RAID controllers implement caching and other techniques that skew the simple math that I provided.  I mentioned that the type of interface that you ought to use on your storage array should not be randomly chosen.  In fact, choosing the right array with the appropriate components and characteristics can only be done when you enlighten your decision with a characterization of workloads it will be running.

The character of your storage workload can be broken down into several traits – random vs. sequential I/O, large vs. small I/O request size, read vs. write ratio, and degree of parallelism.  The traits of your particular workload dictate how it interacts with the components of your storage system and ultimately determine the performance of your environment under a given configuration.  There is an excellent whitepaper available from VMware entitled “Easy and Efficient Disk I/O Workload Characterization inVMware ESX Server” that is authoritative on this subject.  If you want to get down and dirty with the topic, it’s a good read.  I’m aiming for something a bit less academic.  With that said, let’s break down workload characterization a bit so as to better understand how it will impact your real-world systems.

Random vs. Sequential Access

In Part II of this series we looked at the formula for calculating IOPS capabilities for a single disk.  That formula goes something like this:

IOPS = 1000/(Seek Latency + Rotational Latency)

You’ll recall that we divide into 1000 to remove milliseconds from the equation, leaving (Seek Latency + Rotational Latency) as the important part of the equation.  Rotational latency is based on the spindle speed of the disk – 7.2k, 10k, or 15k RPM for standard server or SAN disks.  If we consider the same Seagate Cheetah 15k drive from Part II, we see that rotational latency is 2.0ms.  The only way to change rotational latency is to buy faster (or slower) disks.  This essentially leaves seek latency as the only variable that we can “adjust”.  You’ll also recall that seek latency was the larger of the latencies (3.4ms for read seeks, and 3.9ms for write seeks) and counts more against IOPS capability than does rotational latency.  Seeking is the most expensive operation in terms of performance.

It is next to impossible to adjust seek latency on a disk because it is determined by the speed of the servos that move the heads across the platter.  We can, however, send workloads with different degrees of randomness to the platter.  The more sequential a workload is, the less time that will be spent in seek operations.  A high degree of sequentiality ultimately leads to faster disk response and higher throughput rates.  Sequential workloads may be candidates for slower disks or RAID levels.  Conversely, workloads that are highly randomized ought to be placed on fast spindles in fast RAID configurations.

You’ll notice that I said it was next to impossible to adjust seek latency on a disk.  While not common, some storage administrators employ a method know as ‘short stroking’ when configuring storage.  Short stroking uses less than the full capacity of the disk by placing data at the beginning of the disk where access is faster, and not placing data at the end of the disk where seeks times are greater.  This results in a smaller area on the disk platter for heads to travel over, effectively reducing seek time at the expense of capacity.

While not applicable to all workloads, storage arrays, or file systems, fragmentation can cause higher degrees of randomness leading to degraded performance.  This is the prime reason some vendors recommend that you regularly defragment your file system.  It should be noted that a VMware VMFS file system is resilient against the forces of fragmentation.  Whereas a Windows NTFS parition may hold hundreds, thousands or tens of thousands of files of different sizes, accessed randomly throughout the system’s cycle of operations, a VMFS datastore  typically holds no more than a couple hundred files.  Additionally, most of the files on a VMFS datastore are created contiguously if you are using thick-provisioned virtual disks (VMDK).  Thin-provisioned VMDK’s are slightly more susceptible to fragmentation, but do not typically suffer a high enough degree of fragmentation to register a performance impact.  See this VMware whitepaper for more on VMFS fragmentation: Performance Study of VMware vStorage Thin Provisioning.

Examples of sequential workloads include backup-to-disk operations and the writing of SQL transaction log files.  Random workloads may include collective reads from Exchange Information Stores or OLTP database access.  Workloads are often a mix of random and sequential access, as is the case with most VMware vSphere implmentations.  The degree to which they are random or sequential dictates the type of tuning you should perform to obtain the best possible performance for your environment.

I/O Request Size

I/O request size is another important factor in workload characterization.  Generally speaking, larger reads/writes are more efficient than smaller I/O to a certain point.  The use of larger I/O requests (64KB instead of 2KB, for example) can result in faster throughput and reduced processor time.  Most workloads do not allow you to adjust your I/O request size.  However, knowing your I/O request size can help with appropriate configuration of certain parameters such as array stripe size and file system cluster size.  Check with your storage vendor for more information as it pertains to your specific configuration.

If you are in a Windows shop, you can use perfmon counters such as Avg. Disk Bytes/Read to determine average I/O size.  If you are running a VMware-virtualized workload, you can take advantage of a great tool – vscsiStats – to identify your I/O request size.  More on vscsiStats later in this article.

Read vs. Write

Every workload will display a differing amount of read and write activity.  Sometimes a specific workload, say Microsoft Exchange, can be broken down into sub-workloads for logging (write-heavy) and reading the database (read-heavy).  Understanding the read-to-write ratio may help with designing the underlying storage system.  For example, a write-heavy workload may perform better on a RAID10 LUN than a RAID5 array due to the write penalty associated with RAID5.  The ratio of read:write may also dictate caching strategies.  The read:write ratio, when combined with a degree of randomness measure, can be quite useful in architecting your storage strategy for a given application or workload.

Parallelism/Outstanding I/O’s

Some workloads are capable of performing multi-threaded I/O.  These types of workloads can place a higher amount of stress on the storage system and should be understood when designing storage, both in terms of IOPS and throughput.  Multipathing may help with multi-threaded I/O workloads.  A typical VMware vSphere environment is a good example of a workload capable of queuing up outstanding I/O.

Measuring the Characteristics of Your Workload

So how do we actually characterize storage workloads?  Start with the application vendor – many have published studies that can shed light on specific storage workloads in a standard implementation.  If you are interested in measuring your own for planning/architecture reasons, or performance troubleshooting reasons, read on….  There are several tools to measure storage characteristics, depending on your operating system and storage environment.  Standard OS performance counters, such as Windows Performance Monitor (perfmon) can reveal some of the characteristics.  Array based tools such as NaviAnalyzer on EMC gear can also reveal statistics on the storage end of the equation.

One of the most exciting tools for storage workload characterization comes from VMware in the form of vscsiStats.  vscsiStats is a tool that has been included in VMware ESX server since version 3.5.  Because all I/O commands pass through the Virtual Machine Monitor (VMM), the hypervisor can inspect and report on the I/O characteristics of a particular workload, down to a unique VM running on an ESX host.  There is a ton of great information on using vscsiStats, so I won’t re-hash it all here.  I recommend starting with Using vscsiStats for Storage Performance Analysis as it contains an overview and usage instructions.  If you want to dig a bit deeper into vscsiStats, read both Storage Workload Characterization and Consolidation in Virtualized Environments and vscsiStats: Fast and Easy Disk Workload Characterization on VMware ESX Server.

vscsiStats can generate an enormous amount of data which is best viewed as a histogram.  If you’re a glutton for punishment, the data can be reviewed manually on the COS.  To extract vscsiStat output data, use the -c option to export to a .csv file.  From there you can analyze the data and create histograms using Excel.  Paul Dunn has a nifty Excel macro for analyzing and reporting on vscsiStats output here.  Gabrie van Zanten more detailed instructions for using Paul’s macro here.  Here are a couple histogram examples that I just generated from a test VM.

IO Lengths HistogramIO Distance Between Commands

vscsiStats is only included with ESX, not ESXi.  However, Scott Drummond was kind enough to post a download of vscsiStats for ESXi on his Virtual Pivot blog: https://vpivot.com/2009/10/21/vscsistats-for-esxi/.  Using vscsiStats on ESXi requires dropping into Tech Support Mode (unsupported) and enabling ESXi for scp to transfer the binary to the ESXi server.

VMware esxtop can display some information but is limited in scope and does not currently support NFS.  A community-supported python script called nfstop can parse vscsiStats data and display esxtop-like data per VM on screen.

Experiment

If you are interested in generating workloads with various characteristics, check out Iometer and Bonnie++.  These tools will allow you to generate I/O that you can monitor with the tools I covered in this article.

Put it to Use

If you are provisioning a new workload or expanding an existing, invest some time in understanding your storage workload characteristics and convey those characteristics to your storage team.  A request for storage that includes the workload characteristics I discussed here, as well as expected IOPS requirements, will go much further in ensuring performance for your applications – physical or virtual – than simply asking for a certain capacity of disk.

Keep Reading: