VMtoday

Archive for the ‘General IT’ Category

My Storage Basics series has been neglected for some time (sick kids, snow storms, VMware Upgrades, SAN implementations and some Cisco switch upgrades took all my free time), so let’s jump right in to Part V – Cache, Controllers, and Coalescing.   Between the alliteration and fancy words, it might seem like I am about to tell a tale of international espionage.  Unfortunately, my introductory treatment of these aspects of a storage system will probably not keep you on the edge of your seat – but I’ll try to keep it interesting.

Throughout this series, we’ve been working our way from the basic building block of any storage system – the disks – outwards towards the brains of the operation – the controller.  You’ll recall that in Part II I introduced IOPS and the math that goes into calculating the IOPS capacity of a disk array.  In Part III we considered a RAID implementation’s impact on performance and availability.  And most recently in Part IV we looked at the common interface types when dealing with storage arrays.  If we put the previous parts together we still don’t have a functional storage system.  The missing piece is the controller.  Simply put, the storage controller is the hardware adapter between the disks and the servers that connect to the storage.  The controller has a specific ‘interface‘ type, is responsible for RAID operations, and handles advanced storage functionality.  A controller can be as simple as the Dell PERC or HP Smart Array add-in card on your server, or as complex as the Storage Processor in an enterprise class Storage Area Network (SAN) such as an EMC CLARiiON or NetApp FAS.

Controllers

As we look at controllers and the advanced features they provide we’ll see that some of the earlier performance equations start to break down.  The simplest controllers take disk read/write commands from the operating system and send commands down to the disk(s) attached to be read or written. This gets data onto the disk, but often does not do so in an efficient or reliable manner.  RAID-capable controllers take on the added responsibility of configuring disks in the desired RAID level, calculating & writing parity data, and writing the data in disk-spanning stripes or mirrors depending on the RAID level.

Cache

To increase performance and improve reliability, storage vendors implement a caching system on their controllers.  Cache is memory that acts as a buffer for disk I/O, and is usually battery-backed to prevent data loss in the event of a power failure.  Because of the exponentially greater speed of RAM over spinning magnetic disks, cache can improve performance by orders of magnitude. Cache can operate on both reads and writes to disk.

When dealing with writes, the controller cache is typically used in one of two ways: write-through or write-back.  In write-through mode, data is written to volatile cache and then to disk, and only acknowledged as written once the data resides on the non-volatile disk.  Write-back mode allows the controller to acknowledge the data as having been written as soon as it is held in cache.  This allows the cache to buffer writes quickly and then write them to the slower disk when the disk has cycles to accept I/O.  The greater your cache size, the more data that can be buffered, ultimately resulting in better performance as measured in both IOPS and throughput.  This graph from my article on troubleshooting write performance on an IBM DS3300 iSCSI array shows how throughput increased and latency decreased when enabling write cache.  The extent to which cache increases performance is highly dependent on the workload characteristics (I/O size, randomness, and ratio of reads:writes).

Read-cache acts as a buffer for reads in a couple ways.  First, some controllers attempt to ‘read-ahead’, anticipating future read requests from the operating system and buffering what it expects to be the next blocks of desired data.  Some entry-level controllers simply buffer the next physical chunk of data and fill cache memory with it, while more advanced controllers may attempt to predict the right block of data based on previous requests (you just asked for 3 blocks in a row, I’m guessing you’ll come asking for the 4th next so I’ll just buffer it in fast cache for you now).  Secondly, read cache holds data that has been previously read, regardless of any pre-fetching the controller may have done.  This allows for much faster subsequent access of the same data because it is held in the faster cache, eliminating the need for the controller to go to disk for the data again.  Just like with write cache, the extent to which cache increases performance is highly dependent on the workload characteristics.

A given storage array controller only has so much cache to work with.  A Dell PERC5/E, for example, has 256MB of cache that can be used for both read and write.  While this may be enough for a direct-attached storage array, SAN’s serving multiple systems demand more cache.  In contrast, an EMC CLARiiON CX4-960 has 32GB.  Some storage vendors, such as NetApp, are getting creative with cache.  NetApp’s Performance Acceleration Module (PAM) is an add-in card that provides up to a whopping 512GB of Layer 2 cache to the storage system.

Caching mechanisms can dramatically influence performance under the right conditions.  With healthy cache in place, IOPS calculations become skewed.  However, cache can be exhausted or may not hold the data you are interested in.  If cache is insufficient to satisfy read requests, or has reached its high-water mark for writes, performance can drop off.  When cache is exhausted, the backing disk must be able to satisfy the I/O workload or performance will be unacceptable.  This is where the IOPS calculations kick in, and where having the right disk type and configuration really matters.

Queuing & Coalescing

Advanced storage systems introduce additional features to reduce I/O contention and improve cache utilization.  I won’t go into all of the features here because they vary by storage vendor.  However, I will point out two common techniques – queuing and coalescing.

Queuing refers to the ability of a storage system to queue storage commands for later processing.  Queuing can take place at various points in your storage environment, from the HBA to the storage processor/controller.  A little queuing may be OK depending on your workload, but too many outstanding I/Os can negatively impact performance (this is measured in latency).  Queue depths can be adjusted on many components in your storage and VMware landscape, but check with your vendor’s support group before you make changes to these settings.

Coalescing is performed by some storage systems to modify the character of the workload.  To better understand coalescing, picture a bunch of random write activity.  Without cache in place, the disk heads will be bouncing all over the platters trying to get the data on to disk.  A little write cache will allow the storage array to acknowledge the write for the OS, but the array still needs to de-stage that data from cache to disk quickly to prevent cache exhaustion.  The back-end disks will still be doing the chicken dance, bouncing around trying to write the random workload….  Now picture an intelligent system that re-orders the random writes that are held in cache and writes them to the disk in nice sequential stripes.  The disk heads will be less prone to jumping around the platter and the behavior will start to look more like a nice waltz than the funky chicken dance.  Coalescing is used for writes, not reads, so not all workloads benefit.

Wrap-up

With this article on Controllers, Cache, and Coalescing we’ll end our look at the basic building blocks of a storage array.  Before we end the Storage Basic series I am planning a few more articles on Storage Workload Characterization (which has been mentioned, but not directly addressed in this and previous articles), Identifying a Stressed Storage System, and Best Practices for Storage Performance in a VMware Environment.

If you are interested in more reading on Controllers, Cache, and Coalescing, I recommend the following:

Additional Reading:

• Impact of cache on the performance of the HP StorageWorks XP12000 Disk Array white paper
• Performance impact of controller cache: SQL Server read only workloads
• IOps? – Dig into the article’s comments for some great dialog between some people who really know their stuff!
• Storage Performance for SQL Server
• Storage Caching 101 – Chuck Hollis (EMC)
• Improving Performance with Interrupt Coalescing for Virtual Machine Disk IO in VMware ESX Server

We all know that virtualization allows us to do more with less.  Fewer servers and space-saving storage (talk about an oxymoron) help us put some green in the datacenter and back in the budget.  But with tight budgets demanding greater efficiency, virtualization pushing per-U-space utilization higher, and increasingly rack-dense equipment, proper planning of your physical plant remains an essential part of IT.  I argue that right-sizing your power, cooling, and floor-space is more critical now than it has ever been, and is a knowing how to do it is a darn good skill for a virtualization engineer to possess.

So along those lines… I was just doing some site-prep work for a new Clariion installation and noticed that the EMC Power Calculator has been updated.  It is now a pretty slick little web app that can be found on the PowerLink site (login required) here: https://powerlink.emc.com/nsepn/webapps/powercalculator/Main.aspx.

While I am at it, here are some links to other power consumption calculators.  Let me know if you have others and I will update this post:

• Dell: http://www.dell.com/calc
• IBM: http://www-03.ibm.com/systems/bladecenter/resources/powerconfig/index.html
• NetApp: Storage Efficiency Calculator here - http://www.secalc.com – it doesn’t calculate your consumption, just what you might save over a competitor’s offering.
• HP: http://h30099.www3.hp.com/configurator/powercalcs.asp
• Sun: http://www.sun.com/solutions/eco_innovation/powercalculators.jsp
• Hitachi/HDS: http://www.byhitachi.com/se/go/weight-and-power-calculator/
• APC: http://www.apc.com/prod_docs/results.cfm?DocType=Trade-Off%20Tool&Query_Type=10 and http://www.apcc.com/products/runtime_for_extendedruntime.cfm?upsfamily=165
• Emerson: Efficiency Calculator: http://www.emerson.com/edc/Calculator/default.aspx
• VMware ROI Calculator: http://vmware.com/go/calculator
• This site has a bunch of links to other calculators and resources: http://thegreenandvirtualdatacenter.com/calculator.html

There’s some fun and timely chatter happening right now on Twitter around power consumption and sizing – join in by following me at http://twitter.com/joshuatownsend/!

In Part I of this series, I discussed the important of storage performance in a virtual environment (really any environment, virtual or not, where you want acceptable performance), and introduced some of the basic measures of a storage environment.  In Part II, we will look more closely at what may be the most important storage design consideration in a VMware server-consolidation enviornments, many SQL environments, and VDI environments to name a few: IOPS.

If we stick with a single-disk-centric approach as we did in Part I, IOPS is quite simply a measure of how many read and write commands a disk can complete in a second.  IOPS is an important measure of performance in a shared storage environment (such as VMware) and in high-transaction-rate workloads like SQL.  Because hard drives are forced to abide by the laws of physics, the IOPS capabilities of a disk are consistent and predictable given a specific configuration.  The formula for calculating IOPS for a given disk is pretty straight forward (please show your work):

IOPS = 1000/(Seek Latency + Rotational Latency)

Exact latencies vary by disk type, quality, number of platters, etc.  You can look up the tech specs for most drives on the market.  As an example, I have randomly chosen the technical specifications of the Seagate Cheatah 15k.7 SAS drive.  This particular drive has the following performance characteristics:

- Average (rotational) latency: 2.0msec

- Average read seek (latency): 3.4msec

- Average write seek (latency): 3.9msec

Using the read latency number, the math works out like this:

1000
———- = 185 maximum read IOPS
2.0+3.4

The maximum write IOPS will be a bit less (~169IOPS) because of the higher write seek latency.  Writing is more ‘expensive’ than reading and therefore slower.

Fortunately, there are some widely accepted ‘working’ numbers, so you do not have to use this formula for each and every disk you might consider using.  Because rotational latency is based on the rotational speed, we can use the published Rotations Per Minute (RPM) rating of the drive to guess-timate the IOPS capabilities.  Typical spindle speeds (measured in RPM) and their equivalent IOPS are in the table below.

RPM………IOPS

7,200          80

10,000       130

15,000       180

SSD           2500 – 6000

While not a traditional spinning disk, I have also included Solid State Disks (SSD’s) for reference as SSD’s are starting to see increased market adoption.  I have seen a wide range of sizing IOPS for SSD depending on the technology, type (SLC, MLC, etc.)  Check out http://en.wikipedia.org/wiki/Solid-state_drive for an introduction, and ask your vendors for more in-depth technical information.

If you are brand-new to this (and you are still reading, congrats!), you can see how many IOPS your Windows computer is asking for by opening Performance Monitor and looking at the ‘Disk Transfers/sec’ counter under Physical Disk.  This is a sum of the ‘Disk Reads/sec’ and ‘Disk Writes/sec’ counters as you can see in the screenshot below:

If you are after some stats for your VMware ESX environment, check out esxtop and looking for CMDS/s in the output.  I published a couple articles on using esxtop here and here.  The numbers from PerfMon and esxtop get you pretty close but can be skewed by a few things we’ll discuss in later posts.

Now that was fun and all, but let’s get real: Single-disk configurations are uncommon in servers.  As such, we’ll part ways with our Simple Jack single disk approach to storage and begin to look at more real-world multi-disk enterprise-class storage configurations.  A discussion of IOPS in a multi-disk array is a great way to start.  From a very elementary perspective, you can combine multiple hard drives together to aggregate their performance capabilities.  For example, two 15k RPM disks working together to server a workload could provide a theoretical 360 IOPS (180 + 180). This also  scales out so ten 15k RPM disks could provide 1800 IOPS, and 100 15k RPM disks could provide 18,000 IOPS.

Designing your environment so that your storage can deliver sufficient IOPS to the requesting workload is of utmost importance.  If you are working on a storage design, arm yourself with data from perfmon, top, iostat, esxtop, and vscsiStats.  I typically gather at least 24 hours of performance data from systems under normal conditions (a few days to a week may be good if you have varying business cycles) and take the 95th percentile as a starting point.  So from a very simple approach, if your data and calculations show a 1800 IOPS demand at the 95th percentile, you ought to have at least ten 15k RPM disks (or twenty-three 7.2k RPM SATA disks) to achieve performance goals.  It’s amazing how some simple data and a pretty little Excel spreadsheet can help you understand and justify the right hardware for the job.

Now before you go and start filling out that PO form for a nice new storage system based on these numbers there are a few more things we ought to discuss.  RAID, cache, and advanced storage technologies will skew these numbers and need to be understood.  Stay tuned to future articles in this series for more on those topics and more.

Finally, there has been a bunch of activity in the VMware ecosystem of vendors, bloggers, and twittering-type-folks around storage performance.  As this here post sat in my drafts folder, Duncan Epping posted this gem of an article that pretty much included all of the content of this article, as well as future ones in my series: http://www.yellow-bricks.com/2009/12/23/iops/.  Do yourself a favor and read his post and the comments from his readers – both are filled with a ton of great information, including some vendor-specific implementations.
I was led to Duncan’s article by a post by Chad Sakac on his blog: http://virtualgeek.typepad.com/virtual_geek/2009/12/whats-what-in-vmware-view-and-vdi-land.html.  This is also a great read that covers some of the same information with a focus on VMware View/VDI and is also worth a few minutes of your time.  Also check out http://vpivot.com/2009/09/18/storage-is-the-problem/ for a rubber-meets-the-road post from Scott Drummonds on the importance of storage performance vis-a-vis IOPS in a VMware-virtualized SQL environment.

I am increasingly finding that both my SMB and Enterprise customers are uneducated on the fundamentals of storage sizing and performance.  As a result, storage is often overlooked as a performance bottleneck despite it being a vital component to consider in a virtualization implementation.  Storage will only increase in importance as hosts are getting bigger, data volumes increase, and more workloads are virtualized.  For some reason, most people can grasp the importance of CPU and memory performance constraints but storage performance is often overlooked and can be hard to explain to business users or executives.

Case in point – I have recently been called into some environments that were not performing well – these environments happened to be running Microsoft SQL, but could just have well been running any application or collection of virtual machines.  Fingers were being pointed in all directions: at applications, at the virtualization layer, at a lack of memory, and DBA’s were insisting that there were too few CPU’s.  The situation was getting political and emotional when I walked into it.  A few minutes with Windows Perfmon was all I needed to identify storage performance as the root cause of the firestorm that had been ignited.  Using a bit of data, I was able to turn the discussion from an emotional fight to a simple problem of physics and mathematics (and a bit of simple math could have avoided the problem in the first place).

I have seen this play out a few too many times and so decided to write-up this multi-part series on the basics of storage with a focus on storage performance.  That said, a little math and physics is where we will start as we look at the basic building block of a storage environment: a hard disk drive.  Wikipedia defines a hard disk drive as “a non-volatile storage device that stores digitally encoded data on rapidly rotating platters with magnetic surfaces.” Your computer, server, or VMware cluster uses hard drives to read and write data.  Wikipedia also covers the history and atomic structure of a hard drive pretty well.  For our purposes, the take away is that hard drives are physical objects, and as such, follow the laws of physics (duh) in the following measurable ways:

1.) Capacity, which is measured in bits or bytes and exponents there of (MB, GB, TB, PB).  This is how much data will fit on your disk, from simple text files to virtual disks, and everything in between.  For example, if you have a 500GB SQL database, you darn well better have a hard drive that has a capacity of at least 500GB.  This is a pretty simple concept, so I’ll leave it there for now.

2.) Performance, which is measured in a couple ways:

- at the disk itself in Input-Output Per Second (IOPS) – a measure of how many read and write commands a disk can complete in a second

- interface throughput, measured in MBps or Gbps – a measure of the peak rate that a volume of data can be read from or written to disk

- latency – the amount of time between when you ask a disk (or storage system if you want to read ahead) to do something and when it can actually do it, very closely related to IOPS as you’ll read in a forthcoming article in this series.

Each disk, array, and storage system has its own fixed set of measurements given a specific configuration.  Knowing the physical capabilities of your storage system as measured in the above ways, and your systems storage requirements will go a long way towards a successful design and implementation of your storage environment.  The remaining parts of this series will take a look at these performance characteristics a bit more in-depth and explain what happens as you introduce factors like RAID, cache, data reduction techniques such as snapshots and deduplication, and varying workloads.

Please keep in mind that while I have designed and implemented a variety of DAS, NAS, and SAN technologies from a host of vendors including Dell, EMC, IBM, and NetApp, I am by no means a storage expert.  The information I will provide is generalized, over-simplified, and does not consider varying approaches from different storage vendors.  Nonetheless, I hope you find this useful information if you are designing a solution, troubleshooting a performance issue or preparing to make a storage purchase.

Keep Reading:

Storage Basics – Part II: IOPS