I posted an article in December on how the SVGA driver included with VMware Tools caused the guest VM to freeze.  I referenced VMware’s KB Article 1011709, which directed you to not use the SVGA drivers included with VMware Tools.  KB1011709 has since been updated (as of February 25, 2010) to indicate that the VMware Tools package included with ESX 4.0 Update 1 includes a new WDDM driver that is fully supported.  If you have updated to Update 1, you should upgrade VMware Tools to take advantage of the new driver.

If you followed the KB1011709′s original advice and did a custom install of VMware Tools (leaving out the SVGA driver through a custom install), you may have to do a re-install of VMware Tools before the new driver is available.   Once you get VMware Tools upgraded, the new driver can be found in the guest VM at C:\Program Files\Common Files\VMware\Drivers\wddm_video.  These drivers are not automatically installed, so you’ll have to update your guest’s video adapter driver in Device Manager.

It’s a bummer that the WDDM SVGA drivers are not automatically installed.  You could  probably copy these drivers to other VM’s and use Windows Device Manager to replace the standard driver with the newer WDDM driver without having to do the uninstall, reboot, reinstall of VMware tools on all of your VM’s.

Just as I was about to publish this, I saw a TweetDeck pop-up from @jasonboche saying that he had published pretty much the same update here: http://www.boche.net/blog/index.php/2010/03/28/windows-2008-r2-and-windows-7-on-vsphere/.  Not only does he have pretty pictures to go with his post, but also points out that VMware Tools installs/upgrades executed with VMware Update Manager (VUM) will not install the upgraded SVGA driver.  He also recommends updating templates to include the upgraded drivers.  Great points, Jason.

My Storage Basics series has been neglected for some time (sick kids, snow storms, VMware Upgrades, SAN implementations and some Cisco switch upgrades took all my free time), so let’s jump right in to Part V – Cache, Controllers, and Coalescing.   Between the alliteration and fancy words, it might seem like I am about to tell a tale of international espionage.  Unfortunately, my introductory treatment of these aspects of a storage system will probably not keep you on the edge of your seat – but I’ll try to keep it interesting.

Throughout this series, we’ve been working our way from the basic building block of any storage system – the disks – outwards towards the brains of the operation – the controller.  You’ll recall that in Part II I introduced IOPS and the math that goes into calculating the IOPS capacity of a disk array.  In Part III we considered a RAID implementation’s impact on performance and availability.  And most recently in Part IV we looked at the common interface types when dealing with storage arrays.  If we put the previous parts together we still don’t have a functional storage system.  The missing piece is the controller.  Simply put, the storage controller is the hardware adapter between the disks and the servers that connect to the storage.  The controller has a specific ‘interface‘ type, is responsible for RAID operations, and handles advanced storage functionality.  A controller can be as simple as the Dell PERC or HP Smart Array add-in card on your server, or as complex as the Storage Processor in an enterprise class Storage Area Network (SAN) such as an EMC CLARiiON or NetApp FAS.

Controllers

As we look at controllers and the advanced features they provide we’ll see that some of the earlier performance equations start to break down.  The simplest controllers take disk read/write commands from the operating system and send commands down to the disk(s) attached to be read or written. This gets data onto the disk, but often does not do so in an efficient or reliable manner.  RAID-capable controllers take on the added responsibility of configuring disks in the desired RAID level, calculating & writing parity data, and writing the data in disk-spanning stripes or mirrors depending on the RAID level.

Cache

To increase performance and improve reliability, storage vendors implement a caching system on their controllers.  Cache is memory that acts as a buffer for disk I/O, and is usually battery-backed to prevent data loss in the event of a power failure.  Because of the exponentially greater speed of RAM over spinning magnetic disks, cache can improve performance by orders of magnitude. Cache can operate on both reads and writes to disk.

When dealing with writes, the controller cache is typically used in one of two ways: write-through or write-back.  In write-through mode, data is written to volatile cache and then to disk, and only acknowledged as written once the data resides on the non-volatile disk.  Write-back mode allows the controller to acknowledge the data as having been written as soon as it is held in cache.  This allows the cache to buffer writes quickly and then write them to the slower disk when the disk has cycles to accept I/O.  The greater your cache size, the more data that can be buffered, ultimately resulting in better performance as measured in both IOPS and throughput.  This graph from my article on troubleshooting write performance on an IBM DS3300 iSCSI array shows how throughput increased and latency decreased when enabling write cache.  The extent to which cache increases performance is highly dependent on the workload characteristics (I/O size, randomness, and ratio of reads:writes).

Read-cache acts as a buffer for reads in a couple ways.  First, some controllers attempt to ‘read-ahead’, anticipating future read requests from the operating system and buffering what it expects to be the next blocks of desired data.  Some entry-level controllers simply buffer the next physical chunk of data and fill cache memory with it, while more advanced controllers may attempt to predict the right block of data based on previous requests (you just asked for 3 blocks in a row, I’m guessing you’ll come asking for the 4th next so I’ll just buffer it in fast cache for you now).  Secondly, read cache holds data that has been previously read, regardless of any pre-fetching the controller may have done.  This allows for much faster subsequent access of the same data because it is held in the faster cache, eliminating the need for the controller to go to disk for the data again.  Just like with write cache, the extent to which cache increases performance is highly dependent on the workload characteristics.

A given storage array controller only has so much cache to work with.  A Dell PERC5/E, for example, has 256MB of cache that can be used for both read and write.  While this may be enough for a direct-attached storage array, SAN’s serving multiple systems demand more cache.  In contrast, an EMC CLARiiON CX4-960 has 32GB.  Some storage vendors, such as NetApp, are getting creative with cache.  NetApp’s Performance Acceleration Module (PAM) is an add-in card that provides up to a whopping 512GB of Layer 2 cache to the storage system.

Caching mechanisms can dramatically influence performance under the right conditions.  With healthy cache in place, IOPS calculations become skewed.  However, cache can be exhausted or may not hold the data you are interested in.  If cache is insufficient to satisfy read requests, or has reached its high-water mark for writes, performance can drop off.  When cache is exhausted, the backing disk must be able to satisfy the I/O workload or performance will be unacceptable.  This is where the IOPS calculations kick in, and where having the right disk type and configuration really matters.

Queuing & Coalescing

Advanced storage systems introduce additional features to reduce I/O contention and improve cache utilization.  I won’t go into all of the features here because they vary by storage vendor.  However, I will point out two common techniques – queuing and coalescing.

Queuing refers to the ability of a storage system to queue storage commands for later processing.  Queuing can take place at various points in your storage environment, from the HBA to the storage processor/controller.  A little queuing may be OK depending on your workload, but too many outstanding I/Os can negatively impact performance (this is measured in latency).  Queue depths can be adjusted on many components in your storage and VMware landscape, but check with your vendor’s support group before you make changes to these settings.

Coalescing is performed by some storage systems to modify the character of the workload.  To better understand coalescing, picture a bunch of random write activity.  Without cache in place, the disk heads will be bouncing all over the platters trying to get the data on to disk.  A little write cache will allow the storage array to acknowledge the write for the OS, but the array still needs to de-stage that data from cache to disk quickly to prevent cache exhaustion.  The back-end disks will still be doing the chicken dance, bouncing around trying to write the random workload….  Now picture an intelligent system that re-orders the random writes that are held in cache and writes them to the disk in nice sequential stripes.  The disk heads will be less prone to jumping around the platter and the behavior will start to look more like a nice waltz than the funky chicken dance.  Coalescing is used for writes, not reads, so not all workloads benefit.

Wrap-up

With this article on Controllers, Cache, and Coalescing we’ll end our look at the basic building blocks of a storage array.  Before we end the Storage Basic series I am planning a few more articles on Storage Workload Characterization (which has been mentioned, but not directly addressed in this and previous articles), Identifying a Stressed Storage System, and Best Practices for Storage Performance in a VMware Environment.

If you are interested in more reading on Controllers, Cache, and Coalescing, I recommend the following:

Additional Reading:

• Impact of cache on the performance of the HP StorageWorks XP12000 Disk Array white paper
• Performance impact of controller cache: SQL Server read only workloads
• IOps? – Dig into the article’s comments for some great dialog between some people who really know their stuff!
• Storage Performance for SQL Server
• Storage Caching 101 – Chuck Hollis (EMC)
• Improving Performance with Interrupt Coalescing for Virtual Machine Disk IO in VMware ESX Server

A reader named Mark contacted me today and asked if there was a way to reduce the size of the batch output from an ESXTOP run.  And he asks for good reason: Depending on the number of VM’s on your host, the delay between ESXTOP samplings and the number of samples you collect, using the All Stats option (-a) can yield a massive file in a short period of time.  If written to a partition on your ESX Service Console you run the risk of filling the partition, and forget about actually being able to analyze the data in PERFMON or Excel.  For example, on an ESX host running ~15 VM’s I produced 100MB worth of CSV using the -a switch, sampling every 15 seconds, for just under 2 hours.  ESXTOP uses 10-second intervals by default; I used -d 15 to change the sampling delay.  Had I went with the default my output would have been bigger.

To reduce the size of your output, you can change your sampling delay to something larger, say 30-seconds.  I suppose you could also capture statistics when the host is not busy so you get fewer characters in the results, but that’s just being goofy.

A better way to reduce your ESXTOP output size is to selectively include only the statistics you are interested in, and is really what Mark was asking.  After all, all statistics from ESXTOP can be too many statistics, and chances are you already know what stats you are interested in.  Here’s how you can narrow down the collected stats for easier analysis and smaller output:

1. Enter ESXTOP in interactive mode on the Service Console by simply typing esxtop at the # prompt
2. Switch to a component you are NOT interested in capturing statistics on by pressing the corresponding menu option (c: ESX cpu, m: ESX memory, d: ESX disk adapter, u: ESX disk device, v: ESX disk VM).
3. Press f when viewing the component you do not want to capture.  A list of fields will be displayed.  You can toggle the fields on and off by pressing the letter corresponding to each field.  An * indicates that the field is on.  You want to turn off all of the fields you don’t want to collect.
4. Repeat steps 2 & 3 for the remaining components, leaving only what you want to capture.
5. Switch to the component you want to capture in batch mode and repeat step #3, except you will now enable what you want to capture.
6. Press W (capital W – case sensitive) to write out the ESXTOP configuration file.  You can accept the default or create new configuration files.  You may want to create a CPU-only config file, memory-only, and so forth.
7. Press CTRL+C to stop ESXTOP.
8. Now, invoke ESXTOP in batch mode, calling your updated or new configuration file you created in step #6 using the -c switch.  Here’s an example:# esxtop -b -d 30 -n 480 -c .esxtopcpustats > /tmp/esxtop_cpu_stats.csv where .esxtopcpustats is an ESXTOP config file with only CPU stats.  -d sets your capture interval to 30 seconds, and -n sets the number of samples to 480 (or 4 hours with a delay of 30 seconds).

Once your capture is complete you can replay the sampling in ESXTOP using replay mode (-R), or you can copy the .csv to a Windows system and use PERFMON or Excel to analyze the stats.  If using PERFMON or Excel you will notice that the system summary information displayed at the top of an interactive ESXTOP session is included in the output (console memory, console cpu, etc.).  As far as I know, there is no way to disable this, nor would you want to as it includes the time stamp necessary to interpret your data.

It is possible to use the vSphere CLI or the vSphere Management Assistant (vMA) to run RESXTOP, a version of ESXTOP designed for remote administration of ESXi or ESX.  You may note, however, RESXTOP from the vSphere CLI only works from a Linux client.  Using either of these tools will help you to automate ESXTOP statistics collection from multiple hosts using customized configuration files.

Microsoft published a document named “Getting to Know Hyper-V: A Walkthrough from Initial Setup to Common Scenarios” last week.  According to Microsoft, “this guide provides detailed step-by-step walkthroughs for testing Hyper-V on a pre-production environment. You can use this guide to become familiar with Hyper-V and the process of creating and managing virtual machines. Also included in this guide are useful scenarios that you can test to better understand how Hyper-V can address the business goals of your organization.”  The document serves as a sort of evaluators guide for Hyper-V, stepping the reader through everything from enabling VT in BIOS through virtual networking.  It also includes some sections on using snapshots, base virtual machine templates, and managing Hyper-V based virtual machines remotely with Hyper-V Manager.  If you want more in-depth documentation on Hyper-V you can go through http://technet.microsoft.com.

As a side note, Microsoft has published the Microsoft Manual of Style for Technical Publications to help standardize technical documentation.  I have long been a fan of Microsoft’s technical documentation for its easy to read style, although it sometimes lacks the depth that I desire.

While we’re on the topic of virtualization documentation, I have also been quite pleased with VMware’s technical documentation over the years, and have found it to be continually increasing in quality, providing very specific technical guidance and references to additional resources.  I have also been pleased to see that VMware has improved delivery options for documentation.  VMware offers several formats for documentation delivery, including web-based and PDF’s.  Start with the Documentation Roadmap for a quick introduction to the available documentation, and where to find what you need.

You can find web-based vSphere documentation here: http://pubs.vmware.com/vsp40/.   The web-based documentation is great for running searches on.  All vSphere documentation can be accessed through this page: http://www.vmware.com/support/pubs/vs_pages/vsp_pubs_esx40_vc40.html.  If you want to do a full grab of all of VMware’s documentation for an in-house repository (e.g. SharePoint), check out xtravirt’s VMware Documentation Downloader script.

If you are looking for quick and easy evaluator guide-type documentation from VMware, check out these resources: ESXi Installable and vCenter Server Setup Guide and the Virtualization Kit (registration required) at http://www.vmware.com/resources/wp/virtualization101_register.html.

There is a ton of less formal VMware documentation in several places:

• Technical resources and case studies here: http://www.vmware.com/resources/techresources/
• Proven practices around Strategy, Applications, Security, Management, and Availability at VIOPS.
• Official VMware Blogs at http://www.vmware.com/vmtn/planet/vmware/.
• Community blogs aggregated by VMware at Planet v12n: http://www.vmware.com/vmtn/planet/v12n/
• VMworld Recorded Sessions & Labs (VMworld 2009 Sessions available as of today, September 14th) at http://vmworld.com.
• The VMware Community Forums: http://communities.vmware.com/
• And, 3rd party books like Scott Lowe’s Mastering VMware vSphere 4.

Do you have other sources of virtualization documentation or easy methods of searching documentation to find exactly what you need when you need it?  If so, leave a comment!

I needed to grab some stats from my ESX hosts for off-line analysis so I fired up my trusty ESXTOP intent on using batch mode to capture a .csv formatted output.  I started to manually select the counters I was interested in while working in ESXTOP interactive mode (you can save your selected counters to the esxtop configuration file with the ‘w’ command) and thought that there must be a better way.  I found that better way in the VMware Performance Community: http://communities.vmware.com/docs/DOC-3930.  There is now a -a switch that can be used to include ALL performance counters.  I’m sold.

I wanted detailed information, so I decided on a 15 second capture interval to run for a 2 hour window.  Here’s the command I used:

esxtop -a -b -d 15 -n 480 > /tmp/esxtopout.csv

where -a is for ALL, -b is for batch mode, -d is for delay, and -n is for the number of iterations ((60/15)*60*2).  I wrote out the results to a .csv in /tmp.  The resulting CSV weighed in at a whopping 100MB after 2 hours.

The CSV can be analyzed in Excel (pivot tables work well for this) or in Windows Perfmon.  I opened the log in Perfmon as I was after basic Min/Average/Max counters and Perfmon makes those easy to see.  When adding the CSV log to Perfmon, you are prompted to select counters.  I added all instances of Commands/sec, Reads/sec, and Writes/sec from Physical Disk (I was gathering some IOPS counts for a new storage proposal). I got a bit more than I bargained for: a mostly unresponsive Perfmon window and the ugliest darn graph I’ve ever seen.

Switching from a graph view to the report view allows you to easily view and remove specific counters that you are not interested in, or open the Properties of the data set, switch to the data tab and bulk select counters that you want to remove.  I was not interested in vmhba1:x, specific VM’s or worlds, so I killed all of those, leaving just the base iSCSI device (vmhba32 in my case).

After some cleanup the graph looked a bit better and more importantly, I was able to easily read my Min/Average/Max stats:

Here are the takeaways -

• ESXTOP is a powerful utility for performance monitoring
• All stats (-a) can result in a huge file – use it wisely in batch mode; else use interactive mode to select your counters and write them to the user-defined configuration file.  Invoke the config file with the -c option when running in batch mode.
• Consider using vscsiStats for more granular reporting.
• ESXTOP physical disk stats do not include NFS volumes.

Do you use other tools or methods to collect basic disk IO counters for storage sizing purposes?  If so, leave a comment describing your approach!

I have been meaning to write this up for a while; Scott Drummonds’ ‘Love Your Balloon Driver’ post today at his Virtual Performance blog gave me a nice reminder.  I actually caught a sneak peak at the graphs with an explanation from Scott at his instructor-led lab at VMworld 2009.  Scott calls out that the only workload they discovered suffers from balloon driver activity is Java.  The reason for Java’s problems with balloon driver activity is that Java itself runs in a VM and so the guest OS cannot properly determine which pages should be swapped out when the balloon driver calls for it.

My experiences causes me to agree with Scott and the whitepaper he cites – in a properly designed and equipped environment the balloon driver is not detrimental for most every workload to a point.   However, I recently discovered in a client site that the balloon driver can cause significant issues when the environment is poorly designed and under-sized.  Here the background:

I was called into an already established environment where the client was running on an older blade with VMware ESX 3.5.  The blade maxed out at 16GB RAM and had dual dual-core CPU’s with no hope for an upgrade.  On the blade was a single guest VM running Windows 2003 with SQL 2005, in it’s full 32-bit glory.  The VM was configured with 4 vCPU’s and 16GB of memory.  Some of you can probably already guess where this is going….

The x86 Windows guest had PAE configured, and SQL took advantage of AWE to use the additional memory beyond the 4GB limit of a 32-bit system.  Additionally, the Windows guest had the /3GB switch enabled in boot.ini.  Finally, as per SQL best practices, the ‘Lock Pages in Memory‘ permission was granted to the SQL Server service account.  What the guest was left with was 1GB of kernel mode memory and 15GB of User Mode/Extended addressable memory.

And here’s the problem.  The client was using ESX, not ESX 3.5, so the Service Console required memory.  In this case, the service console had approximately 512MB allocated to it.  Futhermore, VM’s require some overhead on ESX to run.  The memory overhead consumed by a Windows guest on ESX 3.5 with 4 vCPU and 16GB of memory is a bit more than 512MB.  On a properly sized ESX server with multiple similar guests/workloads, you could probably gain much of the overhead back through transparent page sharing; but in this case I had a 1:1 P2V ratio.  If you are any good at math you see that the environment is running about 1GB short of memory.  A quick check of the balloon driver stat in vCenter show that the balloon driver was constantly active and demanding about 1GB back from the guest… constantly.

Under normal circumstances this might not be an issue, but in this case the Windows guest was being absolutely punished.  The guest CPU’s were pegged at 100% with an excessive amount of kernel time, often indicating IO issues.  And indeed I did experience terrible disk and network performance on the guest.  At the root of the problem is this – the Lock Pages in Memory permission allows SQL to get a firm grasp on the user mode memory available to it (15GB) and lock it up.  This left the already starved (because of the 3GB switch in the boot.ini) guest kernel with it’s 1GB the only thing the balloon driver could really swap out.

The client suggested a reservation of 16GB on the VM, knowing that memory reservations prevent balloon driver activity.  I calmly asked them to back away from the keyboard as I explained how if a starved guest was bad, how much worse a starved Service Console would be.  In the end the fix was quiet easy – I convinced the customer that they should reduce the amount of memory allocated to the guest by about 1GB, enough to let the 512MB SC and the 512MB of overhead run without contention.  I was able to show them the difference between allocated and active memory in vCenter – the 1GB being surrendered was not really being actively used, SQL just had it locked up.  In fact, surrendering the 1GB of memory back to ESX breathed new life into the guest VM, bringing its performance back in line with expectations.

Ideally, I would have brought in a bigger ESX server that could serve additional VM’s, driving greater levels of efficiency across the environment.  It just wasn’t an option for the client in this case.  In the end, the problem was fixed and I was reminded just how fun it can be to explain some of these backwards sounding virtualization concepts to customers – fewer vCPU’s can lead to better performance of guests, less guest memory can fix performance issues, and increasing the quantity of similar guests on a host can drive better performance to a point because of transparent page sharing.

Stay tuned over the next few weeks as I digest and write on my VMworld experience – from VMUG activities to Paul Maritz’s press conference announcing the vCloud Express, and plenty of great sessions in between.  Like many of you, I returned from VMworld with quite a backlog of work but I’ll do my best to squeeze in some posts and tweets.

I have been pulling my hair out with a small VI3 implementation running against an IBM DS3300 iSCSI array.  Performance, for lack of a better term, sucked.  Granted, the DS3300 is not an enterprise level workhorse of a storage system, but it fit the budget.  Read performance was decent from the array, but write performance was terrible, maxing out at 10Mpbs throughput and insanely high latencies on long writes when the system was under load.  This led to some long P2V operations, poor guest performance, and some questions from the project sponsors on why I couldn’t make the environment sing.

The system was configured with a single controller with dual GigE NIC’s.  The controller had 512MB of battery backed cache (there is also a 1GB cache upgrade option available).  I wrote off some of the poor performance to a single controller with a less-than-optimal amount of cache; blamed the SAS controller to SATA disk command translation overhead; cringed at the 6 disk RAID5 configuration; and engaged in some self doubting.  I convinced the powers that be that we were IO constrained and got some funds to fill out the 3U chassis to a full 12 SATA disks, and reconfigured the array as a RAID10.  Performance gains were almost unnoticeable with these changes.  In addition, I did some basic troubleshooting of the network environment, verifying multiple paths to the storage, setting Flow Control on the switches to receive only, and double-checked my iSCSI initiator settings.  Note: The DS3300 is only supported with the ESX software initiator.  I found documentation on the DS3300 to be lacking, but did discover that the Dell MD3000i is based on the same LSI Engenio array.  Some Googling on the Dell solution led to to the ‘SMcli’ command line interface for both arrays.   The commands are slighly different for the Dell and IBM.  The links to the IBM CLI documentation were broken, so I had to do a bit of trial and error to get the commands right.  I used the Dell documentation as a starting point.  (Rant: Seriously, IBM?  Can you make your documentation any harder to get through – is it a Redbook, is it an Engineering Whitepaper, is it a support document, is it a case study – and why can I only find these with complex Google searches, not on your own product pages, and why can’t you name for documents intelligently, not with some random string of characters).

Update: The IBM System Storage DS3000, DS4000, and DS5000Command Line Interface and Script Commands Programming Guide is here: IBM System Storage DS3000, DS4000, and DS5000Command Line Interface and Script Commands Programming Guide – DS3k4k5kCLIreference, SMCLI

Moving on… I received an automated alert from the DS3300 about an incomplete battery learn cycle.  Using the IBM Storage Manager GUI I generated a  Storage Subsystem Profile’ from the Support tab to check the battery status.  In the profile I discovered that while write cache was enabled, it had a status of “Enabled (Suspended)”.   Ah ha!  Now I’ve got some decent Google material that led me to this: http://communities.vmware.com/thread/195838.  Hot damn I love the VMware Community Forums!

It turns out that in a single-controller configuration the setting for cache mirroring remains enabled by default.  Because there is no 2nd controller to mirror to, the array suspends write caching.  This is probably a safety thing – loss of high availability on the controllers puts data in cache at risk should the only controller fail.  I weighed my options and decided that the poor performance I was experiencing beat HA concerns, so I enabled write cache on the array using this command:

c:\program files\ibm_ds4000\client>smcli -n <ARRAYNAME> -c “set allLogicalDrives mirrorEnabled=false;”

And then followed with this for good measure:

c:\program files\ibm_ds4000\client>smcli -n <ARRAYNAME> -p <arraypassword> -c “set allLogicalDrives writeCacheEnabled=true;”

The results were immediately noticeable:

DS3300 Performance Improvement when Write Cache is Enabled - Click for a Larger View

The screen shot is from Veeam Monitor Free Edition, taken during 4 concurrent V2V operations from Hyper-V to VMware.  With the write cache fully functional, disk usage peaked at 54MBps, latency dropped to about 6ms, and my blood pressure dropped a few notches.

While poking around the CLI I also found that you can dump performance stats from the array (performance is otherwise hard to find on the thing) using this command:

C:\Program Files\IBM_DS4000\client>smcli -n <ARRAYNAME> -c “set session performanceMonitorInterval=5 performanceMonitorIterations=120;save storageSubsystem performanceStats file=\”c:\\ds3300perfstats.csv\“;”

This will give you a 10 minute record of performance from the array which you can analyze using Excel.  The Dell Enterprise Center TechCenter Wiki has a great write-up on how to efficiently analyze the data from this command here: http://www.delltechcenter.com/page/MD3000i+Performance+Monitoring, complete with a YouTube video that walks you through the process:

I am beginning to think that the DS3300 (and MD3000i) may actually be a viable starter solution for SMB’s starting out on a virtualization project.  But I would recommend the cache upgrade, 2nd controller, SAS disks instead of SATA to eliminate the SAS-to-SATA translation overhead and more faster disks instead of fewer slower disks so you can drive throughput and IOPS to a higher level.

Have any of you deployed the DS3300 or MD3000i (or the generic LSI solution)?  Do you have any performance tuning tips for these arrays?  If so, share in the comments!

VMware vExpert and fellow Northern Virginian, Ken Cline, has posted an excellent article on his Ken’s Virtual Reality blog that aims to demystify VMware networking.  The article, the first in a new series by Ken, provides an overview of networking in an ESX/ESXi environment and breaks down the intricacies of the vSwitch and VLANs.  The article comes complete with some nifty diagrams to help make sense of the topic. The timing of this article is great for me as it helps to frame my thoughts as I delve into the design of my latest VMware project on an IBM BladeCenter with IP SAN storage.

Great article, Ken!  I look forward to reading the rest of the series.

One more post to wrap up the nonsense with my DL380 G3 ESX servers….

Vincent Vlieghe noted that you must make a couple changes to your DL380 G3′s for ESX to work correctly.  His post was written back in 2006 when we were still working with ESX 2.x, but the same appears to be true of ESX 3.5 RTM (Updates are not supported on this hardware per the HCL).  The changes you must make to BIOS are:

For stable operation on these systems, ESX Server requires a BIOS MPS Table Mode setting of Full Table APIC. With the exception of the specific systems referenced below, the following BIOS settings must be applied in order if available:

1. System Options > OS Selection: Select Windows 2000.
2. Advanced Options > MPS Table Mode: Select Full Table APIC.
3. When presented with multiple Windows options (Windows 2000, Windows Server 2003, Windows .NET, and so on) select Windows 2000. If both BIOS settings are available and can be modified, both must be set correctly. You should confirm these settings after any BIOS upgrade operation.

I have seen other references that say that you should also disable hyperthreading on this platform, but I was able to successfully run with Hyperthreading enabled with no performance degradation or stability issues.  I hope this information is helpful to those of you still running these dinosaurs!

I wrote some time back about networking problems with a clean install of ESX 3.5 U3 on a HP DL380 G3 server in a lab environment.  A simple downgrade to ESX 3.5 RTM corrected the issue and I didn’t think much about it.  One of the servers in the lab died and I went about the business of rebuilding it.  Having learned my lesson, I started with an ESX 3.5 RTM install and then patched to Update 3 plus other applicable updates.  Much to my chagrin, the server began crapping out on me randomly.  Some reboots, some networking issues, and other assorted not so good things.  Now the DL380 G3 is not the spring chicken it used to be, so I assumed some faulty hardware was probably to blame.  Some diagnostics and log reviews yielded no hardware issues.

On a whim, I decided to check the VMware HCL to see if the DL380 G3 was still on the list of compatible servers for ESX.  Now, I had checked, or rather ‘remembered’ checking, the HCL before that first problematic install, but a recheck never hurts.  When I arrived at the VMware HCL page I saw the same old trusty PDF link with a slightly newer revision date than my previous visit.  I was pleasantly surprised when I clicked the PDF link to find that I was redirected to a searchable, filterable forms-based version of the HCL.  Nice!  Let’s do this thing….

I’m a little lazy, so I simply used a keyword search to look up ‘DL380 G3′.  Presto-chango: I’ve got results, and I like what I see:

Search Results for DL380 G3 on the VMware HCL

My eyes jump right to ESX 3.5 – Supported, on my platform, no further questions your honor.  Close the old browser window and move on with my life, my life being troubleshooting this darn server.

A few hours later I am still struggling with the server and turn to Ebay for salvation.  “If you can’t beat em, cheat em,” my grandfather used to say.  I’ll find new hardware for my lab.  I identified some other hunk of junk that just might work and decided to check the HCL for it.  That’s when it jumped out at me: there are Update versions included in the HCL and I had been to quick to see it on my DL380 G3 search.  Back to the HCL.

This time I just do a search for ‘DL380′, leaving off the Generational notation and get the following:

Search Results for DL380 from the VMware HCL

The ProLiant DL380 G5 with Quad-core Intel Xeon processors lists ESX 3.5 U3, ESX 3.5 U2, and ESX 3.5 U1 as supported releases, along with the RTM ESX 3.5.  The Update versions are not listed for the G3 or G4.  After some self-deprecating curses and a reinstall of ESX 3.5 Update-nada, stability returned.

The lesson learned, double-check the HCL (or if you are a little slow like me, a triple-check doesn’t hurt).  The HCL is major version and Update-revision sensitive.  And, not all models are treated equally.  You’ll notice in the picture to the left that the DL380 G5 has different supported releases depending on the CPU Model.

Also, keep in mind that you need to verify that all components of your VMware infrastructure are on the HCL from Servers and Systems to IO Devices, and Storage/SAN.  The VMware HCL site offers some basic tips for searching here: http://www.vmware.com/resources/compatibility/help.php.

Here’s the real take-away: The VMware HCL is there for a reason.  Sure, you might be able to get something that is not on the HCL to work, but you may experience instability along the way.  In the event that you are running a non-HCL system you may also find that VMware Support may be limited in what they can do for you.

VMware released version 1.5 of the VI Toolkit for Windows – the PowerShell management and reporting tool of choice for many VMware administrators. The new version carries build number 142961. You can download v1.5 here: http://blogs.vmware.com/vipowershell/. The update includes some 32 new cmdlets, including ones for getting/setting NTP settings on ESX, getting/setting Advanced configuration options on ESX, getting/setting ESX Firewall settings, and the ability to modify DRS rules using PowerShell. Existing cmdlets have also been updated with new parameters, and several fixes have been introduced. Check out the release notes here: http://www.vmware.com/support/developer/windowstoolkit/wintk15/windowstoolkit15-200901-releasenotes.html.

There are plenty of examples on the Internet to get you started with the VI Toolkit for Windows. Check out these sites to get started:

Start at the VMware Community site for the Windows Toolkit for great examples and a little help from some friends: http://communities.vmware.com/community/developer/windows_toolkit/
There are also some good example scripts and resources floating around, such as:

http://vmetc.com/2008/08/27/powershell-scripting-examples-for-vmware-virtual-infrastructure/
http://www.peetersonline.nl/
http://www.ivobeerens.nl/?p=106
http://www.vmguru.com/

Not a hard-core scripter? Grab this handy tool for a little GUI on your PowerShell, and extend it with the VMware Infrastructure PowerPack 2.0

What tools or examples are you using to extend the power of PowerShell into your Virtual Infrasturucture?  Leave a comment to share!

Between budget cuts and New Year’s resolutions, improving your security posture is probably near the top of your to-do list.  Much has been made of security concerns in a virtual environment, but it is always good to re-visit your configurations and make sure they are still on par with recommended best practices.  I began re-reviewing VI security best practices after reading at post by Bob Plankers at The Lone SysAdmin (Bob has been on my reading list for years – he has a great style and always brings fresh insights) on why you would want a second super-user account on your ESX servers.

We certainly all have our own opinions and operations procedures when it comes to configuring and hardening our environments, but I decided to take a look at what the experts had to say on this particular subject and other basic build and hardening recommendations.  Here is what I found:

VMware Security Resources

VMware Security Utilities

VI3.5 Security Hardening Whitepaper

Defense Informaion Systems Agency (DISA) ESX Server Security Technical Implementation Guide

DISA ESX Server Checklist

As a side note, DISA publishes many STIG’s at http://iase.disa.mil/stigs/.  Your tax dollars paid for these, so you might as well check them out.

NSA VMware ESX  Server 3 Configuration Guide

There are also numerous tips and scripts for locking down your virtual infrastructure in the VMware Community Forums (Start here: http://communities.vmware.com/message/941372).

So back to the question of second super user accounts: It seems that best practices are to create a second user account with sufficient access to the console, granting that user SUDO privledges, and then disabling the default root account.

Microsoft released version 2 of their Offline Virtual Machine Servicing Tool (OVMSTv2) yesterday.  The updated version of the tool, a component of Microsoft’s Solution Accelerator family of ‘best-practice-meets-tools’ adds integration with System Center Virtual Machine Manager 2008, and support for System Center Configuration Manager 2007 SP1, System Center Configuration Manager 2007 R2, and Windows Server® Update Services 3.0 SP1.  The tool works in much the same way as VMware’s Update Manager product updates Windows guest VM’s. Support for System Center Virtual Machine Manager 2008 obviously means that this product could supplement, replace or conflict with VMware’s Update Manager’s ability to patch Windows guests hosted on a VMware Virtual Infrastructure.

There is not yet much information on this version of the Offline Virtual Machine Servicing Tool.  A MSDN blog post from May 2008 includes some information on the beta version of the product.  The Getting Started guide included in the OVMSTv2 .zip download reveals a bit more about the requirements and footprint required to implement the solution.  The solution requires a good deal of Microsoft software, including (depending on your environment and requirements) Microsoft Active Directory, Windows Server Update Services (WSUS), Configuration Manager 2007, and Virtual Machine Manager 2007, and an appropriately designed Group Policy (GPO) environment.  Another requirement is that the target VM’s have DHCP assigned IP addresses.

I did not read anywhere in the documentation that the OVMSTv2 integrates snapshot capabilities to provide simple roll-back of guests that suffer problems on Patch Tuesday, as VMware Update Manager does.  I also did not see a reference to network isolation capabilites in OVMSTv2.  VMware Update Manager keeps offline VM’s offline during the patching process.  OVMSTv2 has to reach out to a WSUS or Configuration Manager server, so I suspect that the guest is not isolated during that patching process.

I have not had a chance to build up all of the requirements of this tool in the lab, so I don’t have any practical advise for using the tool.  Have any of you used the beta or release versions of the tool?  Post a comment to let everyone know your experiences.

The Microsoft Offline Virtual Machine Servicing Tool v2 could be a good tool to add to your Microsoft patch management arsenal, but the small footprint and simplicity of VMware Update Manager will most likely keep all but the most dedicated Microsoft shops from implementing OVMSTv2.

I began building up a new lab environment with a few old DL380 G3 servers (courtesy of ebay) and pulled down the latest ISO of ESX 3.5 with Update 3 integrated.  The install went well with my only changes to the defaults being some partitioning changes (I more or less follow the recommendations set forth by Rich at VM/ETC for partitioning) and not checking the option for creating a default network for virtual machines – I prefer to set up all my networking manually once I am in the VI client.  I selected the first on-board NIC for my Service Console connection.  The NIC was detected as a BCM5703, but the server really has the Broadcom NC7781 chipset.  I thought it odd, but continued with the install.  After ESX came up the SC had no network connectivity.  The NIC’s appeared but the drivers and TCP/IP stack could not be initialized against them.  I was able to reproduce the same behavior on two different DL380 G3′s.

A quick Google indicated some issues with the DL380 G3 and ESX (enabling hyperthreading causes performance problems, incorrect ACPI configuration in BIOS can cause system instability, etc.).  The NIC firmware was out of date.  Updated NIC firmware yielded the same problems on a re-install.  I didn’t have much time to troubleshoot the issue as I was trying to stage a demo of NetApp’s SnapManager for Virtual Infrastrucuture, so I reinstalled using an ESX 3.5.0 disk.  The install went fine with no network connectivity issues.  I was able to patch the installation to Update 3 using Update Manager without problems.

I don’t want to shoot from the hip, but I suspect the problems may be related to the updated support for Broadcom NIC’s as described in the ESX Update 3 Release Notes.  I’ll try to troubleshoot this out as time allows and will post an update if I uncover anything of great interest.  Or better yet, maybe I’ll start looking around for some better hardware for the lab.  Leave a comment if you have had a similar issue.