If you followed the KB1011709′s original advice and did a custom install of VMware Tools (leaving out the SVGA driver through a custom install), you may have to do a re-install of VMware Tools before the new driver is available.   Once you get VMware Tools upgraded, the new driver can be found in the guest VM at C:\Program Files\Common Files\VMware\Drivers\wddm_video.  These drivers are not automatically installed, so you’ll have to update your guest’s video adapter driver in Device Manager.

It’s a bummer that the WDDM SVGA drivers are not automatically installed.  You could  probably copy these drivers to other VM’s and use Windows Device Manager to replace the standard driver with the newer WDDM driver without having to do the uninstall, reboot, reinstall of VMware tools on all of your VM’s.

Just as I was about to publish this, I saw a TweetDeck pop-up from @jasonboche saying that he had published pretty much the same update here: http://www.boche.net/blog/index.php/2010/03/28/windows-2008-r2-and-windows-7-on-vsphere/.  Not only does he have pretty pictures to go with his post, but also points out that VMware Tools installs/upgrades executed with VMware Update Manager (VUM) will not install the upgraded SVGA driver.  He also recommends updating templates to include the upgraded drivers.  Great points, Jason.

Related posts:

1. Windows Server 2008 R2 & Windows 7 Freeze When Using SVGA Drivers I recently ran into an issue when installing my first...
2. Networking Problems with ESX 3.5 Update 3 on the DL380 G3 I began building up a new lab environment with a...
3. Installing PowerPath/VE using VMware Update Manager I am finishing up an installation of an EMC Clariion...

The process of setting up an additional VUM patch repository to host PowerPath/VE (and other 3rd party extensions such as the Cisco Nexus 1000v) is pretty straight forward.  3rd party extensions are supported in VUM beginning with vSphere 4.0 Update 1.  Chad Sakac has posted a great video guide on YouTube that covers the setup:

I opted to use the tomcat installation on the environment’s vCenter server to host the PowerPath/VE repository.  To accomplish this, I simply created a new directory in the tomcat root directory.  The default path for the root directory on a vSphere vCenter Server is “C:\Program Files\VMware\Infrastructure\tomcat\webapps” (or C:\Program Files (x86)\VMware\Infrastructure\tomcat\webapps on a 64-bit installation).

I created a directory named ‘depot’ and within that directory created a PowerPathVE folder.  I extracted the contents of the VUM folder from the PowerPath .zip file that I downloaded from http://powerlink.emc.com.  A screenshot of the directory is below:

PowerPath/VE Depot Directory Tree

After creating the directory for the patch repository, I simply added an Extension Repository to VMware Update Manager as Chad shows in his video.  I would like to call out one caveat – Because vCenter may not listen on standard HTTP/HTTPS ports, I used https://vcenter.domain.local:8443/depot/PowerPathVE/index.xml as the path to the source.

VUM Patch Source

Once PowerPath was added to an Extension Baseline in VUM, I simply had to scan my hosts for updates and remediate.  Installation of PowerPath/VE requires the host to be in Maintenance Mode and concludes with a reboot.  Pretty simple.

Then all you have to do is fight through an overly-complex licensing setup (seriously, a 112 page PDF on how to install licenses???), a bit of configuration, and you are multi-pathing with the best of them.  If you are interested in learning more about PowerPath/VE, start with this whitepaper: EMC PowerPath/VE for VMware vSphere Best Practices Planning.  For a bit of real-world insight into the performance increase you might see with PowerPath/VE, check out this blog post from Eric Sloof: Massive I/O power increase using EMC PowerPath/VE.

Update – 3/27/09: VMware published a Knowledge Base article on this procedure a few weeks after I wrote this post.  You can find it in  article 1018740.

Related posts:

1. EMC Virtual Storage Integrator Update I upgraded my in-house VMware vSphere environment to 4.0 Update...
2. Networking Problems with ESX 3.5 Update 3 on the DL380 G3 I began building up a new lab environment with a...
3. Virtualization Bookmarks for August 28th Here are some bookmarks for resources that I have recently...

If we stick with a single-disk-centric approach as we did in Part I, IOPS is quite simply a measure of how many read and write commands a disk can complete in a second.  IOPS is an important measure of performance in a shared storage environment (such as VMware) and in high-transaction-rate workloads like SQL.  Because hard drives are forced to abide by the laws of physics, the IOPS capabilities of a disk are consistent and predictable given a specific configuration.  The formula for calculating IOPS for a given disk is pretty straight forward (please show your work):

IOPS = 1000/(Seek Latency + Rotational Latency)

Exact latencies vary by disk type, quality, number of platters, etc.  You can look up the tech specs for most drives on the market.  As an example, I have randomly chosen the technical specifications of the Seagate Cheatah 15k.7 SAS drive.  This particular drive has the following performance characteristics:

- Average (rotational) latency: 2.0msec

- Average read seek (latency): 3.4msec

- Average write seek (latency): 3.9msec

Using the read latency number, the math works out like this:

1000
———- = 185 maximum read IOPS
2.0+3.4

The maximum write IOPS will be a bit less (~169IOPS) because of the higher write seek latency.  Writing is more ‘expensive’ than reading and therefore slower.

Fortunately, there are some widely accepted ‘working’ numbers, so you do not have to use this formula for each and every disk you might consider using.  Because rotational latency is based on the rotational speed, we can use the published Rotations Per Minute (RPM) rating of the drive to guess-timate the IOPS capabilities.  Typical spindle speeds (measured in RPM) and their equivalent IOPS are in the table below.

RPM………IOPS

7,200          80

10,000       130

15,000       180

SSD           2500 – 6000

While not a traditional spinning disk, I have also included Solid State Disks (SSD’s) for reference as SSD’s are starting to see increased market adoption.  I have seen a wide range of sizing IOPS for SSD depending on the technology, type (SLC, MLC, etc.)  Check out http://en.wikipedia.org/wiki/Solid-state_drive for an introduction, and ask your vendors for more in-depth technical information.

If you are brand-new to this (and you are still reading, congrats!), you can see how many IOPS your Windows computer is asking for by opening Performance Monitor and looking at the ‘Disk Transfers/sec’ counter under Physical Disk.  This is a sum of the ‘Disk Reads/sec’ and ‘Disk Writes/sec’ counters as you can see in the screenshot below:

If you are after some stats for your VMware ESX environment, check out esxtop and looking for CMDS/s in the output.  I published a couple articles on using esxtop here and here.  The numbers from PerfMon and esxtop get you pretty close but can be skewed by a few things we’ll discuss in later posts.

Now that was fun and all, but let’s get real: Single-disk configurations are uncommon in servers.  As such, we’ll part ways with our Simple Jack single disk approach to storage and begin to look at more real-world multi-disk enterprise-class storage configurations.  A discussion of IOPS in a multi-disk array is a great way to start.  From a very elementary perspective, you can combine multiple hard drives together to aggregate their performance capabilities.  For example, two 15k RPM disks working together to server a workload could provide a theoretical 360 IOPS (180 + 180). This also  scales out so ten 15k RPM disks could provide 1800 IOPS, and 100 15k RPM disks could provide 18,000 IOPS.

Designing your environment so that your storage can deliver sufficient IOPS to the requesting workload is of utmost importance.  If you are working on a storage design, arm yourself with data from perfmon, top, iostat, esxtop, and vscsiStats.  I typically gather at least 24 hours of performance data from systems under normal conditions (a few days to a week may be good if you have varying business cycles) and take the 95th percentile as a starting point.  So from a very simple approach, if your data and calculations show a 1800 IOPS demand at the 95th percentile, you ought to have at least ten 15k RPM disks (or twenty-three 7.2k RPM SATA disks) to achieve performance goals.  It’s amazing how some simple data and a pretty little Excel spreadsheet can help you understand and justify the right hardware for the job.

Now before you go and start filling out that PO form for a nice new storage system based on these numbers there are a few more things we ought to discuss.  RAID, cache, and advanced storage technologies will skew these numbers and need to be understood.  Stay tuned to future articles in this series for more on those topics and more.

Finally, there has been a bunch of activity in the VMware ecosystem of vendors, bloggers, and twittering-type-folks around storage performance.  As this here post sat in my drafts folder, Duncan Epping posted this gem of an article that pretty much included all of the content of this article, as well as future ones in my series: http://www.yellow-bricks.com/2009/12/23/iops/.  Do yourself a favor and read his post and the comments from his readers – both are filled with a ton of great information, including some vendor-specific implementations.
I was led to Duncan’s article by a post by Chad Sakac on his blog: http://virtualgeek.typepad.com/virtual_geek/2009/12/whats-what-in-vmware-view-and-vdi-land.html.  This is also a great read that covers some of the same information with a focus on VMware View/VDI and is also worth a few minutes of your time.  Also check out http://vpivot.com/2009/09/18/storage-is-the-problem/ for a rubber-meets-the-road post from Scott Drummonds on the importance of storage performance vis-a-vis IOPS in a VMware-virtualized SQL environment.

Related posts:

1. Storage Basics – Part VI: Storage Workload Characterization Most of what I covered in Storage Basics Parts 1...
2. Storage Basics – Part I: An Introduction I am increasingly finding that both my SMB and Enterprise...
3. Storage Basics – Part V: Controllers, Cache and Coalescing My Storage Basics series has been neglected for some time...

Related posts:

1. Update: SVGA Drivers on Windows 2008 R2 and Windows 7 I posted an article in December on how the SVGA...
2. Balloon Driver Problems with SQL I have been meaning to write this up for a...
3. VI Toolkit for Windows v1.5 Released Today VMware released version 1.5 of the VI Toolkit for Windows...

There are other issues with VMware virtual hardware upgrades, however, that you may not have experienced.  One such issue that I have experienced is highlighted in VMware Knowledge Base article 1013109: “Upgrading virtual hardware in ESX 4 may cause Windows 2008 disks to go offline“.  The problems described in the article are unique to Windows 2008 Enterprise and Datacenter editions only.  The problem is pretty well described in the title of the article – Upgrading virtual hardware in ESX 4 may cause Windows 2008 disks to go offline.  In this case, like with the ghost NIC’s I described last week, is more of a Microsoft issue, but it will rear its head when a VMware Administrator least desires it.  With this particular problem, the Windows Virtual Disk Service (part of the native Storage Management suite) is set to not auto-mount newly discovered disks that do reside on a shared bus.  Microsoft has a MSDN article on the VDS SANS policy here.  Upgrading the virtual hardware version causes the disks to be re-discovered and not auto-mounted.  This can potentially impact all non-system disks on a VM.

You may also experience similar issues when upgrading the vSCSI adapter in a VM from a standard LSI Logic Parallel SCSI adapter to a (new in vSphere 4.0) paravirtualized SCSI (pvSCSI) adapter, move virtual disks to new vSCSI adapters to increase the number of concurrent disk IO operations, or when you change the SCSI node ID of a virtual disk.  These may all trigger a re-discovery of the disks by the Windows Virtual Disk Service, leaving data disks offline on Windows 2008 Enterprise and Datacenter Edition guests.

In my opinion, these issues are not reasons to forgo upgrading your virtual hardware version.  However, when your upgrade/migration plans call for upgrading the virtual hardware version of your guests you should be prepared to resolve any issues caused by ‘ghost hardware’, offline disks, and the like.  Both the MSDN and VMware articles I cited above offer workarounds for the offline disk issue.  Here are the links again:

Related posts:

1. Microsoft Offline Virtual Machine Servicing Tool v2 Released The Microsoft Offline Virtual Machine Servicing Tool v2 could be...
2. vSphere Upgrade Breaks Active Directory I recently completed a VMware VI 3.5 to vSphere upgrade...
3. EMC Virtual Storage Integrator Update I upgraded my in-house VMware vSphere environment to 4.0 Update...

To reduce the size of your output, you can change your sampling delay to something larger, say 30-seconds.  I suppose you could also capture statistics when the host is not busy so you get fewer characters in the results, but that’s just being goofy.

A better way to reduce your ESXTOP output size is to selectively include only the statistics you are interested in, and is really what Mark was asking.  After all, all statistics from ESXTOP can be too many statistics, and chances are you already know what stats you are interested in.  Here’s how you can narrow down the collected stats for easier analysis and smaller output:

1. Enter ESXTOP in interactive mode on the Service Console by simply typing esxtop at the # prompt
2. Switch to a component you are NOT interested in capturing statistics on by pressing the corresponding menu option (c: ESX cpu, m: ESX memory, d: ESX disk adapter, u: ESX disk device, v: ESX disk VM).
3. Press f when viewing the component you do not want to capture.  A list of fields will be displayed.  You can toggle the fields on and off by pressing the letter corresponding to each field.  An * indicates that the field is on.  You want to turn off all of the fields you don’t want to collect.
4. Repeat steps 2 & 3 for the remaining components, leaving only what you want to capture.
5. Switch to the component you want to capture in batch mode and repeat step #3, except you will now enable what you want to capture.
6. Press W (capital W – case sensitive) to write out the ESXTOP configuration file.  You can accept the default or create new configuration files.  You may want to create a CPU-only config file, memory-only, and so forth.
7. Press CTRL+C to stop ESXTOP.
8. Now, invoke ESXTOP in batch mode, calling your updated or new configuration file you created in step #6 using the -c switch.  Here’s an example:# esxtop -b -d 30 -n 480 -c .esxtopcpustats > /tmp/esxtop_cpu_stats.csv where .esxtopcpustats is an ESXTOP config file with only CPU stats.  -d sets your capture interval to 30 seconds, and -n sets the number of samples to 480 (or 4 hours with a delay of 30 seconds).

Once your capture is complete you can replay the sampling in ESXTOP using replay mode (-R), or you can copy the .csv to a Windows system and use PERFMON or Excel to analyze the stats.  If using PERFMON or Excel you will notice that the system summary information displayed at the top of an interactive ESXTOP session is included in the output (console memory, console cpu, etc.).  As far as I know, there is no way to disable this, nor would you want to as it includes the time stamp necessary to interpret your data.

It is possible to use the vSphere CLI or the vSphere Management Assistant (vMA) to run RESXTOP, a version of ESXTOP designed for remote administration of ESXi or ESX.  You may note, however, RESXTOP from the vSphere CLI only works from a Linux client.  Using either of these tools will help you to automate ESXTOP statistics collection from multiple hosts using customized configuration files.

Related posts:

1. ESXTOP Batch Mode & Windows Perfmon I needed to grab some stats from my ESX hosts...
2. Balloon Driver Problems with SQL I have been meaning to write this up for a...
3. vCenter Database Stats Rollup Troubleshooting VMware vCenter collects performance statistics, tasks and events for historical...

I recently migrated an environment from vCenter 2.5 to 4.0 and in the process switched from a Windows 2003 32-bit vCenter host and a SQL 2005 server (remote to vCenter) to a Windows 2008 64-bit vCenter server with a SQL 2008 server (again, a remote SQL server).  I experienced a few issues during the migration and thought I had worked through them all (I’ll post on those at a later date).  However, after a bit of time I found that performance statistics for objects in the vCenter were missing of not rendering at an acceptable pace.  Upon further investigation, I discovered warnings in the vCenter Service Status node indicating that performance rollups within the vCenter database were not taking place.

In a SQL-backed vCenter, statistics rollups are handled by the SQL Server Agent (note: if you are using SQL Server Express, statistics rollups are handled by vCenter itself as SQL Express does not offer SQL Server Agent jobs).  KB 1003570 describes this process (it applies to vCenter 2.5, but the principles in it can be applied to 4.0).  To troubleshoot and resolve the issue I opened SQL Server Management Studio and checked several items:

1. Is the SQL Server Agent running?
2. Are there statistics rollup jobs defined for SQL server agent?
3. Are those jobs running?

In my case, the SQL Server Agent was running (you are prompted to configure this during the vCenter install).  However, when I checked for the presence of rollup jobs, I discovered that only a Past Day job had migrated with the database to the new SQL server.  Upon investigating the job history for that job I discovered that the job had not run since the migration (note to self: add these checks to your standard vCenter migration checklist).

To remediate the problem I completed the following steps:

1. Remove the bad ‘Past Day stats rollupVirtualCenter’ job from the list of SQL Server Agent Jobs.
2. Recreate the three standard stats rollup jobs.  To recreate the jobs, find SQL scripts on your vCenter server in C:\Program Files (x86)\VMware\Infrastructure\VirtualCenter Server.  The .sql scripts you’ll need are stats_rollup1_proc_mssql.sql, stats_rollup2_proc_mssql.sql, and stats_rollup3_proc_mssql.sql.  Run these scripts in SQL Query Analyzer against your VirtualCenter Database in order from 1 to 3.  These scripts should create the rollup jobs and their associated stored procedures (this procedure is detailed at http://communities.vmware.com/thread/123715).
3. After recreating the jobs I took a backup of the vCenter database.  The Past Day job soon kicked off to begin a stats rollup (this runs every 30 minutes by default).

I checked the server several hours later and discovered that rather than completing successfully, the Past Day job was still running and the drive holding my vCenter database transaction log was full.  Back to the drawing board..

1. I disabled the Past Week and Past Month rollup jobs to avoid job conflicts.
2. I backed up the vCenter database and then performed a shrink of the log file to get it back down to size.
3. The vCenter was running as a VM, so I was able to quickly increase its disk size and use diskpart from within the guest to extend the partition.  The space required to process weeks of performance statistics is not included in the vCenter Database Sizing tool as it is assumed that the rollup/purge jobs will run as designed.

I wanted to see how bad the problem was before kicking off another job so I ran:

select count(*) from vpx_hist_stat1

against the vCenter database in SQL Query Analyzer.  The query ran for several hours (never a good sign) and eventually returned well over 20 million rows of performance statistics (thanks to http://communities.vmware.com/message/1318736 for pointing me in this direction).  I investigated options to truncate the tables (see above link), and also looked at a script from VMware KB 1000125: Purging old data from the database used by vCenter Server.  In the end, I decided to try to let the Past Day stats job run.

I stopped the vCenter  Server Service to prevent new statistics from being written to the database.  I also disabled the Past Week and Past Month SQL Agent jobs to prevent job conflicts and then manually started the Past Day job.  I had to stop the job several times as it filled the 100GB transaction log volume.  A backup & shrink operation gave me back the space on the log volume.  I saw about 300GB of transaction logs written over the course of this process, but the Past Day job eventually completed.

Finally, I re-enabled the Past Week and Past Month jobs and manually ran both of them (Past Week first, then Past Month), followed by a backup and shrink of the vCenter database.  I was impressed with the performance increase I saw in the vCenter client.  Lists and performance graphs rendered much faster than when stats rollups were not taking place.

It would be a good idea to include checking stats rollup job status and a count of rows from the vpx_hist_stat tables in the vCenter database in your regular maintenance tasks.  For other vCenter Database best practices, check out breakout session PO2061 from VMworld 2008.  If you did not attend or subscribe to VMworld, Scott Lowe covered the session in this post.  A VMworld 2009 “online only” session entitled VM3237 vCenter Databases: Setup, Management and Best Practices was also offered (subscription required).  I have not viewed this session so I cannot comment on its content.

Related posts:

1. The Skinny on ESXTOP A reader named Mark contacted me today and asked if...
2. vCenter Client Passthrough Authentication I started this blog for a couple reasons: 1.) To...
3. Installing PowerPath/VE using VMware Update Manager I am finishing up an installation of an EMC Clariion...

As a side note, Microsoft has published the Microsoft Manual of Style for Technical Publications to help standardize technical documentation.  I have long been a fan of Microsoft’s technical documentation for its easy to read style, although it sometimes lacks the depth that I desire.

While we’re on the topic of virtualization documentation, I have also been quite pleased with VMware’s technical documentation over the years, and have found it to be continually increasing in quality, providing very specific technical guidance and references to additional resources.  I have also been pleased to see that VMware has improved delivery options for documentation.  VMware offers several formats for documentation delivery, including web-based and PDF’s.  Start with the Documentation Roadmap for a quick introduction to the available documentation, and where to find what you need.

You can find web-based vSphere documentation here: http://pubs.vmware.com/vsp40/.   The web-based documentation is great for running searches on.  All vSphere documentation can be accessed through this page: http://www.vmware.com/support/pubs/vs_pages/vsp_pubs_esx40_vc40.html.  If you want to do a full grab of all of VMware’s documentation for an in-house repository (e.g. SharePoint), check out xtravirt’s VMware Documentation Downloader script.

If you are looking for quick and easy evaluator guide-type documentation from VMware, check out these resources: ESXi Installable and vCenter Server Setup Guide and the Virtualization Kit (registration required) at http://www.vmware.com/resources/wp/virtualization101_register.html.

There is a ton of less formal VMware documentation in several places:

• Technical resources and case studies here: http://www.vmware.com/resources/techresources/
• Proven practices around Strategy, Applications, Security, Management, and Availability at VIOPS.
• Official VMware Blogs at http://www.vmware.com/vmtn/planet/vmware/.
• Community blogs aggregated by VMware at Planet v12n: http://www.vmware.com/vmtn/planet/v12n/
• VMworld Recorded Sessions & Labs (VMworld 2009 Sessions available as of today, September 14th) at http://vmworld.com.
• The VMware Community Forums: http://communities.vmware.com/
• And, 3rd party books like Scott Lowe’s Mastering VMware vSphere 4.

Do you have other sources of virtualization documentation or easy methods of searching documentation to find exactly what you need when you need it?  If so, leave a comment!

Related posts:

1. NetApp Extends 50% Virtualization Guarantee NetApp has extended their 50% Virtualization Guarantee to include Citrix...
2. Microsoft Offline Virtual Machine Servicing Tool v2 Released The Microsoft Offline Virtual Machine Servicing Tool v2 could be...
3. Virtualization Bookmarks for August 28th Here are some bookmarks for resources that I have recently...

I wanted detailed information, so I decided on a 15 second capture interval to run for a 2 hour window.  Here’s the command I used:

esxtop -a -b -d 15 -n 480 > /tmp/esxtopout.csv

where -a is for ALL, -b is for batch mode, -d is for delay, and -n is for the number of iterations ((60/15)*60*2).  I wrote out the results to a .csv in /tmp.  The resulting CSV weighed in at a whopping 100MB after 2 hours.

The CSV can be analyzed in Excel (pivot tables work well for this) or in Windows Perfmon.  I opened the log in Perfmon as I was after basic Min/Average/Max counters and Perfmon makes those easy to see.  When adding the CSV log to Perfmon, you are prompted to select counters.  I added all instances of Commands/sec, Reads/sec, and Writes/sec from Physical Disk (I was gathering some IOPS counts for a new storage proposal). I got a bit more than I bargained for: a mostly unresponsive Perfmon window and the ugliest darn graph I’ve ever seen.

Switching from a graph view to the report view allows you to easily view and remove specific counters that you are not interested in, or open the Properties of the data set, switch to the data tab and bulk select counters that you want to remove.  I was not interested in vmhba1:x, specific VM’s or worlds, so I killed all of those, leaving just the base iSCSI device (vmhba32 in my case).

After some cleanup the graph looked a bit better and more importantly, I was able to easily read my Min/Average/Max stats:

Here are the takeaways -

• ESXTOP is a powerful utility for performance monitoring
• All stats (-a) can result in a huge file – use it wisely in batch mode; else use interactive mode to select your counters and write them to the user-defined configuration file.  Invoke the config file with the -c option when running in batch mode.
• Consider using vscsiStats for more granular reporting.
• ESXTOP physical disk stats do not include NFS volumes.

Do you use other tools or methods to collect basic disk IO counters for storage sizing purposes?  If so, leave a comment describing your approach!

Related posts:

1. The Skinny on ESXTOP A reader named Mark contacted me today and asked if...
2. Update: SVGA Drivers on Windows 2008 R2 and Windows 7 I posted an article in December on how the SVGA...
3. Storage Basics – Part II: IOPS In Part I of this series, I discussed the important...

My experiences causes me to agree with Scott and the whitepaper he cites – in a properly designed and equipped environment the balloon driver is not detrimental for most every workload to a point.   However, I recently discovered in a client site that the balloon driver can cause significant issues when the environment is poorly designed and under-sized.  Here the background:

I was called into an already established environment where the client was running on an older blade with VMware ESX 3.5.  The blade maxed out at 16GB RAM and had dual dual-core CPU’s with no hope for an upgrade.  On the blade was a single guest VM running Windows 2003 with SQL 2005, in it’s full 32-bit glory.  The VM was configured with 4 vCPU’s and 16GB of memory.  Some of you can probably already guess where this is going….

The x86 Windows guest had PAE configured, and SQL took advantage of AWE to use the additional memory beyond the 4GB limit of a 32-bit system.  Additionally, the Windows guest had the /3GB switch enabled in boot.ini.  Finally, as per SQL best practices, the ‘Lock Pages in Memory‘ permission was granted to the SQL Server service account.  What the guest was left with was 1GB of kernel mode memory and 15GB of User Mode/Extended addressable memory.

And here’s the problem.  The client was using ESX, not ESX 3.5, so the Service Console required memory.  In this case, the service console had approximately 512MB allocated to it.  Futhermore, VM’s require some overhead on ESX to run.  The memory overhead consumed by a Windows guest on ESX 3.5 with 4 vCPU and 16GB of memory is a bit more than 512MB.  On a properly sized ESX server with multiple similar guests/workloads, you could probably gain much of the overhead back through transparent page sharing; but in this case I had a 1:1 P2V ratio.  If you are any good at math you see that the environment is running about 1GB short of memory.  A quick check of the balloon driver stat in vCenter show that the balloon driver was constantly active and demanding about 1GB back from the guest… constantly.

Under normal circumstances this might not be an issue, but in this case the Windows guest was being absolutely punished.  The guest CPU’s were pegged at 100% with an excessive amount of kernel time, often indicating IO issues.  And indeed I did experience terrible disk and network performance on the guest.  At the root of the problem is this – the Lock Pages in Memory permission allows SQL to get a firm grasp on the user mode memory available to it (15GB) and lock it up.  This left the already starved (because of the 3GB switch in the boot.ini) guest kernel with it’s 1GB the only thing the balloon driver could really swap out.

The client suggested a reservation of 16GB on the VM, knowing that memory reservations prevent balloon driver activity.  I calmly asked them to back away from the keyboard as I explained how if a starved guest was bad, how much worse a starved Service Console would be.  In the end the fix was quiet easy – I convinced the customer that they should reduce the amount of memory allocated to the guest by about 1GB, enough to let the 512MB SC and the 512MB of overhead run without contention.  I was able to show them the difference between allocated and active memory in vCenter – the 1GB being surrendered was not really being actively used, SQL just had it locked up.  In fact, surrendering the 1GB of memory back to ESX breathed new life into the guest VM, bringing its performance back in line with expectations.

Ideally, I would have brought in a bigger ESX server that could serve additional VM’s, driving greater levels of efficiency across the environment.  It just wasn’t an option for the client in this case.  In the end, the problem was fixed and I was reminded just how fun it can be to explain some of these backwards sounding virtualization concepts to customers – fewer vCPU’s can lead to better performance of guests, less guest memory can fix performance issues, and increasing the quantity of similar guests on a host can drive better performance to a point because of transparent page sharing.

Stay tuned over the next few weeks as I digest and write on my VMworld experience – from VMUG activities to Paul Maritz’s press conference announcing the vCloud Express, and plenty of great sessions in between.  Like many of you, I returned from VMworld with quite a backlog of work but I’ll do my best to squeeze in some posts and tweets.

Related posts:

1. RAMDisk Usage in a vSphere Environment I had some folks from our .NET development team come...
2. Windows Server 2008 R2 & Windows 7 Freeze When Using SVGA Drivers I recently ran into an issue when installing my first...
3. Update: SVGA Drivers on Windows 2008 R2 and Windows 7 I posted an article in December on how the SVGA...

The system was configured with a single controller with dual GigE NIC’s.  The controller had 512MB of battery backed cache (there is also a 1GB cache upgrade option available).  I wrote off some of the poor performance to a single controller with a less-than-optimal amount of cache; blamed the SAS controller to SATA disk command translation overhead; cringed at the 6 disk RAID5 configuration; and engaged in some self doubting.  I convinced the powers that be that we were IO constrained and got some funds to fill out the 3U chassis to a full 12 SATA disks, and reconfigured the array as a RAID10.  Performance gains were almost unnoticeable with these changes.  In addition, I did some basic troubleshooting of the network environment, verifying multiple paths to the storage, setting Flow Control on the switches to receive only, and double-checked my iSCSI initiator settings.  Note: The DS3300 is only supported with the ESX software initiator.  I found documentation on the DS3300 to be lacking, but did discover that the Dell MD3000i is based on the same LSI Engenio array.  Some Googling on the Dell solution led to to the ‘SMcli’ command line interface for both arrays.   The commands are slighly different for the Dell and IBM.  The links to the IBM CLI documentation were broken, so I had to do a bit of trial and error to get the commands right.  I used the Dell documentation as a starting point.  (Rant: Seriously, IBM?  Can you make your documentation any harder to get through – is it a Redbook, is it an Engineering Whitepaper, is it a support document, is it a case study – and why can I only find these with complex Google searches, not on your own product pages, and why can’t you name for documents intelligently, not with some random string of characters).

Moving on… I received an automated alert from the DS3300 about an incomplete battery learn cycle.  Using the IBM Storage Manager GUI I generated a  Storage Subsystem Profile’ from the Support tab to check the battery status.  In the profile I discovered that while write cache was enabled, it had a status of “Enabled (Suspended)”.   Ah ha!  Now I’ve got some decent Google material that led me to this: http://communities.vmware.com/thread/195838.  Hot damn I love the VMware Community Forums!

It turns out that in a single-controller configuration the setting for cache mirroring remains enabled by default.  Because there is no 2nd controller to mirror to, the array suspends write caching.  This is probably a safety thing – loss of high availability on the controllers puts data in cache at risk should the only controller fail.  I weighed my options and decided that the poor performance I was experiencing beat HA concerns, so I enabled write cache on the array using this command:

c:\program files\ibm_ds4000\client>smcli -n <ARRAYNAME> -c “set allLogicalDrives mirrorEnabled=false;”

And then followed with this for good measure:

c:\program files\ibm_ds4000\client>smcli -n <ARRAYNAME> -p <arraypassword> -c “set allLogicalDrives writeCacheEnabled=true;”

The results were immediately noticeable:

DS3300 Performance Improvement when Write Cache is Enabled - Click for a Larger View

The screen shot is from Veeam Monitor Free Edition, taken during 4 concurrent V2V operations from Hyper-V to VMware.  With the write cache fully functional, disk usage peaked at 54MBps, latency dropped to about 6ms, and my blood pressure dropped a few notches.

While poking around the CLI I also found that you can dump performance stats from the array (performance is otherwise hard to find on the thing) using this command:

C:\Program Files\IBM_DS4000\client>smcli -n <ARRAYNAME> -c “set session performanceMonitorInterval=5 performanceMonitorIterations=120;save storageSubsystem performanceStats file=\”c:\\ds3300perfstats.csv\“;”

This will give you a 10 minute record of performance from the array which you can analyze using Excel.  The Dell Enterprise Center TechCenter Wiki has a great write-up on how to efficiently analyze the data from this command here: http://www.delltechcenter.com/page/MD3000i+Performance+Monitoring, complete with a YouTube video that walks you through the process:

I am beginning to think that the DS3300 (and MD3000i) may actually be a viable starter solution for SMB’s starting out on a virtualization project.  But I would recommend the cache upgrade, 2nd controller, SAS disks instead of SATA to eliminate the SAS-to-SATA translation overhead and more faster disks instead of fewer slower disks so you can drive throughput and IOPS to a higher level.

Have any of you deployed the DS3300 or MD3000i (or the generic LSI solution)?  Do you have any performance tuning tips for these arrays?  If so, share in the comments!

Related posts:

1. Free SAN Monitor for DS3300, MD3000i and others One of my most popular posts to date had been...
2. Storage Basics – Part V: Controllers, Cache and Coalescing My Storage Basics series has been neglected for some time...
3. Storage Basics – Part IV: Interface In parts I, II, and III of the Storage Basics...

After performing maintenance on an ESX server (patches, storage re-scan, reboot), VMFS volumes are no longer visible, even though the hosting LUN can be seen on the Storage Adapters page of the ESX Configuration tab.  Most VMware administrators will see this play out at some point; I saw it in one of my environments today and figured I should make a note of the steps required to correct the issue.

Typically, the root cause of the issue is a change on the storage array that causes the h(id) of the LUN(s) in question to change.  This change could be anything from an array firmware update, LUN removal/recreation, or RAID/LUN reconfiguration.  These changes could cause the h(id) of the LUN to be updated.  When a rescan takes place on the ESX storage adapters (through a manual instantiation, reboot, etc.), the new h(id) is observed.  Because it does not match the previously observed ID, the LUN is tagged as a snapshot LUN and access to that LUN is disabled.

Diagnosis of this problem is fairly easy.  In addition to the behavior I have described, as observed through the Virtual Center Client, the problem can also be confirmed through the ESX command line.

To diagnosis this issue from the console, view the vmkernel log by issuing the following command: tail -f /var/log/vmkernel

You will see messages in the log similar to the following:

Jun  2 16:01:29 esx04 vmkernel: 0:00:31:14.543 cpu3:1039)ALERT: LVM: 4482: vml.0200020000600a0b80005add7800000a494a1d0be6313732362d33:1 may be snapshot: disabling access. See resignaturing section in SAN config guide.
Jun  2 16:01:29 esx04 vmkernel: 0:00:31:14.552 cpu3:1039)LVM: 5579: Device vml.0200010000600a0b80005add7800000a474a1d0bc8313732362d33:1 detected to be a snapshot:
Jun  2 16:01:29 tccesx04 vmkernel: 0:00:31:14.552 cpu3:1039)LVM: 5586:   queried disk ID: <type 2, len 22, lun 1, devType 0, scsi 6, h(id) 5103533129706062046>
Jun  2 16:01:29 esx04 vmkernel: 0:00:31:14.552 cpu3:1039)LVM: 5593:   on-disk disk ID: <type 2, len 22, lun 1, devType 0, scsi 6, h(id) 2153359415130143165>

After confirming that this is indeed the problem you are experiencing, stop and take a deep breath.  The fix is easy, but you need to take steps before fixing it to prevent further damage.  If you are lucky, the problem has only manifested itself on one ESX server (and hopefully that ESX was not hosting any VM’s because you put it into maintenance mode).  Prevent your other ESX servers from rescanning storage – don’t reboot them, don’t manually rescan, don’t update them.

If the affected ESX server was hosting running VM’s, HA (if licensed and properly configured) should have kicked in if applicable and restarted the VM’s on another node in the ESX cluster.

If multiple ESX servers (or all of them) are affected, your VM’s are likely all powered off after hard stops, so there is not much you can do but to get on with fixing the issue and trust your backups (you do have backups, right?).  This is where array-level snapshots come in handy.  In my experience, most if not all VM’s recover after a hard stop like this, but don’t let that keep you from having a robust DR plan.

To correct the issue you must not have any running VM’s on the affected VMFS volumes to alternate volumes.  Shut down the VM’s or use Storage VMotion to move running VM’s to alternate LUN’s.

In the VI Client, select the affected ESX host in the Hosts & Clusters view.  Switch to the Configuration Tab.  Click ‘Advanced Settings’ and then choose the LVM node.  Change the LVM.DisallowSnapshotLun from the default setting of ’1′ to ’0′ and click OK.  Next, rescan your storage from the ‘Configuration | Storage Adapters’ pane.  Your missing VMFS volumes should re-appear.  You’re doing fine, but not done yet.

Even if the other hosts that use the affected VMFS volume appear to be fine, they will most likely lose access to the volume once a rescan/reboot takes place.  You need to perform the LVM.DisallowSnapshotLun = 0 setting change on all ESX servers connected to the volume, followed by a re-scan of your storage.

Once all affected ESX servers see the VMFS volumes, change the LVM.DisallowSnapshotLun setting back to the default of 1.  Migrate back and/or power up VM’s on the volume and see what the damage is.  If you are lucky, everything is good to go.  If not, it’s a great time to check out those backups.

If you do not know what caused the storage change, check your ESX logs to try to determine if the server was rebooted or if storage was rescanned. This will give you an idea of when the change occurred – a starting point to work back from to find the root cause.  Use this command to get started: less /var/log/vmksummary

Here are some suggestions on how to avoid this problem:

1.) Minimize changes to LUN’s once configured on an ESX.

2.) Coordinate Storage Maintenance with VMware maintenance windows.

3.) Have stand-by storage so you can Storage VMotion running VM’s off of the affected LUNS.

4.) Consider NFS, as NFS volumes are not impacted by resignaturing.

For more information on this problem, or to better understand the advanced settings changes involved, check out the VMware SAN Configuration Guide at http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_3_server_config.pdf, page 114, or the VMware iSCSI SAN Configuration Guide at http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_iscsi_san_cfg.pdf, page 117.

Related posts:

1. Windows Server 2008 R2 & Windows 7 Freeze When Using SVGA Drivers I recently ran into an issue when installing my first...
2. Upgrading Virtual Hardware in a VMware Virtual Machine May Cause Disks to go Offline I recently posted an article on how specific actions during...
3. The Skinny on ESXTOP A reader named Mark contacted me today and asked if...

I just built up a new laptop and couldn’t remember how to set up pass-through authentication on my VI3 Client. A quick Google search gave me the answer, courtesy of Stu Radnidge‘s post on nothing other than VirtualCenter 2.5 Passthrough Authentication.  This little gem saves the terribly tedius work of having to manually enter your login credentials each time you launch the Virtual Infrastructure 3 Client by passing through your currently logged-in credentials to the VC server.  Thanks for the tip, Stu!

Related posts:

1. Virtual Infrastructure Client Opens Off Screen A user reported an issue with one of the VM’s...
2. vCenter Database Stats Rollup Troubleshooting VMware vCenter collects performance statistics, tasks and events for historical...
3. OT: Joining a Mac to a Windows Active Directory We picked up a few new 17″ MacBook Pro’s at...

Great article, Ken!  I look forward to reading the rest of the series.

Related posts:

1. Networking Problems with ESX 3.5 Update 3 on the DL380 G3 I began building up a new lab environment with a...
2. Landing a Virtualization Job As the New Year rolls around you may find yourself...
3. DL380 BIOS Configuration for VMware One more post to wrap up the nonsense with my...

There are plenty of examples on the Internet to get you started with the VI Toolkit for Windows. Check out these sites to get started:

Start at the VMware Community site for the Windows Toolkit for great examples and a little help from some friends: http://communities.vmware.com/community/developer/windows_toolkit/
There are also some good example scripts and resources floating around, such as:

http://vmetc.com/2008/08/27/powershell-scripting-examples-for-vmware-virtual-infrastructure/
http://www.peetersonline.nl/
http://www.ivobeerens.nl/?p=106
http://www.vmguru.com/

Not a hard-core scripter? Grab this handy tool for a little GUI on your PowerShell, and extend it with the VMware Infrastructure PowerPack 2.0

What tools or examples are you using to extend the power of PowerShell into your Virtual Infrasturucture?  Leave a comment to share!

Related posts:

1. Microsoft Offline Virtual Machine Servicing Tool v2 Released The Microsoft Offline Virtual Machine Servicing Tool v2 could be...
2. Windows Server 2008 R2 & Windows 7 Freeze When Using SVGA Drivers I recently ran into an issue when installing my first...
3. DL380 BIOS Configuration for VMware One more post to wrap up the nonsense with my...

We certainly all have our own opinions and operations procedures when it comes to configuring and hardening our environments, but I decided to take a look at what the experts had to say on this particular subject and other basic build and hardening recommendations.  Here is what I found:

VMware Security Resources

VMware Security Utilities

VI3.5 Security Hardening Whitepaper

Defense Informaion Systems Agency (DISA) ESX Server Security Technical Implementation Guide

DISA ESX Server Checklist

As a side note, DISA publishes many STIG’s at http://iase.disa.mil/stigs/.  Your tax dollars paid for these, so you might as well check them out.

NSA VMware ESX  Server 3 Configuration Guide

There are also numerous tips and scripts for locking down your virtual infrastructure in the VMware Community Forums (Start here: http://communities.vmware.com/message/941372).

So back to the question of second super user accounts: It seems that best practices are to create a second user account with sufficient access to the console, granting that user SUDO privledges, and then disabling the default root account.

Related posts:

1. Virtual Infrastructure Client Opens Off Screen A user reported an issue with one of the VM’s...
2. Upgrading Virtual Hardware in a VMware Virtual Machine May Cause Disks to go Offline I recently posted an article on how specific actions during...
3. Microsoft Offline Virtual Machine Servicing Tool v2 Released The Microsoft Offline Virtual Machine Servicing Tool v2 could be...

When it comes to reporting there are a ton of options available.  The solution you choose will be dependent on your environment and the tools you already have in place.  Small and Medium-sized Businesses (SMB’s) often do not have full blown, network-wide monitoring and management solutions, so VMware-specific solutions are often a great fit.  There are several examples beyond my simple script, and many are free.  The short list includes: Mightycare Solutions MCS StorageView 1.1, Rich Garsthagen’s VCplus, and Rob de Veij’s RVTools.

There are many other mid-tier solutions – both enterprise-wide and VMware specific – constantly emerging as the virtualization ecosystem matures.  Offerings from ManageIQ, Embotics, Veeam, V-Kernel, Zenoss, Hyperic, and others are increasingly able to provide fresh and relevant data on what is happening under the covers in your virtual environment.

Larger IT shops most likely have a systems monitoring solution easily capable of reporting this – think offerings from the likes of Microsoft, Altiris,  BMC, or CA.  The trick in these solutions is narrowing down the information to your virtualized resources and getting the information to the right teams.  Customized reports using fields such as the BIOS Vendor string can help show only servers running VMware, for example.  As a side note, the Vendor BIOS string can also come in handy when applying Group Policies (GPO), allowing you to filter policies for only virtualized resources (disabling screen savers on Windows guests through GPO is a good example of this).

And don’t forget, we’re not reporting for reporting sake.  We’re after relevant information that allows us to be more efficient and proactive in the overall goals of our environments.  Good reporting identifiies areas in need of improvement, and smart system administrators look for creative ways to improve their systems efficiency.

Related posts:

1. Obtaining VMware Guest Disk Free Space for NFS Sizing I am often asked about sizing storage vis-à-vis how much...
2. VI Toolkit for Windows v1.5 Released Today VMware released version 1.5 of the VI Toolkit for Windows...
3. NetApp Extends 50% Virtualization Guarantee NetApp has extended their 50% Virtualization Guarantee to include Citrix...

There are several methods of obtaining the amount of free space in a guest OS, from third party systems management tools (some are vendor or OS specific) to custom scripting (some VB for your Windows hosts, etc.).  VirtualCenter also knows how much free space is within each guest VMDK, but the information is not readily displayed.

The first method of getting guest free space is using the VMware VI Toolkit for Windows.  A simple statement like what I show below will pull the info for you (see Hal Rottenberg’s post on the VMware Communities Forum):

PS > $hdCapacity = @{ N = “Capacity (bytes)”; E = { $_.Guest.Disks | % { $_.Capacity } } }
PS > $hdFreeSpace = @{ N = “FreeSpace (bytes)”; E = { $_.Guest.Disks | % { $_.FreeSpace } } }
PS > Get-VM | select Name, $hdCapacity, $hdFreeSpace

You can re-write the command into a single line and change the output to show Percentage Free space, for example.  The following came from http://communities.vmware.com/message/1046360.

Get-VM | Where { $_.PowerState -eq “PoweredOn” } | Get-VMGuest | Select VmName -ExpandProperty Disks | Select VmName, Path, @{ N=”PercFree”; E={ [math]::Round( (100 * ( $_.FreeSpace / $_.Capacity ) ),0 ) } } | Sort PercFree

These examples are well and good, but there are a couple of catches – The guest must be powered up and VMware Tools must be installed and running inside each VM you want to pull statistics from.
Many people do not know that the guest disk capacity and free space statistics are also captured in the VirtualCenter database and is available for VM’s in any power state (On, Off, or Suspended), so long as VMware Tools has been installed and run at least once in the VM.  A SQL simple query will return the data too you (this could be more simple – I was pulling some additional statistics for a project I am working on):

SELECT VPX_GUEST_DISK.VM_ID, VPX_GUEST_DISK.PATH, VPX_GUEST_DISK.CAPACITY, CONVERT(bigint, VPX_GUEST_DISK.CAPACITY) / 1048576 AS ‘Capacity MB’, VPX_GUEST_DISK.FREE_SPACE, CONVERT(bigint, VPX_GUEST_DISK.FREE_SPACE) / 1048576 AS ‘Free Space MB’,VPX_HOST.ID, VPX_HOST.DATACENTER_ID, VPX_HOST.DNS_NAME, VPX_VM.ID, VPX_VM.DATACENTER_ID, VPX_VM.FILE_NAME, VPX_VM.LOCAL_FILE_NAME, VPX_VM.POWER_STATE, VPX_VM.GUEST_OS, VPX_VM.GUEST_STATE, VPX_VM.MEM_SIZE_MB, VPX_VM.NUM_DISK, VPX_VM.DNS_NAME, VPX_VM.IS_TEMPLATE, VPX_VM.HOST_ID
FROM VPX_GUEST_DISK VPX_GUEST_DISK, VPX_HOST VPX_HOST, VPX_VM VPX_VM
WHERE VPX_VM.ID = VPX_GUEST_DISK.VM_ID AND VPX_HOST.ID = VPX_VM.HOST_ID

Of course, you can always combine the power of PowerShell and the raw SQL data to create formatted output.  Frank Hagen displays a method of obtaining SQL data through a PowerShell script and dumping the data to Excel for further manipulation on his blog.  I modified a few lines of Frank’s code for my purposes:

################################################
# QuerySQL.ps1 # Frank W Hagen – 2008/07/15
# # FWHagen.wordpress.com
# Powershell script to query a SQL database # fwhagen.blog@gmail.com
# and write the output to an Excel file
#
# Usage:
# * Create a SQL query file by putting a valid SQL query in a text file in
# the subdirectory specified in $SQLQueryPath named <$TaskName>.sql
# * Set configuration variables in config section below
# * Run at command line: powershell -nologo .\QuerySQL.ps1
# OR right-click this script and Open With -> Powershell.EXE
#
# If you get a security warning running the script, see the following post:
# http://fwhagen.wordpress.com/2007/10/29/running-local-powershell-scripts/
########################################################

# Function used for binding to Excel
function Invoke([object]$m, [string]$method, $parameters)
{ $m.PSBase.GetType().InvokeMember($method, [Reflection.BindingFlags]::InvokeMethod, $null, $m, $parameters, [System.Globalization.CultureInfo]“en-US”) }

############################################
### Configuration information for specific query #####
$TaskName = “VIGuestDiskFree” # Title and name of query file

#######################################################
### UPDATE THE FOLLOWING LINES FOR YOUR ENVIRONMENT ###
$SqlServer = “”; # SQL Server hosting VirtualCenter Database (include \instance name if not using Default)
$SqlCatalog = “”; # Virtual Center Database name
$SQLUserID = “”; # SQL Server User ID if not using Integrated Security see lines 69 and 70
$SQLPassword = “”; # SQL Server User Password if not using Integrated Security see lines 69 and 70
### END OF CUSTOMER UPDATABLE FIELDS ###
#######################################################
$WriteOutXML = $False
$WriteOutCSV = $False
$WriteOutXLS = $True
#
# Environment Configuration
$TaskPath = “C:\Scripts\” # Root Directory for creating reports
$SQLQueryPath = “SQLQueries\” # Subdirectory for finding the queryfile
######################################################

# Timestamp the output folder and files using ISOdate
$OutPath = ($TaskPath + (Get-Date -Format yyyyMMdd) + “-” + $TaskName + “\”)
$OutFileName = ( (Get-Date -Format yyyyMMdd) + “-” + $TaskName )

# Create the output folder #TODO: Fix Call to eliminate verbose results from system
if (!$(test-path ($OutPath)))
{
New-Item -itemType directory -Name ((Get-Date -Format yyyyMMdd) + “-” + $TaskName) > $null

if ($(test-path ($OutPath)))
{
Write-Host ($OutPath + ” Created”) -ForegroundColor “darkgreen”
}
else
{
Write-Host ($OutPath + ” FAILED”) -ForegroundColor “red”
}
}

# Get the T-SQL Query from .SQL file
$SqlQuery = Get-Content ($TaskPath + $SQLQueryPath + $TaskName + “.sql”)
Write-Host (“Executing Queryfile: ” + ($TaskName + “.sql”) + ” “) -ForegroundColor “darkgreen”
#Write-Host ($SqlQuery) -ForegroundColor “gray”

# Setup SQL Connection (using Integrated Security (your workstation login). Use standard connection string format for other)
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
#$SqlConnection.ConnectionString = “Server = $SqlServer; Database = $SqlCatalog; Integrated Security = True”
$SqlConnection.ConnectionString = “Server = $SqlServer; Database = $SqlCatalog; Integrated Security = False; User ID = $SQLUserID; Password = $SQLPassword;”

# Setup SQL Command
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $SqlQuery
$SqlCmd.Connection = $SqlConnection

# Setup .NET SQLAdapter to execute and fill .NET Dataset
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$DataTable = New-Object System.Data.DataTable

# Execute and Get Row Count
$nRecs = $SqlAdapter.Fill($DataSet)

Write-Host ($nRecs.ToString() + ” Records retrieved.”) -ForegroundColor “Blue”
$SqlConnection.Close();

if ($nRecs -gt 0)
{
# Make copy of successful query in output directory for traceability
if ($(test-path ($OutPath + $OutFileName + “.sql”)))
{
del ($OutPath + $OutFileName + “.sql”)
}
Copy-Item ($TaskPath + $SQLQueryPath + $TaskName + “.sql”) -destination ($OutPath + $OutFileName + “.sql”)

# Very simple to export XML
if($WriteOutXML)
{
Write-Host “Creating XML File…” -ForegroundColor “darkgreen”
if ($(test-path ($OutPath + $OutFileName + “.xml”)))
{
del ($OutPath + $OutFileName + “.xml”)
}

$DataSet.Tables[0].WriteXML($OutPath + $OutFileName + “.xml”);
}

# Very simple to export CSV
if($WriteOutCSV)
{
Write-Host “Creating CSV File…” -ForegroundColor “darkgreen”
if ($(test-path ($OutPath + $OutFileName + “.csv”)))
{
del ($OutPath + $OutFileName + “.csv”)
}

$DataSet.Tables[0] | Export-Csv ($OutPath + $OutFileName + “.csv”)
}

# Very hard to export XSL – This method writes the data to an object array and pastes the array directly into Excel (Thanks go to a few sources on the Internet for this method)
if($WriteOutXLS)
{
Write-Host “Creating Excel File…” -ForegroundColor “darkgreen”
if ($(test-path ($OutPath + $OutFileName + “.xls”)))
{
del ($OutPath + $OutFileName + “.xls”)
}

$sheetIndex = 0;
$oExcel = New-Object -COM Excel.Application
$oExcel.Visible = $false
$oBooks = $oExcel.Workbooks
$oCulture= [System.Globalization.CultureInfo]“en-US”
$oBook=$oBooks.psbase.gettype().InvokeMember(“Add”,[Reflection.BindingFlags]::InvokeMethod,$null,$oBooks,$null,$oCulture)
#$oSheet = $oBook.Worksheets.Item(1)

$DataTable = $DataSet.Tables[0];
$nDr = $DataTable.Rows.Count + 1
$nDc = $DataTable.Columns.Count + 1

# Create the object array
$rawData = new-object ‘object[,]‘ $nDr,$nDc

# Write the field names in the first row
for ($col = 0; $col -lt $DataTable.Columns.Count; $col++)
{
$rawData[0, $col] = $DataTable.Columns[$col].ColumnName;
}

# Copy the dataset to the object array
for ($col = 0; $col -lt $DataTable.Columns.Count; $col++)
{
for ($row = 0; $row -lt $DataTable.Rows.Count; $row++)
{
$rawData[($row + 1), $col] = $DataTable.Rows[$row][$col];
}
}

# Calculate the final column letter
$finalColLetter = “”;
$colCharset = “ABCDEFGHIJKLMNOPQRSTUVWXYZ”;
$colCharsetLen = $colCharset.Length;
if ($DataTable.Columns.Count -gt $colCharsetLen)
{
$finalColLetter = $colCharset.Substring((($DataTable.Columns.Count – 1) / ($colCharsetLen – 1)), 1);
}
$finalColLetter += $colCharset.Substring(($DataTable.Columns.Count – 1) % $colCharsetLen, 1);

### Export it all to Excel #####
Write-Host “Writing to Excel…” -ForegroundColor “darkgreen”

# Create a new Sheet
$excelSheet = $oBook.Worksheets.Item(1)

#$excelSheet.name = $DataTable.TableName; #TODO: Be nice to figure out how to make this work (not critical)

# Create the entire range on the worksheet and dump the data into it
$excelRange = “A1:” + $finalColLetter + “” + ($DataTable.Rows.Count + 1)
$excelSheet.Range($excelRange).FormulaLocal = $rawData;

# Mark the first row as BOLD #TODO: Be nice to figure out how to make this work (not critical)
#$excelSheet.Rows[1].Font.Bold = $True;
#$excelSheet.Cells.Item(1,1).Font.Bold = $True;

# Save the Excel file and we’re done
Invoke $oBook SaveAs ($OutPath + $OutFileName + “.xls”) > $null
Invoke $oBook Close 0 >$null
$oExcel.Quit()
}
}

Write-Host (“Complete”)

I dropped the PowerShell code into a file named VIGuestDiskFree.ps1 in C:\Scripts on my Vista laptop, and the SQL query from earlier in this post to c:\scripts\sqlqueries\VIGuestDiskFree.sql.  Simply run the PowerShell script from a PowerShell command line.  The Excel output file  will be created in a date-stamped folder in C:\Scripts.  Once you have this data you can go about the business of formatting the data as an Excel Table and summing the Free Space column to find out your storage savings with thin provisioning and other storage efficiency technologies.  I have attached a .zip (VIGuestFreeScripts) containing the scripts in this post (formatting is a bit off in WordPress).

How do you report on your storage utilization and storage efficiency efforts? Post a comment to share your methods!

Related posts:

1. Guest Free Disk Space Revisited I wrote about a method for determining guest free disk...
2. Free SAN Monitor for DS3300, MD3000i and others One of my most popular posts to date had been...
3. Right-sizing Your Power and Cooling We all know that virtualization allows us to do more...