virtualization

VMtoday’s One Year Anniversary

November 24, 2009 by Josh Townsend 5 Comments

Today marks the one year anniversary of my first post on VMtoday.com, and an exciting year it has been in many ways. First, some stats:

VMtoday.com has been visited more than 10,000 times in the past year. While the number of site visits is far below some of my fellow virtualization bloggers, it is still exciting for me to see that I am making an impact on the community (despite my meager post count). There are some of you who read my posts through Planet V12n and RSS, which is cool by me.
Yesterday was the busiest day for the site. No coincidence that it comes after adding some new content…. I’ll try to be more faithful in publishing regular, relevant content!
My most popular post to date has been: IBM DS3300 iSCSI Write Performance Solved. I’m glad this has been useful for so many, but I hope that you don’t just apply the workarounds I wrote about. I would rather have you build a “bet the business” iSCSI environment by adding that second controller to your MD3000i or DS3300.
My least popular post to date has been: VMworld Here I Come. I will try not to be so boring in the future.
The first link back to my site was from Scott Lowe’s blog. Thanks for the link, Scott. Scott wrote a darn fine book: Mastering VMware vSphere 4. Buy it.
The site theme, like my Twitter page, is very blue. I am working on a new theme in all of the spare time that a busy professional and father of two boys can muster.

Along with this site, I have made a concerted effort engage the virtualization community in several ways:

Twitter keeps me plugged in to the latest news and discussions, and has been a source of help to me (and I would like to think that I have helped some of you as well). Follow me at https://twitter.com/joshuatownsend.
I have stepped up into a leadership role with the Washington DC VMware User Group (VMUG). It has been awesome to meet with and learn from my local colleagues. I will continue to work with the DC VMUG leadership team to deliver exciting and relevant content and activities (and also some more seating space). I welcome your feedback and ideas for ways to improve the VMUG.
VMworld 2009 was a great way to meet many of you while learning some of the hottest new technology on the planet. My wife came along and enjoyed spending time with some of the other vSpouses (or vWidows?).
I took a new job early in the year with a VMware Partner in the DC area. This new job has been both challenging and rewarding, affording me opportunities to more effectively engage customers and spend more time working on virtualization-specific solutions.

I look forward to contributing to the virtualization community as a blogger, VMUG leader, and practitioner. If you want to learn a bit more about me, check out the About page on this site. I welcome your feedback and appreciate your reading my work!

– Josh

Upgrading Virtual Hardware in a VMware Virtual Machine May Cause Disks to go Offline

November 23, 2009 by Josh Townsend 3 Comments

I recently posted an article on how specific actions during the upgrade of a VMware Virtual Machine’s hardware from v4 to v7 can cause problems with certain services, including DNS, DHCP, and WINS. In that case, the problem was related to Microsoft Windows leaving non-present devices with networking configurations and the failure of the VMware Upgrade Helper service to copy WINS settings when updating the NIC. As my fellow blogger and VMUG leader, Jason Boche, responded on Twitter: “Same gotchas, different version.” And right he is – anyone with experience in P2V or V2V, or who has been working with VMware long enough to have done a 2.5 to 3.0 upgrade experienced the same gotchas.

There are other issues with VMware virtual hardware upgrades, however, that you may not have experienced. One such issue that I have experienced is highlighted in VMware Knowledge Base article 1013109: “Upgrading virtual hardware in ESX 4 may cause Windows 2008 disks to go offline“. The problems described in the article are unique to Windows 2008 Enterprise and Datacenter editions only. The problem is pretty well described in the title of the article – Upgrading virtual hardware in ESX 4 may cause Windows 2008 disks to go offline. In this case, like with the ghost NIC’s I described last week, is more of a Microsoft issue, but it will rear its head when a VMware Administrator least desires it. With this particular problem, the Windows Virtual Disk Service (part of the native Storage Management suite) is set to not auto-mount newly discovered disks that do reside on a shared bus. Microsoft has a MSDN article on the VDS SANS policy here. Upgrading the virtual hardware version causes the disks to be re-discovered and not auto-mounted. This can potentially impact all non-system disks on a VM.

You may also experience similar issues when upgrading the vSCSI adapter in a VM from a standard LSI Logic Parallel SCSI adapter to a (new in vSphere 4.0) paravirtualized SCSI (pvSCSI) adapter, move virtual disks to new vSCSI adapters to increase the number of concurrent disk IO operations, or when you change the SCSI node ID of a virtual disk. These may all trigger a re-discovery of the disks by the Windows Virtual Disk Service, leaving data disks offline on Windows 2008 Enterprise and Datacenter Edition guests.

In my opinion, these issues are not reasons to forgo upgrading your virtual hardware version. However, when your upgrade/migration plans call for upgrading the virtual hardware version of your guests you should be prepared to resolve any issues caused by ‘ghost hardware’, offline disks, and the like. Both the MSDN and VMware articles I cited above offer workarounds for the offline disk issue. Here are the links again:

IBM DS3300 iSCSI Write Performance Solved

June 16, 2009 by Josh Townsend 67 Comments

I have been pulling my hair out with a small VI3 implementation running against an IBM DS3300 iSCSI array. Performance, for lack of a better term, sucked. Granted, the DS3300 is not an enterprise level workhorse of a storage system, but it fit the budget. Read performance was decent from the array, but write performance was terrible, maxing out at 10Mpbs throughput and insanely high latencies on long writes when the system was under load. This led to some long P2V operations, poor guest performance, and some questions from the project sponsors on why I couldn’t make the environment sing.

The system was configured with a single controller with dual GigE NIC’s. The controller had 512MB of battery backed cache (there is also a 1GB cache upgrade option available). I wrote off some of the poor performance to a single controller with a less-than-optimal amount of cache; blamed the SAS controller to SATA disk command translation overhead; cringed at the 6 disk RAID5 configuration; and engaged in some self doubting. I convinced the powers that be that we were IO constrained and got some funds to fill out the 3U chassis to a full 12 SATA disks, and reconfigured the array as a RAID10. Performance gains were almost unnoticeable with these changes. In addition, I did some basic troubleshooting of the network environment, verifying multiple paths to the storage, setting Flow Control on the switches to receive only, and double-checked my iSCSI initiator settings. Note: The DS3300 is only supported with the ESX software initiator. I found documentation on the DS3300 to be lacking, but did discover that the Dell MD3000i is based on the same LSI Engenio array. Some Googling on the Dell solution led to to the ‘SMcli’ command line interface for both arrays. The commands are slighly different for the Dell and IBM. The links to the IBM CLI documentation were broken, so I had to do a bit of trial and error to get the commands right. I used the Dell documentation as a starting point. (Rant: Seriously, IBM? Can you make your documentation any harder to get through – is it a Redbook, is it an Engineering Whitepaper, is it a support document, is it a case study – and why can I only find these with complex Google searches, not on your own product pages, and why can’t you name for documents intelligently, not with some random string of characters).

Update: The IBM System Storage DS3000, DS4000, and DS5000Command Line Interface and Script Commands Programming Guide is here: IBM System Storage DS3000, DS4000, and DS5000Command Line Interface and Script Commands Programming Guide – DS3k4k5kCLIreference, SMCLI

Moving on… I received an automated alert from the DS3300 about an incomplete battery learn cycle. Using the IBM Storage Manager GUI I generated a Storage Subsystem Profile’ from the Support tab to check the battery status. In the profile I discovered that while write cache was enabled, it had a status of “Enabled (Suspended)”. Ah ha! Now I’ve got some decent Google material that led me to this: https://communities.vmware.com/thread/195838. Hot damn I love the VMware Community Forums!

It turns out that in a single-controller configuration the setting for cache mirroring remains enabled by default. Because there is no 2nd controller to mirror to, the array suspends write caching. This is probably a safety thing – loss of high availability on the controllers puts data in cache at risk should the only controller fail. I weighed my options and decided that the poor performance I was experiencing beat HA concerns, so I enabled write cache on the array using this command:

c:program filesibm_ds4000client>smcli -n <ARRAYNAME> -c “set allLogicalDrives mirrorEnabled=false;”

And then followed with this for good measure:

c:program filesibm_ds4000client>smcli -n <ARRAYNAME> -p <arraypassword> -c “set allLogicalDrives writeCacheEnabled=true;”

The results were immediately noticeable:

The screen shot is from Veeam Monitor Free Edition, taken during 4 concurrent V2V operations from Hyper-V to VMware. With the write cache fully functional, disk usage peaked at 54MBps, latency dropped to about 6ms, and my blood pressure dropped a few notches.

While poking around the CLI I also found that you can dump performance stats from the array (performance is otherwise hard to find on the thing) using this command:

C:Program FilesIBM_DS4000client>smcli -n <ARRAYNAME> -c “set session performanceMonitorInterval=5 performanceMonitorIterations=120;save storageSubsystem performanceStats file=”c:ds3300perfstats.csv“;”

This will give you a 10 minute record of performance from the array which you can analyze using Excel. The Dell Enterprise Center TechCenter Wiki has a great write-up on how to efficiently analyze the data from this command here: https://www.delltechcenter.com/page/MD3000i+Performance+Monitoring, complete with a YouTube video that walks you through the process:

I am beginning to think that the DS3300 (and MD3000i) may actually be a viable starter solution for SMB’s starting out on a virtualization project. But I would recommend the cache upgrade, 2nd controller, SAS disks instead of SATA to eliminate the SAS-to-SATA translation overhead and more faster disks instead of fewer slower disks so you can drive throughput and IOPS to a higher level.

Have any of you deployed the DS3300 or MD3000i (or the generic LSI solution)? Do you have any performance tuning tips for these arrays? If so, share in the comments!

VMware Networking Demysified

March 20, 2009 by Josh Townsend 1 Comment

VMware vExpert and fellow Northern Virginian, Ken Cline, has posted an excellent article on his Ken’s Virtual Reality blog that aims to demystify VMware networking. The article, the first in a new series by Ken, provides an overview of networking in an ESX/ESXi environment and breaks down the intricacies of the vSwitch and VLANs. The article comes complete with some nifty diagrams to help make sense of the topic. The timing of this article is great for me as it helps to frame my thoughts as I delve into the design of my latest VMware project on an IBM BladeCenter with IP SAN storage.

Great article, Ken! I look forward to reading the rest of the series.

DL380 BIOS Configuration for VMware

March 16, 2009 by Josh Townsend 1 Comment

One more post to wrap up the nonsense with my DL380 G3 ESX servers….

Vincent Vlieghe noted that you must make a couple changes to your DL380 G3’s for ESX to work correctly. His post was written back in 2006 when we were still working with ESX 2.x, but the same appears to be true of ESX 3.5 RTM (Updates are not supported on this hardware per the HCL). The changes you must make to BIOS are:

For stable operation on these systems, ESX Server requires a BIOS MPS Table Mode setting of Full Table APIC. With the exception of the specific systems referenced below, the following BIOS settings must be applied in order if available:

System Options > OS Selection: Select Windows 2000.

Advanced Options > MPS Table Mode: Select Full Table APIC.

When presented with multiple Windows options (Windows 2000, Windows Server 2003, Windows .NET, and so on) select Windows 2000. If both BIOS settings are available and can be modified, both must be set correctly. You should confirm these settings after any BIOS upgrade operation.

I have seen other references that say that you should also disable hyperthreading on this platform, but I was able to successfully run with Hyperthreading enabled with no performance degradation or stability issues. I hope this information is helpful to those of you still running these dinosaurs!