One of the things I’ve really enjoyed since joining Clearpath Solutions Group is participating in our learning events.  Coming from the customer side, I didn’t get out to learn and interact with the community as often as I would have liked.  With the incredible rate of innovation in IT and paradigm shifting technologies like virtualization, cloud computing, & Everything-as-a-Service, it is increasingly important to take a break from the day-to-day and pause to re-imagine what you could and should be doing to prepare for the next wave of technology.  Events like the VMware User Group (VMUG), the VMware Virtualization Forum (coming to Washington, DC on June 19, 2012), and Clearpath’s Lunch & Learn’s are a great way to get out and get educated while having some fun.

Clearpath’s next big event will be in Baltimore, on Tuesday, May 29, 2012 at M&T Bank Stadium (Ravens Stadium).  Together with VMware, Cisco, and EMC, we’ll discuss private cloud technologies from EMC, VMware and Cisco, along with complimentary cloud computing services for disaster recovery, virtual desktops and infrastructure services. Learn what a hybrid cloud strategy can do for your IT operations and budget. You’ll learn about technologies that are available for immediate deployment and impact.  Expect a bit of hands on with gear from Cisco (Unified Computing System – UCS) and EMC (VNX arrays), as well as product demo’s from VMware (vSphere, View, and vCloud Director).

A light lunch will be provided followed later by an appetizer and cocktail hour towards to the conclusion of the event. PLUS - The event will include a tour of the stadium and special appearance with autograph signing from Ravens Wide Receiver, Torrey Smith. There will also be raffle prizes and giveaways including a trip to VMworld 2012 in San Francisco with hotel and airfare paid for!

Space is limited!  Sign up for this special event today!

Planning for the Cloud | Event Details:
Date: Tuesday, May 29th, 2012
Time: 1 – 7:30 pm
Location: M&T Bank Stadium
1101 Russell Street, Baltimore, MD

Register for the event at Clearpath’s website here: http://clearpathsg.com/planning-for-the-cloud

I recently presented at a couple of Clearpath Solutions Group Lunch & Learn style events that focused on VMware View and touched on many of the components of VMware’s growing End User Computing portfolio.  The presentation walked attendees through the concept of VDI, why it matters, and how IT groups can leverage desktop virtualization to transform from an or organization servicing devices (legacy desktops) to an organization that delivers services (apps & data) to users.

A few key points that I wanted to call out from the presentation are:

• Cloud is both a journey & a destination (and an overused buzz word).  Desktop virtualization is the beanstalk in the cloud-provided end user computing environment – it let’s us climb from an environment rooted in Windows to the cloud.
• Desktop virtualization acts as a decoupling agent between the OS and apps/data that users really want.
• Desktop virtualization provides a reset button for IT, allowing us to re-imagine, reclaim, and redeploy IT assets (people, equipment & budget).
• Desktop virtualization changes the economics of end-user computing from CapEx to OpEx, preparing way for consumption-based cloud models.

I’ve embedded the presentation below  (I originally built it in PowerPoint and converted it to SlideRocket – some of the slides and animations didn’t import well).  I’m happy to do a WebEx of the presentation if you want to learn more.

If you have questions or want to learn more about how View can fit in your environment feel free to reach out.  If you are local to Washington DC, Virgina, or Maryland watch for upcoming Clearpath events here: http://clearpathsg.com/calendar.

VMware released an update to VMware Workstation 8.0.3 Build 703057 yesterday (May 3, 2012).  The update is a security update addresses two vulnerabilities that would allow out-of-bounds memory writes on floppy and SCSI devices under certain conditions.  The vulnerabilities may allow specially crafted attacks to crash the VMX process (i.e. crash the running VM) or theoretically execute commands on the host.  The VMware Workstation 8.0.3 release notes can be found here: https://www.vmware.com/support/ws80/doc/releasenotes_workstation_803.html.

VMware’s security advisories for these vulnerabilities can be found here: http://www.vmware.com/security/advisories/VMSA-2012-0009.html.

Users of VMware Workstation should be prompted to update to the latest version when launching the application:

If you are not automatically prompted to update VMware Workstation, manually check for updates from the Help menu:

You can also download the full install package of VMware Workstation from http://www.vmware.com/products/workstation.

While not specifically mentioned in the release notes, I wonder if this update was prompted by the VMware source code leak described in the VMware Security Blog: http://blogs.vmware.com/security/2012/05/vmware-security-note.html.  If so, kudos to VMware’s security and engineering teams for quickly identifying, fixing, and releasing updates to correct flaws that may be exposed through this code leak.

I’ve had several people ask me about using CD/DVD drives on VMware View virtual desktops.  Specifically,

• Can I use a CD or DVD drive with VMware View Desktops?
• Why doesn’t the CD or DVD drive on my re-purposed PC with the VMware View Client pass through to the View desktop?
• How can I make a CD or DVD drive work on my View Desktop?
• Are there specific CD/DVD drives that are supported on VMware View?
• How can I present a CD image (.ISO) to all View Desktops?

CD/DVD Devices in a View Environment – What works, what doesn’t?

First, the bad news.  When using PC over IP (PCoIP) as the connection protocol, IDE, SCSI, or SATA attached CD or DVD (or BluRay, HD-DVD, LaserDisc) drives local to the VMware View Client are not passed through to the VMware View virtual desktop.  This quite simply means that the internal CD drive of your re-purposed desktop will not be mapped to the View desktop when using PCoIP.  That said, mapping local CD or DVD drives to a View virtual desktop is supported when using Microsoft Remote Display Protocol (RDP).  Unfortunately, user experience (video performance, audio quality) declines and low/latent bandwidth  can increase if you switch from PCoIP to RDP.

The better approach to presenting a local CD/DVD drive to a View Desktop is to use a USB-connected drive.  This approach will work with both re-purposed PC’s acting as your View client as well as with Thin Clients and Zero Clients.  USB-connected devices can be passed through from a View Client to a VMware View virtual desktop while using PCoIP.  Teradici (the creators of PCoIP and maker of most of the Tera chips in industry leading Thin and Zero clients) has published a list of tested USB CD/DVD drives on their KB here http://techsupport.teradici.com/ics/support/default.asp?deptID=15164&task=knowledge&questionID=566.  Note that if you use some of these drives with a Windows XP View desktop, you may experience a View disconnect when ejecting a disc from the drive.  Other USB CD and DVD drives may also work – I grabbed this drive (OWC SuperSlim USB 2.0 Enclosure) from a colleague and was able to pass it through to my Clearpath Cloud hosted Windows 7 View desktop with no problems.

CD/DVD Drive Performance in a View Environment

While a USB connection will get your CD or DVD drive connected to your View desktop, performance may not be amazing.  For many tasks like software installs, reading images, or copying files performance will be acceptable.  A Teradici client can support up to 15Mbps for USB connected devices.  On a low bandwidth connection, or a connection with high latency, a local USB connected CD/DVD may not work well, if at all.  On a fast LAN connection, I would expect that redirected CD/DVD performance would be acceptable for many tasks, including reading medical images (PACS, DICOM viewer, RIS, HIS).  Do not expect to stream media or watch movies from a USB connected CD/DVD drive – but because View is typically a business solution, and watching the latest Hollywood hit at work is not often a business requirement, this should present few problems in all but the smallest niche cases (so get back to work!).  Disc burning would work with this configuration too – better for burning small images than a mass disc duplication effort.

Alternative Methods and Niche Cases

I’ve run into a few situations over the past decade or so of IT work where a desktop needed to have a CD/DVD perpetually connected to support some crummy application (FoxPro comes to mind here).  If you are supporting an environment like this, you can configure your parent VM with a persistently connected ISO on a VMFS or NFS vSphere datastore. Using this method, you could present the same CD image (.ISO) to all the VM’s based on the parent VM.  The connected ISO setting carries over to View Composer replicas and linked clones.  If you wanted to target this configuration to just a specific pool of View desktops, create a new snapshot of the parent image with the virtual CD/DVD drive set to ‘Connect at power on’ and an ISO file selected in the ‘Datastore ISO File’ field, as pictured below:

Note that in the VMware View Optimization Guide for Windows 7, and many other View tuning guides, it is recommended to remove the virtual CD/DVD drive from the parent VM.  If you follow this recommendation, you could still leverage a virtual CD/DVD drive, such as Microsoft Virtual CD Control Tool or UltraISO can also be used inside the View guest operating system, with an ISO library established on a low cost shared storage tier network share to give users a self-service way to mount a CD or DVD image.

Restricting Access to CD/DVD Devices in a VMware View Environment

While VMware-empowered IT is all about giving users freedom and control, there may be times where you want to restrict access to USB CD/DVD devices (or any other USB devices) in your VMware View environment.  You can do this in several ways – a complete solution will likely use a combination of the following methods based on user requirements, administrative overhead, and security posture:

• Use VMware View Group Policy administrative templates (.adm) applied to the View Client to control either ‘Connect all USB devices to the desktop on launch’ or ‘Connect USB devices to the desktop when they are plugged in’.
• If using RDP, use the same View Client (vdm_client.adm) administrative template to disable the ‘Redirect supported plug-and-play devices’
• Use built-in Active Directory Group Policy Objects (GPO), applied to your View Desktops OU’s, such as

• Administrative Templates | System | Removable Storage Access
• Administrative Templates | Windows Components | Remote Desktop Services | Device and Resource Redirection
• Administrative Templates | System | Device Installation | Device Installation Restrictions
• Use your Thin Client or Zero Client vendor’s management software to restrict access to, or disable, USB devices on Thin Clients.

Serial port devices in a VMware View environment

I’ve also had a couple questions about serial device (DB9 COM port) connections to View desktops.  Most of the above applies to serial devices too.  If serial devices are required in your environment, pick up a USB to Serial adapter (something like this will do Cables To Go 26886 USB To DB9 Male Serial Adapter, Blue (0.45 Meters/1.5 Feet)) and use GPO’s to control access and redirection of the USB device.

Wrap-up

Hopefully this answers your questions on using CD and DVD drives in a VMware View virtual desktop environment.  A combination of VMware View settings, vSphere VM configuration settings, and Microsoft Group Policy Objects (GPO’s) give you the ability to use and control removable media devices in your View environment.

Have you struggled with any other devices in your VMware View environment – leave a comment with your questions or suggested work-arounds!

Note: This article is cross-posted at the Clearpath Solutions Group blog at http://clearpathsg.com/weblog.

I’ve been asked several times recently to recommend training resources for VMware, so I thought I might write my responses up in a blog post to help out folks in the community who are looking for the best resources to gain VMware knowledge, prepare for their VCP and other certifications, and continue on their journey to becoming a virtualization rockstar.

I’ve picked up a bunch of certifications over the past 10 years.  For me, certification is not the means to an end, but the end of some long, intensive studying and lab work, then doing some deep dive studying and doing.  By the time I get to the test, passing should be a forgone conclusion.  I’ll save details of my lab for a future post and focus on the books and other learning resources that I use.  When getting into a new or updated technology, I start out my studying with a good overall survey of the technology I want to learn.  I like a good book that hits all of the major components, provides background information to help explain why the technology, component, or module really matters and how it fits into the big picture.  Then I get into technology specific books – deep dives, command line references, and architecture books.

Books

Mastering VMware vSphere 5, by Scott Lowe

My go-to book for VMware vSphere has been Scott Lowe’s Mastering VMware vSphere 4.  Scott’s updated book, Mastering VMware vSphere 5 started shipping yesterday.  Scott covers everything from the basics of what a hypervisor is to VMware vSphere best practices.  This is a great book to accompany lab work as it includes licensing, planning and installation, setting up virtual networking, storage basics, security, resource allocation, HA, DRS, and even some automation with the CLI and PowerCLI (PowerShell).  The book is well written, taking you methodically through vSphere, while providing plenty of helpful hints along the way.  Do yourself a favor and click the picture to the left to order it from Amazon now (paperback or Kindle format).  This book is a great way to get started with studying for your VCP certification.

VMware vSphere 5 Clustering Technical Deepdive

Once I have the basics down, I get into the deep dive work. The first deep-dive book for VMware vSphere 5 is VMware vSphere 5 Clustering Technical Deepdive, by Duncan Epping and Frank Denneman.  This is Duncan and Frank’s second book that focuses on the clustering and high availability technologies available in VMware vSphere.  Readers of Duncan and Frank’s first book, VMware vSphere 4.1 HA and DRS Technical deepdive (Volume 1), got an incredibly deep look at how to configure VMware HA and DRS.  The new vSphere 5 Clustering Technical Deepdive includes Storage DRS as well.  I’ve talked to several readers of both these books and Duncan and Frank’s blogs who have remarked that 1.) I’ve been doing it wrong all along, 2.) I totally understand how HA and DRS work after reading this, and 3.) My environment really is resilient and reliable thanks to this book.

Pearson and VMware teamed up earlier this year to create VMware Press.  There are several books coming from VMware Press, as well as other authors/publishers that are now available for pre-order from Amazon.com.  These include:

There are not many vSphere 5 specific books out yet, but many of the vSphere 4 resources are still very useful.  My library includes these:

Video Training

If you are not a big reader or you are looking for additional topics, check out TrainSignal’s VMware Training Videos.  TrainSignal offers a whole slew of courses (many taught by VMware vExperts), including:

• vSphere 5 Training
• VMware View Administration Training
• vSphere Troubleshooting
• vSphere Performance Monitoring
• vSphere Security Design
• vSphere PowerCLI.

I have a couple of TrainSignal DVD’s and found them to be good quality with deep technical content.

Blogs and BrownBags

Once you are comfortable with the material, you can start to study for your VCP.  Several bloggers have published collections of materials to help you prepare for the VCP, VCAP, and even the VCDX.  I recommend Simon Long’s collection here: http://www.simonlong.co.uk/blog/vcp-vsphere-upgrade-study-notes/ and Cody Bunch’s VCP4 Resource Page and BrownBag sessions.

Instructor Led & Certification

Finally, once you are all read up, head to a VMware Education instructor led class.  You need to take a VMware Authorized Training course to qualify to sit for the VMware Certified Professional (VCP) certification exam.  VMware also offers a nice catalog of eLearning courses.  If you want to get a discount on eLearning, Instructor-Led training, and certification exams from VMware, check out the VMUG Advantage program.

Here’s the SlideRocket presentation for tomorrow’s DC VMUG.  My first real time playing with VMware’s Slide Rocket. It took me a few crashes of PowerPoint to move to SlideRocket (even after I had included it in the recent VMware Acquisitions slide).  Overall a pretty good experience!

This presentation is subject to change (I’m running out of steam tonight).  If/when I do update it on SlideRocket, this embedded deck will just magically update – the magic of the cloud!

Feel free to comment with changes or suggestions for other free utilities and scripts!  I  added a Scripts & Free Tools page to http://vmtoday.com today with some embedded Google Docs spreadsheets to keep a running list of the cool VMware scripts and tools that I come across. I’ve found and promptly forgotten so many over the years – hopefully this helps me as much as it helps you in finding and saving some great community resources!

This is your last chance to register for tomorrow’s (July 12, 2011) big VMware online event, “Raising the Bar, Part V”.  During the event, VMware CEO Paul Maritz and CTO Steve Herrod will announce the next generation of VMware cloud infrastructure offerings.  The event will be broadcasted live beginning at 9:00 AM PDT

The agenda looks like this:

• 9:00-9:45 Maritz and Herrod present
• 10:00-12:00 Deep dive breakout sessions
• 10:00-12:00 Live Q&A with VMwarevExperts

While I don’t have any inside knowledge on the event, I expect new product and service offerings, along with details on recent acquisition announcements on how VMware is bringing to bear its vision for a seamless cloud infrastructure – public, private, hybrid – that is simple to manage, easy to scale, and with all the dynamism that one could imagine.   And if that weren’t enough, one lucky attendee for this online event will win a ticket to VMworld 2011.  This event promises to be a great intro to what is shaping up to be a great VMworld 2011 in Las Vegas!

VMware has been investing in the VMware User Group (VMUG) program over the past year, adding new benefits and opportunities to an already great program.  In addition to local meetings where you can interact with fellow VMware professionals, the VMUG program offers other great benefits – the latest is the VMUG Advantage program. The VMUG Advantage program offers the following benefits:

• $100 Discount to VMworld
• 20% Discount on VMware Instructor Led Training
• 20% Discount on VMware Certification
• 1-Year Free Subscription: All Access eLearning Courses ($750 Value)
• 30% Discount on Workstation and Fusion Software html Licenses

The VMUG Advantage program has an introductory price of $170 for individuals (with corporate/multi-user discounts available) until May 21.  Not a bad deal if you plan to attend VMworld in Las Vegas (Aug 29 – Sept 1), complete a certification (VCP, VCAP, VCDX, VCA-DT), or take some instructor led training through VMware.  Heck – the eLearning subscription alone more than pays for program subscription cost!

As a VMUG member be sure to login to the new www.myvmug.org and update your profile or add an picture or avatar. Over the next month you’ll see exciting new benefits added to the website, including VMUG Community Pages, Event Listings, and Education Resources.

If you’re not already a VMUG member, go ahead and sign up at www.myvmug.org to learn about local/regional events and to get access to the other great benefits like the VMUG Advantage.

My last post described a problem I experienced with VMware HA after upgrading to vSphere 4.1.  Here is my experience with a similar issue after applying the ESXi410-201010401-SG patch to one of my test/dev ESXi clusters.  The patch, released on November 15th and weighing in at a hefty 212MB, fixes a number of issues from Likewise authentication on ESXi hosts to allowing configurable NOOP timout and interval values for faster failover of certain iSCSI arrays (like the DS3300 or MD3000i).

The environment where this problem occured has a single vCenter server managing both a production cluster and the test/dev cluster.  After applying this particular update to the ESXi hosts in the cluster, the vCenter server began to crash every 5 minutes or so.  The crash was logged on the vCenter server with Event ID 7031: The VMware VirtualCenter Server service terminated unexpectedly.  My go-to troubleshooting question (“What changed?”) pointed at the ESXi patch, but a VMware KB search and a little Google action yielded no results directly related to ESXi410-201010401-SG and the vCenter Server service terminating unexpectedly.  VMware KB article 1003926 provides some basic troubleshooting steps for vCenter Server, such as checking for port conflicts, vCenter DB health & availability, and log locations.  The environment was healthy until the patch was applied to a sub-set of my ESXi hosts so I could confidently eliminate credentials, port conflicts and the like as the cause of the problem, so I jumped right to the log files for vCenter.  The vpxd-*.log is found in “C:\ProgramData\VMware\VMware VirtualCenter\Logs” on Windows 2008 vCenter servers and “%ALLUSERSPROFILE%\VMware\VMware VirtualCenter\Logs\vpxd.log” on Windows 2003 servers.  I found a few lines of interest in the log file but decided I had better call VMware Support to further analyze the issue.

To make a long story short, what the logs revealed is a bug that is triggered whenever VMware Distributed Resource Scheduler (DRS) ran on the updated test/dev cluster.  Disabling DRS stopped the symptom of the vCenter Server Service terminating unexpectedly, but this was obviously not a long-term solution.  A bit more digging by my VMware support rep led to VMware Distributed Power Management (DPM) being enabled on the cluster as the root cause of the issue.  Disabling DPM but leaving DRS enabled on the cluster fixed the glitch.  I can live without DPM, but DRS is pretty darn handy.

At this point, VMware engineering knows about the issue, and a fix is planned for vCenter 4.1 Update 1.  Interesting that DPM was fingered in this case, as well as in the case I wrote about last week where HA and DPM apparently do not always play well together.  It seems like DPM is not fully baked, even though it is now officially supported.  This is unfortunate as DPM is promising to me – I can imagine the technology behind DPM being used for intelligent load shedding during peak electrical cost hours, power outages, or cooling outages in datacenters with some good integration between a DPM API and environmental management and monitoring systems like APC’s NetBotz.  Anyone else using DPM without having problems?  Any ideas for extending DPM or leveraging it for other purposes in the datacenter – I’d love to hear ideas in the comments.

I recently ran into an issue with one of my vSphere clusters after upgrading from vSphere 4.0 to vSphere 4.1 (with ESXi 4.1 and vCenter 4.1).  After the upgrade, I attempted to enable VMware High Availability (HA) on the upgraded cluster.  Each of the ESXi hosts in the cluster appeared to have been properly configured for HA (as observed in the ‘Recent Tasks’ pane of the vSphere Client).  Despite having appeared to configure HA correctly, I found that each host in the cluster was displaying an error on the Summary tab of the vSphere Client that read ‘Error <date> <time> HA agent on <host> in cluster <clustername> in <datacenter> has an error: Error while running health check script’.

I’ve dealt with HA errors in the past, so I quickly jumped into my standard troubleshooting and quick-fixes proceedure:

1. Verify host connectivity.
2. Right-click on each host and choose ‘Reconfigure for VMware HA’
3. Disable & Re-enable HA on the cluster.
4. Disable HA, place hosts into Maintenance Mode & Reboot (one at a time).  Re-enable HA.
5. Get frustrated that a quick fix is not probably not in my future….
6. Verify host name resolution for each host in the cluster from the service console/tech support mode of each host.
7. Review log files on vCenter Server and each host for glaring issues.  All Greek to me in this case….
8. Call VMware Support.

VMware Support reviewed the log files I had attached to my Service Request (SR) when I opened the case and had me try a few different things to fix the issue.  First, we verified the steps I had taken and collected some fresh logs.  Next, the support rep had me verify that Distributed Power Management (DPM) was not enabled on the cluster as there apparently is a known issue (although a KB is not available at this time) with configuring HA when DPM is enabled under certain circumstances.  I did not have DPM enabled on this particular cluster so I didn’t spend time chasing down this particular bug.

Finally, the following proceedure, run on each ESXi server in the cluster, resolved the issue (Note – this procedure is safe to do during normal operations as it does not affect running VM’s):

1. Verify SSH or Console access to the host (this requires enabling Remote SSH/Tech Support Mode on ESXi hosts on the Configuration tab | Security Profile node of the vSphere Client, or by pressing F2 to login to ESXi 4.1 | troubleshooting options | enable remote SSH.
2. Disable HA on the affected cluster.
3. Right-click | Disconnect each host in the cluster from the ‘Hosts & Clusters’ view of the vSphere Client.
4. SSH to the host and run the following commands:

services.sh stop
/opt/vmware/uninstallers/VMware-vpxa-uninstall.sh
/opt/vmware/uninstallers/VMware-aam-ha-uninstall.sh
services.sh start

5. In the vSphere Client, right-click on each host and Connect.
6. Enable HA on the cluster.

This procedure cleanly removes the VMware vCenter agent and the VMware HA agent from the ESX or ESXi host.  Reconnecting the host to vCenter pushes the vCenter management agent back to the host and installs it cleanly.  Enabling HA on the cluster re-installs the HA agent.  After completing these steps I had no further issues with HA on the cluster – case closed.  I hope this is helpful for anyone else who might be experiencing HA errors after upgrading to vSphere 4.1.

For those wanting to learn HA best practices or go a bit deeper into the inner workings of VMware HA, I highly recommend Duncan Epping’s VMware HA Deep Dive article and/or VMware vSphere 4.1 HA and DRS Technical Deepdive (Volume 1) book.

Time:
9:30am – 2:30pm

Location:
Fair Lakes Hyatt
12777 Fair Lakes Circle
Ballroom
Fairfax, VA 22033

Directions

I’ve heard some grumbling about the long lines popping up at VMworld with folks blaming the new first-come format for admission into the various sessions. A few folks have asked me my take on it – here’s what I’ve got:

- Yes, the lines are long but they seem to move fast. Whether that’s because people are bailing out or because the staff here is efficiently moving people into session rooms as they open up I don’t know. I suspect a bit of both.
- I anticipate lines getting shorter over the next few days as the Solutions Exchange opens up a ton of space and activities for VMworld participants.
- As people get a feel for the flow of the event, things will start moving along better. Rather than jam-packing as many sessions into their schedule, people will start to balance vendor time on the Solution Exchange floor, Hands-On Labs, and other activities.

What has your experience been so far? Any suggestions for improving the VMworld experience? I’d love to hear your ideas in the comments!

Way back in the day (we’re talking way back in high school here) I worked my summers away at a Six Flags park in Western New York.  Every fall the park hosted an International Food Festival.  Let me tell you – the perogies, sausage and gyro’s slid down like nobody’s business.  But the real prize was the Italian bakery’s cannoli’s.  The 5 or so folks that shared an office with me decided we should track our cannoli consumption.  We did (like you do a drug or spin up VM’s just cause you can) something like 126 cannoli’s in one weekend. We called it the Cannoli Count – kept a tally o a whiteboard in the office.  Sickening, right?  So, what does this have to do with virtualization, you ask.  Well, not much really, but here is where I am going with it.  T-shirts are handed out like nobody’s business at VMworld, and they accumulate like a pack-rat’s pile of newspapers in my dresser drawers (I still have a high-school wrestling t-shirt that dates back to 1994 in the rotation).  My VMware Widow wife hates them, so I figure I’ll see how many more I can collect this week.  With two already in my hands after VMworld check-in, enter the:

Josh’s VMworld T-Shirt Count

||

I’ll keep the running tally going throughout VMworld – let’s see where this ends up (besides me cleaning out my dresser to avoid sleeping on the couch when I get home).

I was fortunate enough to be offered a sneak peak at the VMworld 2010 Hands-on Labs setup this morning, and let me tell you – I am impressed.  A lot of hard work has gone into planning, architecting and deploying the Labs environment, promising to make it the most user-friendly VMworld Labs setup yet.  Here is what you need to know:

Location: The Labs will be held at Moscone West, on the corner of 5th & Howard St. This is a change from last year.

Format: There will be two types of hands-on labs – instructor led (they’re calling these Advanced Lab Tutorials) and self-paced.

• The instructor labs are more of a tutorial for those who want to be walked through the lab manual by a subject matter expert in an open discussion format.  The Advanced Lab Tutorial sessions support 250 seats.  The Advanced Lab Tutorials will be useful for preparing for the associated self-paced labs.  Take the Advanced Lab Tutorial first, then head downstairs to the Self-Paced lab.
• The self paced labs are designed with a ton of flexibility, allowing you to choose what and when you work through the material.  For an overview of the Lab topics, check out the VMworld 2010 Program Guide.
• When you arrive at the Self-Paced Labs area, you will register for the lab you want and head to a nice waiting area if no seats are available.  When your number is called, you will be lead to your seat and will fire up your lab.  You’ll have an hour to work through the lab.  If you need more time, ask.

Technical Specs: The VMware Core Team has obviously put an enormous amount of thought and time into improving the lab experience.  For those who attended VMworld 2009, the lab experience folks a bit disappointed due to some technical glitches and scheduling issues.  This year’s Labs are built with a ton of redundancy and allow for a much smoother, user-directed schedule.  The scale and scope of the labs is astonishing to say the least.  Here are some stats I gleaned on the lab setup:

• There are 30 self-paced lab topics, each demanding their own unique environment.
• There are 480 seats available for the self-paced labs, in a stadium seating configuration.  This allows a huge number of people to flow through the lab environment efficiently, with minimal wait time.  The lab schedule has some 40 hours of time for you to get in and work over the next several days.  This equates to more than 20,000 lab-seat hours (up from about 5000 hours last year).
• The labs run from one of three data centers: Miami, FL (Terremark); Ashburn, VA (Verizon); and locally in the Moscone Center.  This provides a great deal of redundancy and positions the labs as a cloud offering to fit the theme of this year’s VMworld.  The Miami and Ashburn sites have been running for a while, and will be reused for VMworld Europe next month.  This is a change from last year where the gear was fork-lifted in for the show (remember all the racks at the bottom of the escalators?).  This has given the team more time to work on the setup and iron out any problems.
• The self-paced labs are based on VMware’s Cloud Lab infrastructure, purpose built for VMworld Labs.  Cloud Lab provides a slick interface for provisioning labs to participants while doing some really smart things in the background to enhance performance and flexibility.
• It is estimated that more than 100,000 VM’s will be provisioned in Labs this week – more than 5000 VM’s built and destroyed per lab hour! <- Read that again. Astonishing, no?
• The gear driving the labs is provided by HP, Dell, EMC, NetApp, Cisco, and Xsigo.  Xangati is used for monitoring performance of PCoIP to the Wyse thin clients at each seat.
• There are 4 racks of compute power and 2 racks of storage per datacenter.
• The storage environment is mostly 10GbE.  EMC FastCache and NetApp Dedupe are both in use.  Storage is mostly NFS-based.
• The memory footprint required to run the labs is some 36TB.
• Labs are running a few levels deep – ESX nested inside of ESX with VM’s running inside.
• Host Profiles are heavily leveraged to ensure a consistent environment.
• Twin DS3′s provide Internet connectivity for the Labs.
• In true cloud fashion, the Lab Cloud product dynamically pre-populates lab environments based on demand.  As some labs rise in popularity, the Lab Cloud will stage up environments based on that demand.  This will reduce wait time for the lab environment to be readied.  In years past, students would wait 5-7 minutes for their custom lab environments to be readied (building, deploying and booting a unique Active Directory, vCenter, ESXi, nested VM’s and associated products takes some time).  No guarantees that there won’t be some wait time, but this is a huge step in the right direction.
• There will be some 150 moderators ready to help with Self-Paced labs.  Moderators are subject matter experts.  If you request help through the Lab Cloud interface, a moderator who is a SME in your topic will be dispatched to help you.

A few more things to note:

• There will be prize drawings for those who do the most labs, as well as those who complete the labs the fastest.  Prizes will include a full pass to VMworld 2011 in Las Vegas.
• Lab manuals will be made available after the show.
• Some of the labs look really cool. You can find a list in the VMworld 2010 Program Guide.  I am excited to see the VMware vSphere Sandbox lab – an everything-but-the-kitchen-sink setup of as many products as they could cram in.  This provides a playground for you to see all of the VMware products working together, where you can create, destroy and otherwise play as you wish.
• I would love to see this environment be made available for other uses after VMworld.  I think VMUG’s could really benefit, as could VMware’s partner community.

Special thanks to Adam Zipman who leads the team putting this together, Dan Anderson (Dan is the lead architect behind this massive operation) and Curtis Pope who led development of the Cloud Lab interface.  Also, thanks to John Troyer for setting up this morning’s breifing.  I appreciate your time today, guys.

I hope you all are as excited about the labs this year as I am.  I am planning to spend a good chunk of time working through the lab environments.

I ran into an issue with a customer today where a VM was performing terribly.  From within the guest OS (a Windows 2003 application server running .NET in IIS which I will call BigBadServer) things appeared sluggish and CPU time was high.  The amount of time being spent on the kernel was notably high.  The VM in question had 4 vCPU’s and a good helping of memory.

I don’t have access to the VMware client at this particular site – just some of the guests, so I was flying blind.  Gut feeling told me that I was dealing with a resource contention issue.  I had the VMstats provider running in the guest (http://vpivot.com/2009/09/17/using-perfmon-for-accurate-esx-performance-counters/) showed me that there was no ballooning or swapping going on, and that the vCPU’s were not limited and the CPU share value seemed to be at the default.

I strongly suspected that the physical server running VMware ESX was oversubscribed on physical CPU (pCPU) resources.  Essentially, the guest VM’s that are sharing the resources of the physical machine are demanding more resources than the machine can handle.  To verify this theory, I had the client check the ‘CPU Ready’ metric on BigBadServer and bingo!

CPU Ready is a measure of the amount of time that the guest VM is ready to run against the pCPU, but the VMware CPU Scheduler cannot find time to run the VM because other VM’s are competing for the same resources.

From the stats the customer provided on our phone call, the CPU Ready for any one of the 4 vCPU’s on the BigBadServer was on average 3723ms (min: 1269ms, max:8491ms).  (Update 8/25/2010 to clarify summation stat) The summation for the entire VM was around 12,000ms on average and peaked around 35,000.  The stats came from the real-time performance  graph/table in the vSphere client. The real-time stats in the vSphere Client update every 20 seconds, so the CPU Ready summation value should be divided by 20,000 to get a percentage of CPU ready for the 20 second time slice.  If I take the worst case scenario of 8491ms per vCPU, this VM spent nearly 43% (8491/20,000) of the 20 second time slice waiting for CPU resources.

The CPU Ready summation in milliseconds counter in the vCenter Client is not always the most accurate or easy to interpret stat – to better quantify the problem it might be best to go to the ESX command line and run ESXTOP.  CPU Ready over 5% could be a sign of trouble, over 10% and there is a problem.  Running ESXTOP in batch mode and then analyzing the output using Windows Perfmon or Excel might be a good way to go on this to get a view over several hours rather than the realtime stats we were looking at.  I wrote a post a while back with more info on ESXTOP batch mode: http://vmtoday.com/2009/09/esxtop-batch-mode-windows-perfmon/

To help quantify the problem a bit more, the BigBadServer is on an ESX 4.0 server with about 10 other servers.  The physical blade has two dual-core CPU’s (AMD Opteron 2218HE’s which are not hyperthreaded).  The other VM’s on the blade have different vCPU and vMemory configurations.  3 VM’s (including BigBadServer) have 4 vCPU’s.  A couple have 2 vCPU’s, and the remainder are configured with 1 vCPU.  In ESX 4.x, the VMware console OS actually runs as a hidden VM, pegged to pCPU #1.

I generally recommend a pCPU:vCPU ration of 1:4 for mid-sized VMware deployments of single vCPU VM’s.  The blade we are running on is a 1:5 with several multi-vCPU VM’s.  The multi-vCPU’s start to skew the ratio recommendation and require some advanced design decisions.  VMware’s scheduler requires that all the vCPU’s on a VM run concurrently (even if the Guest OS is trying to execute a single thread).  Also, the VMware CPU Scheduler prefers to have all the vCPU’s from a VM run on the same pCPU.  As workloads are bounced around between pCPU’s, the benefits of CPU cache are lost.  This is one of those ‘more-is-less’ situations that you run into on virtualized environments.

What this CPU Scheduler nonsense means in this case is that the 4 vCPU’s on BigBadServer have to wait until all logical pCPU’s on the box are idle (including the one that runs ESX itself) before it can run.  If ESX can’t accomplish that (we are experiencing resource contention) it starts prioritizing workloads according to what it can best run.  It is much easier to schedule the smaller VM’s, so it tends to run those on pCPU more frequently.  The larger VM’s tend to suffer a bit more than the smaller ones.  We are competing with 2 other VM’s with 4 vCPU’s that use up all of the logical pCPU’s when they need to run, as well as with the smaller VM’s.

I suggested a few ways to fix this issue for the BigBadServer web server:

1. Using Shares and/or Reservations on the VM.  This probably won’t work in our situation as the physical server is too over-subscribed.  We might see a slight improvement in BigBadServer (or we might not see any change), but possibly at the extreme expense of the other VM’s sharing the blade.
2. Reduce the number of vCPU’s on BigBadServer AND the other multi-vCPU VM’s on the same physical server.  This would reduce resource contention and open up a whole bunch of scheduling options for the VMware CPU Scheduler.  This is the quickest/cheapest fix, but will not work if the VM’s really do need 4 vCPU’s.  A little workload analysis should determine which can be made smaller (the vCenter server graphs/stats should be enough for this).  For what it’s worth, by our analysis BigBadServer seems to be happier with 4 vCPU assuming we can run with a low CPU Ready on those 4.
3. Move the BigBadServer VM to a physical ESX server with fewer multi-vCPU VM’s so there is less contention.
4. Move the BigBadServer VM to a physical ESX server with quad-core pCPU’s (ideally two quad-cores or bigger).  This would give a lot more flexibility to the VMware CPU Scheduler and allow it to run quad-vCPU VM’s on the same pCPU for greater efficiency.
5. Split BigBadServer into 2 smaller VM’s – The server currently runs a couple sites.  We could split them onto two servers – one for Project1 and one for Proejct2.  This configuration would take some design, testing, and time but could scale out better, give more flexibility and availability in the long run.

I’m not sure which way the customer will go on this one yet, but I feel good having armed them with enough knowledge and options to make an informed decision.

To avoid problems like this in the future, I recommend these rules of thumb:

• Design your hosts for your guests.  Taking your Guest VM sizes into account when designing your environment and choosing physical hardware is crucial if you need bigger VM’s.
• Don’t make your VM’s bigger than you have to.  It is always easier to add resources than take them away.  Hot Add of CPU and Memory in vSphere make adding incredibly easy.
• Monitor your environment for CPU Ready, Swapping, and other metrics that can indicate an inefficient design.
• Call for help when you can’t figure out what is going on (I’m happy to help!).  VMware is super powerful, but some things can be downright backwards when it comes to resource allocation on a fixed set of hardware.

If you are looking for some resources to help explain CPU Scheduling a bit more, I recommend:

• VMware’s Official documentation of CPU Scheduler in vSphere 4.1 – http://www.vmware.com/files/pdf/techpaper/VMW_vSphere41_cpu_schedule_ESX.pdf.
• A nice summary of co-scheduling from VMware’s Performance Blog: http://blogs.vmware.com/performance/2008/06/esx-scheduler-s.html
• Description and stats on Ready Time metrics for VI3: http://www.vmware.com/pdf/esx3_ready_time.pdf
• Understanding Virtual Center Performance Statistics: http://communities.vmware.com/docs/DOC-5230.pdf

(Updated 8/25/2010 to include a few additional reference links and corrected summation divided by time slice to get accurate values)

I’ve held off on posting this as I don’t like to toot my own vuvuzela too loudly, but in a fit of shameless self promotion and a desire to see the program grow, here goes nothin:

I am a VMware vExpert!

VMware notifed recipients of the designation last month.  The 300 or so awardees are recognized for their “advocacy of VMware solutions, your contributions to the community of VMware users, and your willingness to share your expertise with others.”  I find it an absolute honor to be included amongst the vExpert class of 2010 – a group full of folks who are on the cutting edge of virtualization and cloud computing, many of whom have helped me through their books, blogs, and contributions to the VMware Community.  Arnim van Lieshout has compiled a nice list of vExperts here: http://www.van-lieshout.com/vexpert-2010/.

It’s an exciting time to be a virtualization professional – in terms of the technology, people and seemingly endless opportunities.  I look forward to continuing my contributions to the community through this here blog, my leadership of the Washington, DC Metro Area VMware User Group, and through my work as a implementer of VMware solutions!  Special thanks to John Troyer for sponsoring the program and doing so much to organize the VMware community through the weekly VMware Community Roundtable, Social Media and the many other programs he touches.

I upgraded my in-house VMware vSphere environment to 4.0 Update 2 last week.  After upgrading my vSphere Client to the Update 2 version I was greeted with a series of 7 ugly error messages stating: Method not found: ‘VMware.CustomControls.LabelEx VpxClient.Common.Util.Helper.AddLabel(System.Windows.Froms.Control,Int32, int32, System.String, System.Drawing.FontStyle, Boolean)”.I assumed a plug-in had caused the error message.  I started my troubleshooting by disabling the 3rd-party plug-ins in the environment, beginning with the free EMC Storage Viewer.  Upon disabling the EMC Storage Viewer 2.x plug-in the problem was resolved.  I went out to EMC PowerLink to see if an update was available for the plug-in and was surprised to find that I had missed a major update/rebranding of the plug-in.  EMC now calls the plug-in the ‘EMC Virtual Storage Integrator’.  A hotfix was released on July 2nd to bring Update 2 support to the plug-in (version 3.0.0.32).

I updated the Solution Enabler installation (I installed Solution Enabler on my vCenter server, but it is also available as a SUSE based virtual appliance), and then updated the plug-in.

The update appeared to install without any problems.  The vSphere Client launched like a champ after the update – no errors, but no EMC Storage plug-in either.  Odd.  The problem occurred on both my vCenter Server’s vSphere client and my workstation, so seems to be more than an isolated issue.

I did an uninstall of the plug-in using Add/Remove Programs, and then reinstalled it.  After the reinstall the EMC Storage plug-in icon appeared in my vSphere Client as pictured below.

The EMC Virtual Storage Integrator (VSI) plug-in is free – installing it is a no-brainer for anyone running Clariion or Celerra storage arrays.  The VSI simplifies the job of mapping vSphere Datastores to LUN’s and NFS shares on your EMC storage, and helps pinpoint the location of VM’s and RDM’s on your array.  This visibility for the VMware administrator into the storage layer can go a long way in helping troubleshoot storage performance issues and simplifying communication between server, storage, and virtulization teams.

EMC actually offers three different free vSphere plug-ins, including the VSI.  The EMC Unified Block plug-in and the EMC Unified NAS plug-in round out the trio.  EMC’s Virtual Geek, Chad Sakac, covers all three in his blog post here: http://virtualgeek.typepad.com/virtual_geek/2010/06/update-on-emc-vsphere-plugins.html.

At the risk of beating a dead horse, it’s time to resurrect my Storage Basics series.  I’ve recently had some great feedback on the series and figured I should round out a few of the concepts before I wrap it up.  I want to cover a topic often discussed amongst virtualization professionals, but one I often find general practitioners and server admins not understanding: storage alignment.  Storage alignment, or the lack of alignment, is not a new issue and is not unique to VMware or virtualization in general.  However, the effects of misaligned storage can be more greatly felt in terms of reduced performance and strain on a storage system in shared, oversubscribed or high I/O environments.  Many others in the virtualization and storage communities have already covered partition alignment (see Duncan Epping, Vaughn Stewart, and most recently Chad Sakac), but I feel it is an important enough topic for me to re-hash as part of this series.

What is Storage Alignment?

Let’s start with a quick overview of what storage alignment means.  Quite simply, storage alignment refers to the positioning (starting offset) of the various pieces of a systems storage components – the physical disk sectors or array’s chunks, the VMware File System (VMFS) in a VMware environment, and the guest file system’s clusters within a partition – in relation to the layer directly under the element in question.  A quick graphic often makes quick work of explaining this (I often whiteboard this concept for colleagues and clients):

As you can see, the starting offset of the VMFS partition does not correspond to the physical segmentation of the underlying disks (in this case, the chunks on a SAN – but could be conceptually replaced with the sectors of a single disk).  Furthermore, the clusters (or blocks) of the guest VM are not aligned to the VMFS partition nor to the underlying storage.  For traditional (physical) systems or VMware RDM’s, the VMFS layer could be abstracted, but the result would be the same – the clusters of a partition would be misaligned to the underlying disk.

What Does it Mean?

Quite simply, misaligned storage (both VMFS partitions and Guest File Systems) can lead to poor performance under certain conditions.  How badly performance is impacted depends on the degree of I/O strain your server and storage are under, the caching mechanisms in your environment, and the architecture of your SAN.  Again, a visual can help explain how misaligned storage can hurt you. For simplicity let’s leave out the VMFS layer as we consider the following diagram (pardon my hasty Visio visualization):

What we see is that the target data in a tiny 16kb read request spans two 64kb chunks on our storage array.  Any reads of that piece of data will result in twice the amount of data as would be minimally necessary being transferred to the host’s storage stack.  The net effect is an increase in the work the storage array must do – gobbling up IOPS that would otherwise be available for the real work of reading data, reducing throughput on the interface, and messing with cache algorithms and dedupe mechanisms on some arrays.  In short, misaligned storage is an efficiency killer.  Now add in the VMFS layer back in and you’ll see how things get complicated.

If (and we’re talking a big IF here) every bit of data you wanted to read spanned a chunk or sector boundary, you could experience half the expected performance due to misalignment.  In reality, depending on your workload and storage technology your performance increase from properly aligning your storage will probably be somewhere between 10-30%.

Want to dig deeper?

There have been some great resources published on this issue over the past few years on storage alignment.  Major vendors have all begun pushing information on the problem – here are some of the best that I have found:

Microsoft has a Knowledge Base article (http://support.microsoft.com/kb/929491) that describes the problem and symptoms of misaligned partitions, how to determine if your partition is aligned, and the use of diskpart to create aligned partitions.

Microsoft also has an in-depth article on MSDN, including some performance numbers at http://msdn.microsoft.com/en-us/library/dd758814.aspx.  Also check out Jimmy May’s series Partition (Sector) alignment for SQL Server here: http://blogs.msdn.com/b/jimmymay/archive/2008/10/14/disk-partition-alignment-for-sql-server-slide-deck.aspx.  One of the best descriptions of the complexities of the problem can be found in Jimmy’s blog series.

VMware has an article here: http://www.vmware.com/pdf/esx3_partition_align.pdf.  Be aware that this article is for Virtual Infrastructure 3, not vSphere 4.0.  Some of the information is now a bit dated.

Netapp has a few documents to check out: http://media.netapp.com/documents/tr-3428.pdf (VI3), and http://media.netapp.com/documents/tr-3749.pdf (vSphere)

EMC covers alignment in their TechBooks for Clariion, Celerra, and Symmetrix.

Tools to Align Partitions:

Ok – so you’ve bought into this whole partition alignment thing as being a real issue.  How to you fix it?  Here are some tools:

1. MSInfo32.exe, wmic, and dmdiag will show you misaligned partitions on Windows machines (check the Microsoft links above for usage info).
2. Diskpart.exe (or diskpar.exe on versions of Windows previous to 2003) creates aligned partitions on Windows systems.  Diskpart cannot be used to realign a previously created partition, only to create new correctly aligned partitions.
3. MBRScan/MBRAlign from NetApp can report on and realign existing virtual disks on a VMware ESX server.  Also a nifty PowerShell script from NetApp to find if your partitions are aligned: http://communities.netapp.com/docs/DOC-6175
4. vOptimizer from Vizioncore can report on and realign existing virtual disks.
5. GParted can be used to create aligned partitions on both Windows and Linux machines, and to realign some existing partitions.
6. VMware vCenter – VMFS datastores created using vCenter are aligned automatically. Note – Guest VMDK’s are not aligned automatically by vCenter – you must manually create aligned partitions on your VMDK’s or use a Guest OS that creates properly aligned partitions (Windows 2008 and later).

Best Practices:

Before I wrap this installment up, here are some best practices for storage alignment in your environment:

• Create aligned partitions in your VMware templates. Do it once, do it right – every machine you deploy from the template will be aligned.
• Use caution with tools like Symantec Ghost.  Ghost can take images of aligned partitions and misalign them when laying down on a new system.
• Use caution when performing P2V’s using VMware vCenter Converter – it does not align guest disks on import.  You might consider using Converter to perform a P2V of the system disk only, then create new VMDK’s on the converted guest.  Use Diskpart, gparted, or another tool to create aligned partitions on the new VMDK’s and finally copy the data over to the newly virtualized server using a tool like Robocopy, RichCopy, or rsync.
• SSD’s are particularly sensitive to misalignment, leading to poor performance and excessive wear.
• Local VMFS volumes created by the ESX installer are not aligned.  If you are using an installer-created local VMFS for anything where performance matters, you might consider re-creating it through vCenter.
• Watch out when attaching a data disk from an older VM to a new VM.  For example, you are upgrading your SQL servers to Windows 2008 R2 from 2003.  You decide to do a side-by-side upgrade, using the detach/attach method.  You install (or better yet, deploy from template) a new Windows 2008 R2 VM, detach your databases from the old server, move your SQL data and log virtual disks from your 2003 VM to the new VM and attach the SQL DB’s on the new server.  Those old VMDK’s may be misaligned!  Consider using Robocopy, RichCopy or rsync to ensure an aligned disk.
• Check your storage vendors best practices for your particular environment (OS, workload, SAN, etc.).
• There is some debate on whether or not it is advised to align your OS partitions.  There is no clear-cut answer on this as it depends so much on your environment and particular needs.  For help in deciding if you should align your Guest OS drives, see the comments in the blogs by Duncan Epping, Vaughn Stewart, and Chad Sakac.
• While working the VMware User Group booth at the Washington, DC Virtualization Forum 2010 I had a user ask me if rules and procedures for alignment on 4k sector disks are different. I forgot to research it until just now, so I honestly don’t know (please comment if you do know!).  Check with your storage vendor if this is an issue for you.
• Finally, you can’t realign partitions using tools like mbralign or vOptimizer in ESXi -Aaaron Delp explains the problem here: http://blog.aarondelp.com/2010/06/my-1-issue-with-vmware-esxi-today.html.

I hope this is helpful for you in understanding the problem of storage alignment and how it can impact your environment.  Comments or questions are welcomed!

One of my most popular posts to date had been IBM DS3300 Write Performance Problem Solved.  I am pleased to have upgrade my internal environment to an EMC Clariion CX4 array, but still have customers using the DS3300 and the MD3000i from Dell.  For that reason, I keep an eye out for news on these arrays.  You’ll recall that these arrays are based on the same LSI kit.  Truth be told, properly designed and configured, these arrays are a technically proficient and cost efficient solution for SMB’s looking to get their VMware environments going.

While reviewing my email today I found an email from SolarWinds about their new Free SAN Monitor.  I have used several products from SolarWinds in the past, and so thought I would give this a quick try.  This tool can monitor the LSI-based arrays from Dell (including the MD3000 series), IBM (DS3000, DS4000, and DS5000), and SunStorageTek 2000 and 60000.  Installation and configuration were both fast and easy.  A wizard asked me for the array’s IP address, monitoring interval, and some thresholds (which I left blank).  Once I clicked Finish the app launched on my desktop and immediately began to display data.

This screen-shot is of the app running on my Windows 7 x64 workstation.  You’ll notice that the array is reporting performance, in terms of IOPS, and response time for individual LUNS.  The health of my array is also displayed.  You’ll see that my ‘Controller 2′ shows a failure.  I do not have a dual-controller configuration (hence my wildly popular post on performance being in the tank due to a disabled write cache).  The SAN Monitor incorrectly reported that I had two controllers, and that the 2nd is failed.

This is not bad for a free tool, in my opinion.  I would love to see SolarWinds add some info on write cache status for the array (cache hit %, cache enabled per LUN, etc.) as well as LUN utilization in terms of free space remaining.  For a free desktop GUI, it’s worth the 5 minutes to download and setup.

Reporting on I/O Response time (in milliseconds) is helpful in determining if your array is stressed.  If response times are consistently above 15ms, you ought to look at adding some spindles.  If you are over 25ms on a regluar basis, you’re likely feeling some pain on the servers generating the workload against your array.

If you are into the whole free tool thing, SolarWinds also has a free VM Monitor that is capable of monitoring a single ESX server via SNMP and it’s associated VM’s.  It too runs as a desktop application, requiring minimal setup and configuration.  While it won’t monitoring your whole VMware Essentials (3 ESX servers + vCenter) environment, it’s a start.

You might also check out Nick Weaver’s vSphere Mini Monitor for basic real-time alerts and monitors, as well as NetWrix’s Change Reporter for VMware.  A final free (and nothing short of awesome) tool is Alan Renouf’s vCheck Daily Report PowerShell Script.

Do you have free tools, scripts or utilities for your VMware environment?  I’d love to hear more about them in the comments section!

It was great seeing so many VMware User Group (VMUG) members at this week’s Virtualization Forum in Washington, DC.  There are several more Virtualization Forum‘s scheduled for North America, Europe, and Asia – these events are a great chance to see live demo’s, connect with experts, and catch a peek of some exciting technologies in the virtualization space.  I hope to see everyone I met at the VMware booth at the next Washington, DC VMUG.

Our next event will be on Thursday, April 29th at the Sheraton Crystal City Hotel in Arlington, VA where we will be discussing best practices when Virtualizing Tier 1 Applications.

Here is the agenda for the meeting:

11:30 a.m.    Registration
11:50 a.m.    Opening Remarks
12:00 p.m.    Riverbed Presentation by Jim Borda
12:45 p.m.    Lunch
01:30 p.m.    What’s New by VMUG Leader Jason Langone
01:45 p.m.    Double-Take Presentation: “The Secret to Disaster Recovery for Any Application, Anywhere” by James Senecal
02:30 p.m.    Beverage Break
02:45 p.m.    Virtualization of Tier 1 Apps
03:30 p.m.    Wrap-up

This is a great opportunity to meet with your peers to discuss virtualization trends, best practices and the latest technology.  Register today to join us for this free informative event. Space is limited, so respond as soon as possible to reserve your seat.

The official registration form can be found here: http://campaign.vmware.com/usergroup/invites/WashingtonDC_4-29-10.html.