Time to make it official!  I have moved to Clearpath Solutions Group where I’ll take on the role of Virtualization Practice Manager, focusing on delivering VMware solutions and services to our VMware, Cisco, and EMC customers.  I’m looking forward to joining a high energy company where I can focus on the VMware technologies that I am passionate about.  I’m going to hit the ground running with VMware View implementations, Site Recovery Manager (SRM) work, and vSphere upgrades on EMC storage and Cisco UCS servers.

A few perks that I am looking forward to in this new job:

• With the changing roles, I’ll be ramping up my writing on VMtoday, as well as on Clearpath’s blog.  A consistent focus on virtualization ought to help provide fodder for my blogging.
• My commute will shorten by about 33% (new commute will be a whopping 4 miles each way on side roads – unheard of in Northern Virginia / Washington, DC!).
• More active in social media – @ClearpathSG gets it (+Clearpath Solutions Group, http://www.facebook.com/ClearpathSolutions, http://www.linkedin.com/company/100452)
• VMware Partner Exchange (PEX) – I look forward to seeing all my virtualization friends at PEX next week!
• New colleagues – there’s some great folks at Clearpath, and we’re still hiring.  Check it out and join the team!

Leaving my role as IT Manager at Tiber Creek Consulting was not an easy decision, but it is time to move from being a constantly distracted, generalist IT Manager to something with a bit more focused. Tiber Creek has been amazingly flexible and encouraging over the past three years as Stephanie Townsend and I have worked though her health issues and the stress it imposes on our family (with Stephanie having improved over the past few months after a new procedure to patch her cerebral spinal fluid (CSF) leaks, I feel some freedom to go after great new things).   Tiber Creek is a great group of people doing great work for our armed forces, and I’ll miss the great team that I have been a part of for the past three years.  Thanks for everything, Tiber Creek!

Here’s to an awesome new role, at an awesome new company.  Thumbs up – let’s do this thing!

This is a re-post from http://dcvmug.com/dc-vmug-january-17th-2011/.  Please follow the DC VMUG site for updates!

The Washington DC VMUG invites you to the first meeting of the DC VMUG in 2012. The event will be held on Tuesday, January 17th, 2011 at Nationals Park.

Our sponsors, Tintri and Veeam, will have some great give-aways for attendees. We’ll have a tour of the new Nationals Park at the end of the VMUG meeting.

Register Now!

Agenda

7:30am – 8:00am: Registration & Breakfast
8:00am – 8:45: Tintri Presentation
8:45 – 9:00am: Break
9:00 – 9:45am: Apps in the Enterprise : Horizon App Manager vs. Citrix Xenapp; Michael Letschin, vExpert, Convergence Technology Consulting
9:45 – 10:00am: Break
10:00 – 10:45am: Veeam Presentation
10:45 – 11:15am: Ask the Experts and Closing Remarks
11:30am: Nationals Park Tour

Location

Washington Nationals Park

1500 South Capitol St., SE
Washington, DC 20003

Free Parking in Lot C, or Metro to Navy Yard (Green Line) and walk 0.3 mi SW to Nationals Park

Proceed to Red Porch Restaurant

I’ve been asked several times recently to recommend training resources for VMware, so I thought I might write my responses up in a blog post to help out folks in the community who are looking for the best resources to gain VMware knowledge, prepare for their VCP and other certifications, and continue on their journey to becoming a virtualization rockstar.

I’ve picked up a bunch of certifications over the past 10 years.  For me, certification is not the means to an end, but the end of some long, intensive studying and lab work, then doing some deep dive studying and doing.  By the time I get to the test, passing should be a forgone conclusion.  I’ll save details of my lab for a future post and focus on the books and other learning resources that I use.  When getting into a new or updated technology, I start out my studying with a good overall survey of the technology I want to learn.  I like a good book that hits all of the major components, provides background information to help explain why the technology, component, or module really matters and how it fits into the big picture.  Then I get into technology specific books – deep dives, command line references, and architecture books.

Books

Mastering VMware vSphere 5, by Scott Lowe

My go-to book for VMware vSphere has been Scott Lowe’s Mastering VMware vSphere 4.  Scott’s updated book, Mastering VMware vSphere 5 started shipping yesterday.  Scott covers everything from the basics of what a hypervisor is to VMware vSphere best practices.  This is a great book to accompany lab work as it includes licensing, planning and installation, setting up virtual networking, storage basics, security, resource allocation, HA, DRS, and even some automation with the CLI and PowerCLI (PowerShell).  The book is well written, taking you methodically through vSphere, while providing plenty of helpful hints along the way.  Do yourself a favor and click the picture to the left to order it from Amazon now (paperback or Kindle format).  This book is a great way to get started with studying for your VCP certification.

VMware vSphere 5 Clustering Technical Deepdive

Once I have the basics down, I get into the deep dive work. The first deep-dive book for VMware vSphere 5 is VMware vSphere 5 Clustering Technical Deepdive, by Duncan Epping and Frank Denneman.  This is Duncan and Frank’s second book that focuses on the clustering and high availability technologies available in VMware vSphere.  Readers of Duncan and Frank’s first book, VMware vSphere 4.1 HA and DRS Technical deepdive (Volume 1), got an incredibly deep look at how to configure VMware HA and DRS.  The new vSphere 5 Clustering Technical Deepdive includes Storage DRS as well.  I’ve talked to several readers of both these books and Duncan and Frank’s blogs who have remarked that 1.) I’ve been doing it wrong all along, 2.) I totally understand how HA and DRS work after reading this, and 3.) My environment really is resilient and reliable thanks to this book.

Pearson and VMware teamed up earlier this year to create VMware Press.  There are several books coming from VMware Press, as well as other authors/publishers that are now available for pre-order from Amazon.com.  These include:

There are not many vSphere 5 specific books out yet, but many of the vSphere 4 resources are still very useful.  My library includes these:

Video Training

If you are not a big reader or you are looking for additional topics, check out TrainSignal’s VMware Training Videos.  TrainSignal offers a whole slew of courses (many taught by VMware vExperts), including:

• vSphere 5 Training
• VMware View Administration Training
• vSphere Troubleshooting
• vSphere Performance Monitoring
• vSphere Security Design
• vSphere PowerCLI.

I have a couple of TrainSignal DVD’s and found them to be good quality with deep technical content.

Blogs and BrownBags

Once you are comfortable with the material, you can start to study for your VCP.  Several bloggers have published collections of materials to help you prepare for the VCP, VCAP, and even the VCDX.  I recommend Simon Long’s collection here: http://www.simonlong.co.uk/blog/vcp-vsphere-upgrade-study-notes/ and Cody Bunch’s VCP4 Resource Page and BrownBag sessions.

Instructor Led & Certification

Finally, once you are all read up, head to a VMware Education instructor led class.  You need to take a VMware Authorized Training course to qualify to sit for the VMware Certified Professional (VCP) certification exam.  VMware also offers a nice catalog of eLearning courses.  If you want to get a discount on eLearning, Instructor-Led training, and certification exams from VMware, check out the VMUG Advantage program.

Did you miss VMworld this year, but want a chance to get a peek at the exciting new technologies on display at the show?  The VMworld 2011 Snapshot Series is your chance to catch the excitement of VMworld in a city near you.

Please join VMware for this complimentary half-day event to get a taste of VMworld 2011 and learn how VMware and our partners are enabling organizations like yours to understand the true value of cloud computing and business agility. Cloud computing is an organizations strategic weapon-not just for IT, but to enable full business transformation, and will eventually change how the business operates. Attend, participate and learn from our Subject Matter Experts on how to modernize your IT operations. Improve, simplify and save with VMware. Topics to be discussed:

• VMware Keynote Review / General Session Highlights
• Build and Own Your Cloud: vSphere 5.0, vCloud Director 1.5 vCenter SRM 5 and vShield 5
• Desktop and End User Computing Strategies

This is a unique opportunity to learn and understand more about infrastructure, applications and end user transformation that is happening today making organizations more automated, competitive and agile. Hear from our subject matter experts, see live demos as well as network with your peers. This is your VMworld Snapshot. This is your Cloud. Own it.

For the folks here in Washington, DC with me, the VMworld 2011 Snapshot will be on October 18th.

Join your fellow virtualization professionals on Thursday, September 15th from 6:00pm to 11:00pm for an independent networking event. vBeers is designed to be vendor agnostic and independent and allow IT pros to get together and network and talk without the influence of the vendors we all get pushed. Shortly after the VMworld conference we should have plenty to talk about. Old Dominion has a standard happy hour until 7 PM with discounts on food and drink along with some specials running all night. We have a central area reserved just for the group. Just one block from the Mt. Vernon Square /7th St-Convention Center metro stop, you can take the green or yellow line right to us.

Sign up here: http://dcvbeers.eventbrite.com

More info on the venue here: http://olddominionbrewhouse.com/

Here’s the SlideRocket presentation for tomorrow’s DC VMUG.  My first real time playing with VMware’s Slide Rocket. It took me a few crashes of PowerPoint to move to SlideRocket (even after I had included it in the recent VMware Acquisitions slide).  Overall a pretty good experience!

This presentation is subject to change (I’m running out of steam tonight).  If/when I do update it on SlideRocket, this embedded deck will just magically update – the magic of the cloud!

Feel free to comment with changes or suggestions for other free utilities and scripts!  I  added a Scripts & Free Tools page to http://vmtoday.com today with some embedded Google Docs spreadsheets to keep a running list of the cool VMware scripts and tools that I come across. I’ve found and promptly forgotten so many over the years – hopefully this helps me as much as it helps you in finding and saving some great community resources!

This is your last chance to register for tomorrow’s (July 12, 2011) big VMware online event, “Raising the Bar, Part V”.  During the event, VMware CEO Paul Maritz and CTO Steve Herrod will announce the next generation of VMware cloud infrastructure offerings.  The event will be broadcasted live beginning at 9:00 AM PDT

The agenda looks like this:

• 9:00-9:45 Maritz and Herrod present
• 10:00-12:00 Deep dive breakout sessions
• 10:00-12:00 Live Q&A with VMwarevExperts

While I don’t have any inside knowledge on the event, I expect new product and service offerings, along with details on recent acquisition announcements on how VMware is bringing to bear its vision for a seamless cloud infrastructure – public, private, hybrid – that is simple to manage, easy to scale, and with all the dynamism that one could imagine.   And if that weren’t enough, one lucky attendee for this online event will win a ticket to VMworld 2011.  This event promises to be a great intro to what is shaping up to be a great VMworld 2011 in Las Vegas!

The Washington DC VMUG Invites You to a Joint Event with VMware on vCenter™ Management Solutions.
Join us in the morning and learn from VMware subject matter experts how VMware vCenter™ Operations are fundamentally simplifying how IT is managed:

• Automate operations with intelligence and visibility
• Interpret workload, health and capacity metrics
• Identify and resolve workload stress and population problems
• Correlate change events and performance patterns

Stick around for the VMUG lunch and meeting:

• Networking Lunch
• VMware Updates and What’s New with VMUG
• VMworld 2011 Preview
• Community Scripts and Free Tools to Manage and Monitor Your VMware Environment. *Attendees are invited to bring your favorite scripts for monitoring, managing, and reporting on your VMware environments to share with the group.*

This is a unique opportunity to learn from best practices, see a demo and to speak with vCenter™ Operation experts directly.

Space is limited, so register today!
Wednesday, July 20, 2011
9:00 a.m. – 2:30 p.m.

Location
JW Marriott Washington DC
1331 Pennsylvania Avenue NW
Washington, DC 20004
Driving directions

Agenda
8:30 a.m. – Breakfast & Registration
9:00 a.m. - VMware Presentation: Master Your Virtualized Infrastructure
10:15 a.m. - Partner Presentation
11:00 a.m. – VMware Product Demo
11:30 a.m. – Morning Wrap up
12:15 p.m. - VMUG Lunch, Updates, and Announcements
01:00 p.m. - VMware Security and Compliance, Rob Babb, VMware

01:30 p.m. - Community Scripts & Free Tools to Manage & Monitor Your VMware Environment

Sponsored by

We’ll kick off a normal VMUG meeting after the VMware event. Please make special note – *Attendees are invited to bring your favorite scripts for monitoring, managing, and reporting on your VMware environments to share with the group.* Prizes for the best scripts shared!

This one is for all you in the DC area (I”m looking at you, DC VMUG members) looking to network with fellow VMware folks….

For all the guys out there that have talked for weeks and months about our choice of beer and bourbon, the time has come.  We will be having a vBeers Happy Hour at one of the newest and hottest places in DC, Jack Rose Dining Salon.

For those of you who don’t know it was featured in the Washingtonian Magazine this month and is just opening with one of the largest Scotch selection ever assembled from a private collector in DC.  They also should have 20+ beers on tap.

Heres the details, hope you all can make it and if we find the date I picked is really bad… don’t fret this will not be the only time we do this..

Location: Jack Rose Dining Saloon
Address: 2007 18th Street Northwest, Washington D.C., District of Columbia 20009
Date: July 13, 2011
Time: 6PM – 9PM ish

Get yourself to the vBeers website to register:  Direct link to the event here: http://www.vbeers.org/2011/06/08/vbeers-and-vbourbon-dcnovamd-weds-13th-july-2011/

As both an IT Manager and storage & virtualization consultant, I have shepherded SMB’s through SAN purchases in support of VMware environments.  This may not hold true for the bigger IT shops, but as I take these smaller companies through the SAN purchase process, a common set of questions is often asked – Do we really need a SAN, and why is it so darn expensive!?!? I quickly learned that my (cheeky) answer “because it’s enterprise class, baby” just didn’t cut it.  So how do you move beyond “I can get a 2TB hard drive for $100 at Best Buy, why are you selling me on 300GB hard drives for $500 each, and you want me to buy how many?”

For me, the answer boils down to a handful of key factors: performance, reliability, availability, supportability, data integrity, and ease of administration.  When we’re talking business-class workloads, your 2TB USB disk  might have the ease of administration thing going for it (if you can’t plug-n-play that you probably shouldn’t be talking about storage…), but probably falls far short in the other areas.

The Disks

Disks are the basic building block of storage arrays - usually spinning disks, so we’ll start with looking at the difference between consumer-grade and enterprise grade disks.  (Note: I talked disks back in the early posts in this series if you want to do some catch up reading).  I/O activity on disks make the pretty little lights go blink blink – whether we’re talking the hard drive in my mom’s 10-year old PC or the dozens of FC disks in that newfangled EMC array in my office.  The similarities stop there.

There are substantial differences between desktop class hard drives and enterprise class hard drives.  Enterprise class drives are designed for longer duty cycles, higher throughput, and greater reliability.  Some key differences include:

• Enterprise class drives are built with bigger, better, and faster components – from stronger actuator magnets and faster spindle speeds to increased platter counts, enterprise class drives are more capable of delivering more IOPS than desktop class drives.  This addresses the performance factor.
• Enterprise class drives are rated for a much heavier duty cycle, capable of serving more data, more often.  This addresses the reliability and availability factors.
• Enterprise class drives such carry longer warranties from the manufacturer and, better yet, are supported by your equipment vendor (EMC, NetApp, Dell, HP, etc.).  Supportability: check.
• Enterprise class drives have more ‘intelligent’ firmware that is capable of identifying and recovering from corruption and other errors.  I’ve got your data integrity right here, buddy.
• Enterprise class arrays running enterprise class drives have automatic RAID rebuild, proactive hot-sparing, and monitoring/alerting capabilities.  These all ease administrative burden – especially where the stuff hits the fan.

If you want more formal documentation, I recommend reading this whitepaper from Intel that covers some of the differences in enterprise and desktop class drives: http://download.intel.com/support/motherboards/server/sb/enterprise_class_versus_desktop_class_hard_drives_.pdf.  There are always exceptions – I’ve had brand new enterprise class drives show up dead on arrival, and my mom’s poor old 4.3 GB IDE drive is still crunching away.  Check out Google’s 2007 Hard Drive Study results if you want some real life metrics: http://labs.google.com/papers/disk_failures.pdf.  Interestingly enough, Google’s study specifically states that they are using consumer-grade hard drives….

Magnetic spinning media is not the only disk in the game – Solid State Disks’s are rapidly gaining in popularity in both consumer and enterprise applications.  Just like with traditional hard drives, the SSD space has differences between Enterprise Flash Drives (EFD) and consumer SSD’s. Multi Layer Cell (MLC) SSD’s are cheaper, slower, and more prone to errors, but are sufficient for consumer use.  Single Layer Cell SSD’s are typically more performant, reliable, and are often built with improved error-detection/correction abilities.  EFD’s are increasingly working their way into the mid-range storage appliances from several vendors.

The Array

If we think beyond the single disk to the array, the same enterprise vs. consumer class questions arise.  I’ll talk about best practices in choosing a storage array for your virtualization project in a later post, but there are a few items that compliment the consumer vs. enterprise class drive discussion that I want to present now.

First, if you’re building a bet-the-business solution, make it complete solution.  While that Drobo, Qnap, or Synology may be on the VMware HCL and is sporting more features than you can shake a stick at, those disks you got for cheap on Tiger Direct aren’t enterprise class, have a short manufacturer warranty, and won’t be supported by the array vendor when you need that support the most (usually 2am on a Saturday).  Nothing against Drobo, Qnap or Synology mind you – I’d love to have one of their NAS devices for my home lab – just suggesting that you think through your decisions and their impact on performance, reliability, availability, supportability, data integrity, and ease of administration.

Second, an array is more than a bunch of disks that go blink blink.  From advanced caching and vSphere integration, to seemingly simple things like rack mount design and vibration dampening, enterprise class storage arrays complex!  Don’t believe me?  Check out this video on shouting in the data center and the subsequent effects of the hard drive vibrations caused by the sound waves.

httpv://www.youtube.com/watch?v=tDacjrSCeq4

Extra Credit

Speaking of hard drive vibrations and reliability, there are several storage vendors getting creative.  Check out this write up on how Xiotech is innovating: http://www.virtualinsanity.com/index.php/2010/10/18/the-spinning-disks-stay-of-execution/

Need some extra ammunition to argue for that new SAN?  Check out this Hitachi Data Systems (HDS) whitepaper “Is DAS really cheaper than SAN? Storage Economic Methods help to differentiate Price and Cost“.

Bottom Line

If you serious about building a virtualization platform or private cloud environment for your organization, choose components that are built to withstand your workload’s performance and availability requirements.

VMware has been investing in the VMware User Group (VMUG) program over the past year, adding new benefits and opportunities to an already great program.  In addition to local meetings where you can interact with fellow VMware professionals, the VMUG program offers other great benefits – the latest is the VMUG Advantage program. The VMUG Advantage program offers the following benefits:

• $100 Discount to VMworld
• 20% Discount on VMware Instructor Led Training
• 20% Discount on VMware Certification
• 1-Year Free Subscription: All Access eLearning Courses ($750 Value)
• 30% Discount on Workstation and Fusion Software html Licenses

The VMUG Advantage program has an introductory price of $170 for individuals (with corporate/multi-user discounts available) until May 21.  Not a bad deal if you plan to attend VMworld in Las Vegas (Aug 29 – Sept 1), complete a certification (VCP, VCAP, VCDX, VCA-DT), or take some instructor led training through VMware.  Heck – the eLearning subscription alone more than pays for program subscription cost!

As a VMUG member be sure to login to the new www.myvmug.org and update your profile or add an picture or avatar. Over the next month you’ll see exciting new benefits added to the website, including VMUG Community Pages, Event Listings, and Education Resources.

If you’re not already a VMUG member, go ahead and sign up at www.myvmug.org to learn about local/regional events and to get access to the other great benefits like the VMUG Advantage.

My last post described a problem I experienced with VMware HA after upgrading to vSphere 4.1.  Here is my experience with a similar issue after applying the ESXi410-201010401-SG patch to one of my test/dev ESXi clusters.  The patch, released on November 15th and weighing in at a hefty 212MB, fixes a number of issues from Likewise authentication on ESXi hosts to allowing configurable NOOP timout and interval values for faster failover of certain iSCSI arrays (like the DS3300 or MD3000i).

The environment where this problem occured has a single vCenter server managing both a production cluster and the test/dev cluster.  After applying this particular update to the ESXi hosts in the cluster, the vCenter server began to crash every 5 minutes or so.  The crash was logged on the vCenter server with Event ID 7031: The VMware VirtualCenter Server service terminated unexpectedly.  My go-to troubleshooting question (“What changed?”) pointed at the ESXi patch, but a VMware KB search and a little Google action yielded no results directly related to ESXi410-201010401-SG and the vCenter Server service terminating unexpectedly.  VMware KB article 1003926 provides some basic troubleshooting steps for vCenter Server, such as checking for port conflicts, vCenter DB health & availability, and log locations.  The environment was healthy until the patch was applied to a sub-set of my ESXi hosts so I could confidently eliminate credentials, port conflicts and the like as the cause of the problem, so I jumped right to the log files for vCenter.  The vpxd-*.log is found in “C:\ProgramData\VMware\VMware VirtualCenter\Logs” on Windows 2008 vCenter servers and “%ALLUSERSPROFILE%\VMware\VMware VirtualCenter\Logs\vpxd.log” on Windows 2003 servers.  I found a few lines of interest in the log file but decided I had better call VMware Support to further analyze the issue.

To make a long story short, what the logs revealed is a bug that is triggered whenever VMware Distributed Resource Scheduler (DRS) ran on the updated test/dev cluster.  Disabling DRS stopped the symptom of the vCenter Server Service terminating unexpectedly, but this was obviously not a long-term solution.  A bit more digging by my VMware support rep led to VMware Distributed Power Management (DPM) being enabled on the cluster as the root cause of the issue.  Disabling DPM but leaving DRS enabled on the cluster fixed the glitch.  I can live without DPM, but DRS is pretty darn handy.

At this point, VMware engineering knows about the issue, and a fix is planned for vCenter 4.1 Update 1.  Interesting that DPM was fingered in this case, as well as in the case I wrote about last week where HA and DPM apparently do not always play well together.  It seems like DPM is not fully baked, even though it is now officially supported.  This is unfortunate as DPM is promising to me – I can imagine the technology behind DPM being used for intelligent load shedding during peak electrical cost hours, power outages, or cooling outages in datacenters with some good integration between a DPM API and environmental management and monitoring systems like APC’s NetBotz.  Anyone else using DPM without having problems?  Any ideas for extending DPM or leveraging it for other purposes in the datacenter – I’d love to hear ideas in the comments.

I recently ran into an issue with one of my vSphere clusters after upgrading from vSphere 4.0 to vSphere 4.1 (with ESXi 4.1 and vCenter 4.1).  After the upgrade, I attempted to enable VMware High Availability (HA) on the upgraded cluster.  Each of the ESXi hosts in the cluster appeared to have been properly configured for HA (as observed in the ‘Recent Tasks’ pane of the vSphere Client).  Despite having appeared to configure HA correctly, I found that each host in the cluster was displaying an error on the Summary tab of the vSphere Client that read ‘Error <date> <time> HA agent on <host> in cluster <clustername> in <datacenter> has an error: Error while running health check script’.

I’ve dealt with HA errors in the past, so I quickly jumped into my standard troubleshooting and quick-fixes proceedure:

1. Verify host connectivity.
2. Right-click on each host and choose ‘Reconfigure for VMware HA’
3. Disable & Re-enable HA on the cluster.
4. Disable HA, place hosts into Maintenance Mode & Reboot (one at a time).  Re-enable HA.
5. Get frustrated that a quick fix is not probably not in my future….
6. Verify host name resolution for each host in the cluster from the service console/tech support mode of each host.
7. Review log files on vCenter Server and each host for glaring issues.  All Greek to me in this case….
8. Call VMware Support.

VMware Support reviewed the log files I had attached to my Service Request (SR) when I opened the case and had me try a few different things to fix the issue.  First, we verified the steps I had taken and collected some fresh logs.  Next, the support rep had me verify that Distributed Power Management (DPM) was not enabled on the cluster as there apparently is a known issue (although a KB is not available at this time) with configuring HA when DPM is enabled under certain circumstances.  I did not have DPM enabled on this particular cluster so I didn’t spend time chasing down this particular bug.

Finally, the following proceedure, run on each ESXi server in the cluster, resolved the issue (Note – this procedure is safe to do during normal operations as it does not affect running VM’s):

1. Verify SSH or Console access to the host (this requires enabling Remote SSH/Tech Support Mode on ESXi hosts on the Configuration tab | Security Profile node of the vSphere Client, or by pressing F2 to login to ESXi 4.1 | troubleshooting options | enable remote SSH.
2. Disable HA on the affected cluster.
3. Right-click | Disconnect each host in the cluster from the ‘Hosts & Clusters’ view of the vSphere Client.
4. SSH to the host and run the following commands:

services.sh stop
/opt/vmware/uninstallers/VMware-vpxa-uninstall.sh
/opt/vmware/uninstallers/VMware-aam-ha-uninstall.sh
services.sh start

5. In the vSphere Client, right-click on each host and Connect.
6. Enable HA on the cluster.

This procedure cleanly removes the VMware vCenter agent and the VMware HA agent from the ESX or ESXi host.  Reconnecting the host to vCenter pushes the vCenter management agent back to the host and installs it cleanly.  Enabling HA on the cluster re-installs the HA agent.  After completing these steps I had no further issues with HA on the cluster – case closed.  I hope this is helpful for anyone else who might be experiencing HA errors after upgrading to vSphere 4.1.

For those wanting to learn HA best practices or go a bit deeper into the inner workings of VMware HA, I highly recommend Duncan Epping’s VMware HA Deep Dive article and/or VMware vSphere 4.1 HA and DRS Technical Deepdive (Volume 1) book.

Time:
9:30am – 2:30pm

Location:
Fair Lakes Hyatt
12777 Fair Lakes Circle
Ballroom
Fairfax, VA 22033

Directions

I’ve heard some grumbling about the long lines popping up at VMworld with folks blaming the new first-come format for admission into the various sessions. A few folks have asked me my take on it – here’s what I’ve got:

- Yes, the lines are long but they seem to move fast. Whether that’s because people are bailing out or because the staff here is efficiently moving people into session rooms as they open up I don’t know. I suspect a bit of both.
- I anticipate lines getting shorter over the next few days as the Solutions Exchange opens up a ton of space and activities for VMworld participants.
- As people get a feel for the flow of the event, things will start moving along better. Rather than jam-packing as many sessions into their schedule, people will start to balance vendor time on the Solution Exchange floor, Hands-On Labs, and other activities.

What has your experience been so far? Any suggestions for improving the VMworld experience? I’d love to hear your ideas in the comments!

Way back in the day (we’re talking way back in high school here) I worked my summers away at a Six Flags park in Western New York.  Every fall the park hosted an International Food Festival.  Let me tell you – the perogies, sausage and gyro’s slid down like nobody’s business.  But the real prize was the Italian bakery’s cannoli’s.  The 5 or so folks that shared an office with me decided we should track our cannoli consumption.  We did (like you do a drug or spin up VM’s just cause you can) something like 126 cannoli’s in one weekend. We called it the Cannoli Count – kept a tally o a whiteboard in the office.  Sickening, right?  So, what does this have to do with virtualization, you ask.  Well, not much really, but here is where I am going with it.  T-shirts are handed out like nobody’s business at VMworld, and they accumulate like a pack-rat’s pile of newspapers in my dresser drawers (I still have a high-school wrestling t-shirt that dates back to 1994 in the rotation).  My VMware Widow wife hates them, so I figure I’ll see how many more I can collect this week.  With two already in my hands after VMworld check-in, enter the:

Josh’s VMworld T-Shirt Count

||

I’ll keep the running tally going throughout VMworld – let’s see where this ends up (besides me cleaning out my dresser to avoid sleeping on the couch when I get home).

I was fortunate enough to be offered a sneak peak at the VMworld 2010 Hands-on Labs setup this morning, and let me tell you – I am impressed.  A lot of hard work has gone into planning, architecting and deploying the Labs environment, promising to make it the most user-friendly VMworld Labs setup yet.  Here is what you need to know:

Location: The Labs will be held at Moscone West, on the corner of 5th & Howard St. This is a change from last year.

Format: There will be two types of hands-on labs – instructor led (they’re calling these Advanced Lab Tutorials) and self-paced.

• The instructor labs are more of a tutorial for those who want to be walked through the lab manual by a subject matter expert in an open discussion format.  The Advanced Lab Tutorial sessions support 250 seats.  The Advanced Lab Tutorials will be useful for preparing for the associated self-paced labs.  Take the Advanced Lab Tutorial first, then head downstairs to the Self-Paced lab.
• The self paced labs are designed with a ton of flexibility, allowing you to choose what and when you work through the material.  For an overview of the Lab topics, check out the VMworld 2010 Program Guide.
• When you arrive at the Self-Paced Labs area, you will register for the lab you want and head to a nice waiting area if no seats are available.  When your number is called, you will be lead to your seat and will fire up your lab.  You’ll have an hour to work through the lab.  If you need more time, ask.

Technical Specs: The VMware Core Team has obviously put an enormous amount of thought and time into improving the lab experience.  For those who attended VMworld 2009, the lab experience folks a bit disappointed due to some technical glitches and scheduling issues.  This year’s Labs are built with a ton of redundancy and allow for a much smoother, user-directed schedule.  The scale and scope of the labs is astonishing to say the least.  Here are some stats I gleaned on the lab setup:

• There are 30 self-paced lab topics, each demanding their own unique environment.
• There are 480 seats available for the self-paced labs, in a stadium seating configuration.  This allows a huge number of people to flow through the lab environment efficiently, with minimal wait time.  The lab schedule has some 40 hours of time for you to get in and work over the next several days.  This equates to more than 20,000 lab-seat hours (up from about 5000 hours last year).
• The labs run from one of three data centers: Miami, FL (Terremark); Ashburn, VA (Verizon); and locally in the Moscone Center.  This provides a great deal of redundancy and positions the labs as a cloud offering to fit the theme of this year’s VMworld.  The Miami and Ashburn sites have been running for a while, and will be reused for VMworld Europe next month.  This is a change from last year where the gear was fork-lifted in for the show (remember all the racks at the bottom of the escalators?).  This has given the team more time to work on the setup and iron out any problems.
• The self-paced labs are based on VMware’s Cloud Lab infrastructure, purpose built for VMworld Labs.  Cloud Lab provides a slick interface for provisioning labs to participants while doing some really smart things in the background to enhance performance and flexibility.
• It is estimated that more than 100,000 VM’s will be provisioned in Labs this week – more than 5000 VM’s built and destroyed per lab hour! <- Read that again. Astonishing, no?
• The gear driving the labs is provided by HP, Dell, EMC, NetApp, Cisco, and Xsigo.  Xangati is used for monitoring performance of PCoIP to the Wyse thin clients at each seat.
• There are 4 racks of compute power and 2 racks of storage per datacenter.
• The storage environment is mostly 10GbE.  EMC FastCache and NetApp Dedupe are both in use.  Storage is mostly NFS-based.
• The memory footprint required to run the labs is some 36TB.
• Labs are running a few levels deep – ESX nested inside of ESX with VM’s running inside.
• Host Profiles are heavily leveraged to ensure a consistent environment.
• Twin DS3′s provide Internet connectivity for the Labs.
• In true cloud fashion, the Lab Cloud product dynamically pre-populates lab environments based on demand.  As some labs rise in popularity, the Lab Cloud will stage up environments based on that demand.  This will reduce wait time for the lab environment to be readied.  In years past, students would wait 5-7 minutes for their custom lab environments to be readied (building, deploying and booting a unique Active Directory, vCenter, ESXi, nested VM’s and associated products takes some time).  No guarantees that there won’t be some wait time, but this is a huge step in the right direction.
• There will be some 150 moderators ready to help with Self-Paced labs.  Moderators are subject matter experts.  If you request help through the Lab Cloud interface, a moderator who is a SME in your topic will be dispatched to help you.

A few more things to note:

• There will be prize drawings for those who do the most labs, as well as those who complete the labs the fastest.  Prizes will include a full pass to VMworld 2011 in Las Vegas.
• Lab manuals will be made available after the show.
• Some of the labs look really cool. You can find a list in the VMworld 2010 Program Guide.  I am excited to see the VMware vSphere Sandbox lab – an everything-but-the-kitchen-sink setup of as many products as they could cram in.  This provides a playground for you to see all of the VMware products working together, where you can create, destroy and otherwise play as you wish.
• I would love to see this environment be made available for other uses after VMworld.  I think VMUG’s could really benefit, as could VMware’s partner community.

Special thanks to Adam Zipman who leads the team putting this together, Dan Anderson (Dan is the lead architect behind this massive operation) and Curtis Pope who led development of the Cloud Lab interface.  Also, thanks to John Troyer for setting up this morning’s breifing.  I appreciate your time today, guys.

I hope you all are as excited about the labs this year as I am.  I am planning to spend a good chunk of time working through the lab environments.

I ran into an issue with a customer today where a VM was performing terribly.  From within the guest OS (a Windows 2003 application server running .NET in IIS which I will call BigBadServer) things appeared sluggish and CPU time was high.  The amount of time being spent on the kernel was notably high.  The VM in question had 4 vCPU’s and a good helping of memory.

I don’t have access to the VMware client at this particular site – just some of the guests, so I was flying blind.  Gut feeling told me that I was dealing with a resource contention issue.  I had the VMstats provider running in the guest (http://vpivot.com/2009/09/17/using-perfmon-for-accurate-esx-performance-counters/) showed me that there was no ballooning or swapping going on, and that the vCPU’s were not limited and the CPU share value seemed to be at the default.

I strongly suspected that the physical server running VMware ESX was oversubscribed on physical CPU (pCPU) resources.  Essentially, the guest VM’s that are sharing the resources of the physical machine are demanding more resources than the machine can handle.  To verify this theory, I had the client check the ‘CPU Ready’ metric on BigBadServer and bingo!

CPU Ready is a measure of the amount of time that the guest VM is ready to run against the pCPU, but the VMware CPU Scheduler cannot find time to run the VM because other VM’s are competing for the same resources.

From the stats the customer provided on our phone call, the CPU Ready for any one of the 4 vCPU’s on the BigBadServer was on average 3723ms (min: 1269ms, max:8491ms).  (Update 8/25/2010 to clarify summation stat) The summation for the entire VM was around 12,000ms on average and peaked around 35,000.  The stats came from the real-time performance  graph/table in the vSphere client. The real-time stats in the vSphere Client update every 20 seconds, so the CPU Ready summation value should be divided by 20,000 to get a percentage of CPU ready for the 20 second time slice.  If I take the worst case scenario of 8491ms per vCPU, this VM spent nearly 43% (8491/20,000) of the 20 second time slice waiting for CPU resources.

The CPU Ready summation in milliseconds counter in the vCenter Client is not always the most accurate or easy to interpret stat – to better quantify the problem it might be best to go to the ESX command line and run ESXTOP.  CPU Ready over 5% could be a sign of trouble, over 10% and there is a problem.  Running ESXTOP in batch mode and then analyzing the output using Windows Perfmon or Excel might be a good way to go on this to get a view over several hours rather than the realtime stats we were looking at.  I wrote a post a while back with more info on ESXTOP batch mode: http://vmtoday.com/2009/09/esxtop-batch-mode-windows-perfmon/

To help quantify the problem a bit more, the BigBadServer is on an ESX 4.0 server with about 10 other servers.  The physical blade has two dual-core CPU’s (AMD Opteron 2218HE’s which are not hyperthreaded).  The other VM’s on the blade have different vCPU and vMemory configurations.  3 VM’s (including BigBadServer) have 4 vCPU’s.  A couple have 2 vCPU’s, and the remainder are configured with 1 vCPU.  In ESX 4.x, the VMware console OS actually runs as a hidden VM, pegged to pCPU #1.

I generally recommend a pCPU:vCPU ration of 1:4 for mid-sized VMware deployments of single vCPU VM’s.  The blade we are running on is a 1:5 with several multi-vCPU VM’s.  The multi-vCPU’s start to skew the ratio recommendation and require some advanced design decisions.  VMware’s scheduler requires that all the vCPU’s on a VM run concurrently (even if the Guest OS is trying to execute a single thread).  Also, the VMware CPU Scheduler prefers to have all the vCPU’s from a VM run on the same pCPU.  As workloads are bounced around between pCPU’s, the benefits of CPU cache are lost.  This is one of those ‘more-is-less’ situations that you run into on virtualized environments.

What this CPU Scheduler nonsense means in this case is that the 4 vCPU’s on BigBadServer have to wait until all logical pCPU’s on the box are idle (including the one that runs ESX itself) before it can run.  If ESX can’t accomplish that (we are experiencing resource contention) it starts prioritizing workloads according to what it can best run.  It is much easier to schedule the smaller VM’s, so it tends to run those on pCPU more frequently.  The larger VM’s tend to suffer a bit more than the smaller ones.  We are competing with 2 other VM’s with 4 vCPU’s that use up all of the logical pCPU’s when they need to run, as well as with the smaller VM’s.

I suggested a few ways to fix this issue for the BigBadServer web server:

1. Using Shares and/or Reservations on the VM.  This probably won’t work in our situation as the physical server is too over-subscribed.  We might see a slight improvement in BigBadServer (or we might not see any change), but possibly at the extreme expense of the other VM’s sharing the blade.
2. Reduce the number of vCPU’s on BigBadServer AND the other multi-vCPU VM’s on the same physical server.  This would reduce resource contention and open up a whole bunch of scheduling options for the VMware CPU Scheduler.  This is the quickest/cheapest fix, but will not work if the VM’s really do need 4 vCPU’s.  A little workload analysis should determine which can be made smaller (the vCenter server graphs/stats should be enough for this).  For what it’s worth, by our analysis BigBadServer seems to be happier with 4 vCPU assuming we can run with a low CPU Ready on those 4.
3. Move the BigBadServer VM to a physical ESX server with fewer multi-vCPU VM’s so there is less contention.
4. Move the BigBadServer VM to a physical ESX server with quad-core pCPU’s (ideally two quad-cores or bigger).  This would give a lot more flexibility to the VMware CPU Scheduler and allow it to run quad-vCPU VM’s on the same pCPU for greater efficiency.
5. Split BigBadServer into 2 smaller VM’s – The server currently runs a couple sites.  We could split them onto two servers – one for Project1 and one for Proejct2.  This configuration would take some design, testing, and time but could scale out better, give more flexibility and availability in the long run.

I’m not sure which way the customer will go on this one yet, but I feel good having armed them with enough knowledge and options to make an informed decision.

To avoid problems like this in the future, I recommend these rules of thumb:

• Design your hosts for your guests.  Taking your Guest VM sizes into account when designing your environment and choosing physical hardware is crucial if you need bigger VM’s.
• Don’t make your VM’s bigger than you have to.  It is always easier to add resources than take them away.  Hot Add of CPU and Memory in vSphere make adding incredibly easy.
• Monitor your environment for CPU Ready, Swapping, and other metrics that can indicate an inefficient design.
• Call for help when you can’t figure out what is going on (I’m happy to help!).  VMware is super powerful, but some things can be downright backwards when it comes to resource allocation on a fixed set of hardware.

If you are looking for some resources to help explain CPU Scheduling a bit more, I recommend:

• VMware’s Official documentation of CPU Scheduler in vSphere 4.1 – http://www.vmware.com/files/pdf/techpaper/VMW_vSphere41_cpu_schedule_ESX.pdf.
• A nice summary of co-scheduling from VMware’s Performance Blog: http://blogs.vmware.com/performance/2008/06/esx-scheduler-s.html
• Description and stats on Ready Time metrics for VI3: http://www.vmware.com/pdf/esx3_ready_time.pdf
• Understanding Virtual Center Performance Statistics: http://communities.vmware.com/docs/DOC-5230.pdf

(Updated 8/25/2010 to include a few additional reference links and corrected summation divided by time slice to get accurate values)

I’ve held off on posting this as I don’t like to toot my own vuvuzela too loudly, but in a fit of shameless self promotion and a desire to see the program grow, here goes nothin:

I am a VMware vExpert!

VMware notifed recipients of the designation last month.  The 300 or so awardees are recognized for their “advocacy of VMware solutions, your contributions to the community of VMware users, and your willingness to share your expertise with others.”  I find it an absolute honor to be included amongst the vExpert class of 2010 – a group full of folks who are on the cutting edge of virtualization and cloud computing, many of whom have helped me through their books, blogs, and contributions to the VMware Community.  Arnim van Lieshout has compiled a nice list of vExperts here: http://www.van-lieshout.com/vexpert-2010/.

It’s an exciting time to be a virtualization professional – in terms of the technology, people and seemingly endless opportunities.  I look forward to continuing my contributions to the community through this here blog, my leadership of the Washington, DC Metro Area VMware User Group, and through my work as a implementer of VMware solutions!  Special thanks to John Troyer for sponsoring the program and doing so much to organize the VMware community through the weekly VMware Community Roundtable, Social Media and the many other programs he touches.

I upgraded my in-house VMware vSphere environment to 4.0 Update 2 last week.  After upgrading my vSphere Client to the Update 2 version I was greeted with a series of 7 ugly error messages stating: Method not found: ‘VMware.CustomControls.LabelEx VpxClient.Common.Util.Helper.AddLabel(System.Windows.Froms.Control,Int32, int32, System.String, System.Drawing.FontStyle, Boolean)”.I assumed a plug-in had caused the error message.  I started my troubleshooting by disabling the 3rd-party plug-ins in the environment, beginning with the free EMC Storage Viewer.  Upon disabling the EMC Storage Viewer 2.x plug-in the problem was resolved.  I went out to EMC PowerLink to see if an update was available for the plug-in and was surprised to find that I had missed a major update/rebranding of the plug-in.  EMC now calls the plug-in the ‘EMC Virtual Storage Integrator’.  A hotfix was released on July 2nd to bring Update 2 support to the plug-in (version 3.0.0.32).

I updated the Solution Enabler installation (I installed Solution Enabler on my vCenter server, but it is also available as a SUSE based virtual appliance), and then updated the plug-in.

The update appeared to install without any problems.  The vSphere Client launched like a champ after the update – no errors, but no EMC Storage plug-in either.  Odd.  The problem occurred on both my vCenter Server’s vSphere client and my workstation, so seems to be more than an isolated issue.

I did an uninstall of the plug-in using Add/Remove Programs, and then reinstalled it.  After the reinstall the EMC Storage plug-in icon appeared in my vSphere Client as pictured below.

The EMC Virtual Storage Integrator (VSI) plug-in is free – installing it is a no-brainer for anyone running Clariion or Celerra storage arrays.  The VSI simplifies the job of mapping vSphere Datastores to LUN’s and NFS shares on your EMC storage, and helps pinpoint the location of VM’s and RDM’s on your array.  This visibility for the VMware administrator into the storage layer can go a long way in helping troubleshoot storage performance issues and simplifying communication between server, storage, and virtulization teams.

EMC actually offers three different free vSphere plug-ins, including the VSI.  The EMC Unified Block plug-in and the EMC Unified NAS plug-in round out the trio.  EMC’s Virtual Geek, Chad Sakac, covers all three in his blog post here: http://virtualgeek.typepad.com/virtual_geek/2010/06/update-on-emc-vsphere-plugins.html.