VMtoday

Archive for the ‘VMworld’ Category

I’ve heard some grumbling about the long lines popping up at VMworld with folks blaming the new first-come format for admission into the various sessions. A few folks have asked me my take on it – here’s what I’ve got:

- Yes, the lines are long but they seem to move fast. Whether that’s because people are bailing out or because the staff here is efficiently moving people into session rooms as they open up I don’t know. I suspect a bit of both.
- I anticipate lines getting shorter over the next few days as the Solutions Exchange opens up a ton of space and activities for VMworld participants.
- As people get a feel for the flow of the event, things will start moving along better. Rather than jam-packing as many sessions into their schedule, people will start to balance vendor time on the Solution Exchange floor, Hands-On Labs, and other activities.

What has your experience been so far? Any suggestions for improving the VMworld experience? I’d love to hear your ideas in the comments!

Way back in the day (we’re talking way back in high school here) I worked my summers away at a Six Flags park in Western New York.  Every fall the park hosted an International Food Festival.  Let me tell you – the perogies, sausage and gyro’s slid down like nobody’s business.  But the real prize was the Italian bakery’s cannoli’s.  The 5 or so folks that shared an office with me decided we should track our cannoli consumption.  We did (like you do a drug or spin up VM’s just cause you can) something like 126 cannoli’s in one weekend. We called it the Cannoli Count – kept a tally o a whiteboard in the office.  Sickening, right?  So, what does this have to do with virtualization, you ask.  Well, not much really, but here is where I am going with it.  T-shirts are handed out like nobody’s business at VMworld, and they accumulate like a pack-rat’s pile of newspapers in my dresser drawers (I still have a high-school wrestling t-shirt that dates back to 1994 in the rotation).  My VMware Widow wife hates them, so I figure I’ll see how many more I can collect this week.  With two already in my hands after VMworld check-in, enter the:

Josh’s VMworld T-Shirt Count

||

I’ll keep the running tally going throughout VMworld – let’s see where this ends up (besides me cleaning out my dresser to avoid sleeping on the couch when I get home).

I was fortunate enough to be offered a sneak peak at the VMworld 2010 Hands-on Labs setup this morning, and let me tell you – I am impressed.  A lot of hard work has gone into planning, architecting and deploying the Labs environment, promising to make it the most user-friendly VMworld Labs setup yet.  Here is what you need to know:

Location: The Labs will be held at Moscone West, on the corner of 5th & Howard St. This is a change from last year.

Format: There will be two types of hands-on labs – instructor led (they’re calling these Advanced Lab Tutorials) and self-paced.

• The instructor labs are more of a tutorial for those who want to be walked through the lab manual by a subject matter expert in an open discussion format.  The Advanced Lab Tutorial sessions support 250 seats.  The Advanced Lab Tutorials will be useful for preparing for the associated self-paced labs.  Take the Advanced Lab Tutorial first, then head downstairs to the Self-Paced lab.
• The self paced labs are designed with a ton of flexibility, allowing you to choose what and when you work through the material.  For an overview of the Lab topics, check out the VMworld 2010 Program Guide.
• When you arrive at the Self-Paced Labs area, you will register for the lab you want and head to a nice waiting area if no seats are available.  When your number is called, you will be lead to your seat and will fire up your lab.  You’ll have an hour to work through the lab.  If you need more time, ask.

Technical Specs: The VMware Core Team has obviously put an enormous amount of thought and time into improving the lab experience.  For those who attended VMworld 2009, the lab experience folks a bit disappointed due to some technical glitches and scheduling issues.  This year’s Labs are built with a ton of redundancy and allow for a much smoother, user-directed schedule.  The scale and scope of the labs is astonishing to say the least.  Here are some stats I gleaned on the lab setup:

• There are 30 self-paced lab topics, each demanding their own unique environment.
• There are 480 seats available for the self-paced labs, in a stadium seating configuration.  This allows a huge number of people to flow through the lab environment efficiently, with minimal wait time.  The lab schedule has some 40 hours of time for you to get in and work over the next several days.  This equates to more than 20,000 lab-seat hours (up from about 5000 hours last year).
• The labs run from one of three data centers: Miami, FL (Terremark); Ashburn, VA (Verizon); and locally in the Moscone Center.  This provides a great deal of redundancy and positions the labs as a cloud offering to fit the theme of this year’s VMworld.  The Miami and Ashburn sites have been running for a while, and will be reused for VMworld Europe next month.  This is a change from last year where the gear was fork-lifted in for the show (remember all the racks at the bottom of the escalators?).  This has given the team more time to work on the setup and iron out any problems.
• The self-paced labs are based on VMware’s Cloud Lab infrastructure, purpose built for VMworld Labs.  Cloud Lab provides a slick interface for provisioning labs to participants while doing some really smart things in the background to enhance performance and flexibility.
• It is estimated that more than 100,000 VM’s will be provisioned in Labs this week – more than 5000 VM’s built and destroyed per lab hour! <- Read that again. Astonishing, no?
• The gear driving the labs is provided by HP, Dell, EMC, NetApp, Cisco, and Xsigo.  Xangati is used for monitoring performance of PCoIP to the Wyse thin clients at each seat.
• There are 4 racks of compute power and 2 racks of storage per datacenter.
• The storage environment is mostly 10GbE.  EMC FastCache and NetApp Dedupe are both in use.  Storage is mostly NFS-based.
• The memory footprint required to run the labs is some 36TB.
• Labs are running a few levels deep – ESX nested inside of ESX with VM’s running inside.
• Host Profiles are heavily leveraged to ensure a consistent environment.
• Twin DS3′s provide Internet connectivity for the Labs.
• In true cloud fashion, the Lab Cloud product dynamically pre-populates lab environments based on demand.  As some labs rise in popularity, the Lab Cloud will stage up environments based on that demand.  This will reduce wait time for the lab environment to be readied.  In years past, students would wait 5-7 minutes for their custom lab environments to be readied (building, deploying and booting a unique Active Directory, vCenter, ESXi, nested VM’s and associated products takes some time).  No guarantees that there won’t be some wait time, but this is a huge step in the right direction.
• There will be some 150 moderators ready to help with Self-Paced labs.  Moderators are subject matter experts.  If you request help through the Lab Cloud interface, a moderator who is a SME in your topic will be dispatched to help you.

A few more things to note:

• There will be prize drawings for those who do the most labs, as well as those who complete the labs the fastest.  Prizes will include a full pass to VMworld 2011 in Las Vegas.
• Lab manuals will be made available after the show.
• Some of the labs look really cool. You can find a list in the VMworld 2010 Program Guide.  I am excited to see the VMware vSphere Sandbox lab – an everything-but-the-kitchen-sink setup of as many products as they could cram in.  This provides a playground for you to see all of the VMware products working together, where you can create, destroy and otherwise play as you wish.
• I would love to see this environment be made available for other uses after VMworld.  I think VMUG’s could really benefit, as could VMware’s partner community.

Special thanks to Adam Zipman who leads the team putting this together, Dan Anderson (Dan is the lead architect behind this massive operation) and Curtis Pope who led development of the Cloud Lab interface.  Also, thanks to John Troyer for setting up this morning’s breifing.  I appreciate your time today, guys.

I hope you all are as excited about the labs this year as I am.  I am planning to spend a good chunk of time working through the lab environments.

I have been meaning to write this up for a while; Scott Drummonds’ ‘Love Your Balloon Driver’ post today at his Virtual Performance blog gave me a nice reminder.  I actually caught a sneak peak at the graphs with an explanation from Scott at his instructor-led lab at VMworld 2009.  Scott calls out that the only workload they discovered suffers from balloon driver activity is Java.  The reason for Java’s problems with balloon driver activity is that Java itself runs in a VM and so the guest OS cannot properly determine which pages should be swapped out when the balloon driver calls for it.

My experiences causes me to agree with Scott and the whitepaper he cites – in a properly designed and equipped environment the balloon driver is not detrimental for most every workload to a point.   However, I recently discovered in a client site that the balloon driver can cause significant issues when the environment is poorly designed and under-sized.  Here the background:

I was called into an already established environment where the client was running on an older blade with VMware ESX 3.5.  The blade maxed out at 16GB RAM and had dual dual-core CPU’s with no hope for an upgrade.  On the blade was a single guest VM running Windows 2003 with SQL 2005, in it’s full 32-bit glory.  The VM was configured with 4 vCPU’s and 16GB of memory.  Some of you can probably already guess where this is going….

The x86 Windows guest had PAE configured, and SQL took advantage of AWE to use the additional memory beyond the 4GB limit of a 32-bit system.  Additionally, the Windows guest had the /3GB switch enabled in boot.ini.  Finally, as per SQL best practices, the ‘Lock Pages in Memory‘ permission was granted to the SQL Server service account.  What the guest was left with was 1GB of kernel mode memory and 15GB of User Mode/Extended addressable memory.

And here’s the problem.  The client was using ESX, not ESX 3.5, so the Service Console required memory.  In this case, the service console had approximately 512MB allocated to it.  Futhermore, VM’s require some overhead on ESX to run.  The memory overhead consumed by a Windows guest on ESX 3.5 with 4 vCPU and 16GB of memory is a bit more than 512MB.  On a properly sized ESX server with multiple similar guests/workloads, you could probably gain much of the overhead back through transparent page sharing; but in this case I had a 1:1 P2V ratio.  If you are any good at math you see that the environment is running about 1GB short of memory.  A quick check of the balloon driver stat in vCenter show that the balloon driver was constantly active and demanding about 1GB back from the guest… constantly.

Under normal circumstances this might not be an issue, but in this case the Windows guest was being absolutely punished.  The guest CPU’s were pegged at 100% with an excessive amount of kernel time, often indicating IO issues.  And indeed I did experience terrible disk and network performance on the guest.  At the root of the problem is this – the Lock Pages in Memory permission allows SQL to get a firm grasp on the user mode memory available to it (15GB) and lock it up.  This left the already starved (because of the 3GB switch in the boot.ini) guest kernel with it’s 1GB the only thing the balloon driver could really swap out.

The client suggested a reservation of 16GB on the VM, knowing that memory reservations prevent balloon driver activity.  I calmly asked them to back away from the keyboard as I explained how if a starved guest was bad, how much worse a starved Service Console would be.  In the end the fix was quiet easy – I convinced the customer that they should reduce the amount of memory allocated to the guest by about 1GB, enough to let the 512MB SC and the 512MB of overhead run without contention.  I was able to show them the difference between allocated and active memory in vCenter – the 1GB being surrendered was not really being actively used, SQL just had it locked up.  In fact, surrendering the 1GB of memory back to ESX breathed new life into the guest VM, bringing its performance back in line with expectations.

Ideally, I would have brought in a bigger ESX server that could serve additional VM’s, driving greater levels of efficiency across the environment.  It just wasn’t an option for the client in this case.  In the end, the problem was fixed and I was reminded just how fun it can be to explain some of these backwards sounding virtualization concepts to customers – fewer vCPU’s can lead to better performance of guests, less guest memory can fix performance issues, and increasing the quantity of similar guests on a host can drive better performance to a point because of transparent page sharing.

Stay tuned over the next few weeks as I digest and write on my VMworld experience – from VMUG activities to Paul Maritz’s press conference announcing the vCloud Express, and plenty of great sessions in between.  Like many of you, I returned from VMworld with quite a backlog of work but I’ll do my best to squeeze in some posts and tweets.