VMtoday

VMware News, Views, and How-To's from vExpert Josh Townsend

You are here: Home / Archives for best practices

best practices

Optimize Microsoft Office and Outlook in VDI Environments

by 3 Comments

I’ve been working on a VMware View Optimization and Best Practices whitepaper for Clearpath.  The document isn’t quite ready for publication yet, but I wanted to share a section that I’ve been working on as I’ve had several people ask me about this topic recently.  This section is dealing with deploying Microsoft Office, and Microsoft Outlook in particular in VMware View VDI environments.  Stay tuned for the full document.  In the meantime, I hope this bit is helpful to you.

Microsoft Office Outlook in VMware ViewMicrosoft Office in View Desktops

Microsoft Office, including Outlook, can be deployed as a ThinApp to View desktops.  The resulting ThinApp can be very large, unless individual components are packaged separately and AppLink used to link the ThinApp packages.  Because Microsoft Office is a large application, and has a regular patching schedule in line with Windows Updates ‘Patch Tuesday’, many administrators choose to thick-install office in their base image and update using a View recompose operation during regular Windows patch maintenance schedules.

Resources for creating and troubleshooting Office 2010 ThinApp packages can be found at the following links:

Outlook Cache Mode

Because of the prevalence of Microsoft Outlook running against Microsoft Exchange in corporate IT environments, special attention is given in this whitepaper on how to best configure Outlook to minimize resource consumption on VMware View desktops.  As with other recommendations, these are geared towards saving resources and improving performance, but may be at the cost of user experience.  Balance user experience and expectations with your performance and resource requirements.

Outlook Cached Mode provides offline access to an Exchange mailbox by building an offline cache of the mailbox in a local .OST file stored within a user’s profile.  Because View desktops execute in the data center they are typically always online.  The always online nature of View desktops negates the need for offline mailbox access.

Outlook Cached Mode Configuration

With stateless View desktops, the OST file may be regenerated and re-indexed at each user login to a desktop.  This can create high amounts of disk I/O, network bandwidth consumption, and compute resource utilization.

Note: View desktops configured for Local Mode offline use may benefit from offline mailbox access.  Consider this when configuring policies for Cached Mode in your environment.

What You Didn’t Know You Needed To Know Before Implementing VMware View

by Leave a Comment

Clearpath Sponsors VMUGI had the privilege of presenting at the Potomac Regional VMware User Group (VMUG) conference yesterday and at the inaugural Boston VMUG last week.  The Potomac Regional event was a combination of Washington DC, Northern Virginia, Maryland, and Federal VMUG’s.  Clearpath participated in both the Potomac and Boston events as a gold sponsor.  My session was entitled, “What You Didn’t Know You Needed To Know Before Implementing VMware View”.

My session provided attendees with an overview of how VMware View integrates with existing infrastructures, Windows images, applications, and processes.  Building on Clearpath’s experience in implementing VMware View for customers of many sizes in a variety of industries, we looked at many of the issues that can cause a View pilot or production implementation to get off track.  Prerequisites and best practices for the pieces of your environment that touch View were reviewed, including Active Directory, Distributed File System (DFS), Group Policies, Load Balancers, ThinApp packaging, and base image optimization.  Finally, we discussed how to integrate the several facets of your IT group that will have to be involved in your View roll-out: application, desktop/helpdesk, network, storage, and infrastructure teams.

The presentation is embedded below.

If you have questions, comments, or need clarification around any of the items covered in my presentation please don’t hesitate to ask!

VMware Press Releases “Managing and Optimizing VMware vSphere Deployments” For Pre-Order

by Leave a Comment

VMware Press has released Managing and Optimizing VMware vSphere Deployments by Sean Crookston & Harley Stagner for pre-order on Amazon.  The book is billed as “The Complete Guide to Planning, Deploying, and Operating  High-Performance VMware vSphere 5 Virtual Infrastructure.”  The description of the book on Amazon (copied below) sounds like this title will be well worth the price ($15 on the Kindle, $25 for the paperback).

To make the most of VMware’s vSphere 5, IT professionals need knowledge, tips, and insights they’ll  never find in the manuals–or in any book, until now! In Managing and Optimizing VMware vSphere Deployments, two world-class VMware experts offer start-to-finish lessons for vSphere planning, implementation, operation, management, and troubleshooting: expert insights drawn from their unsurpassed  “in-the-trenches” consulting experience.

Writing for experienced VMware professionals, the authors focus on high-value techniques optimized for the new vSphere 5, helping you establish frameworks that support your virtual infrastructure’s evolution for years to come. They present scenarios and examples drawn from real-world data, helping you address crucial issues ranging from sizing and performance to redundancy. The book concludes with a full  case study that walks you through a design from inception through implementation and explores the reasons for each key decision.

Coverage includes:

  • Building a strong foundation for virtual infrastructure: design considerations and  best practices
  • Smoothly integrating vSphere 5 into current environments and considerations
  • Establishing a more stable infrastructure
  • Choosing hardware and making optimal configuration decisions
  • Transforming your VMware design from  blueprint to completion
  • Operating vSphere solutions more efficiently  on a day-to-day basis
  • Automating tasks and maximizing availability
  • Streamlining the installation of updates, patches, and upgrades
  • Forecasting and planning capacity on an ongoing basis to support growth
  • Overcoming roadblocks on the journey to  100% virtualization
  • Monitoring vSphere 5 with tools provided by VMware and its community
  • Discovering the most valuable and current online VMware resources
  • Examples using Cisco Unified Computing System (UCS) are provided throughout the book

I’ve got a good queue of books lined up already, but I’ll be ordering this book for my library.

Storage Basics – Part VIII – The Difference in Consumer vs. Enterprise Class Disks and Storage Arrays; or ‘Why is the SAN you are proposing so darn expensive?’

by 5 Comments

StorageBasics7As both an IT Manager and storage & virtualization consultant, I have shepherded SMB’s through SAN purchases in support of VMware environments.  This may not hold true for the bigger IT shops, but as I take these smaller companies through the SAN purchase process, a common set of questions is often asked – Do we really need a SAN, and why is it so darn expensive!?!? I quickly learned that my (cheeky) answer “because it’s enterprise class, baby” just didn’t cut it.  So how do you move beyond “I can get a 2TB hard drive for $100 at Best Buy, why are you selling me on 300GB hard drives for $500 each, and you want me to buy how many?”

For me, the answer boils down to a handful of key factors: performance, reliability, availability, supportability, data integrity, and ease of administration.  When we’re talking business-class workloads, your 2TB USB disk  might have the ease of administration thing going for it (if you can’t plug-n-play that you probably shouldn’t be talking about storage…), but probably falls far short in the other areas.

The Disks

Disks are the basic building block of storage arrays – usually spinning disks, so we’ll start with looking at the difference between consumer-grade and enterprise grade disks.  (Note: I talked disks back in the early posts in this series if you want to do some catch up reading).  I/O activity on disks make the pretty little lights go blink blink – whether we’re talking the hard drive in my mom’s 10-year old PC or the dozens of FC disks in that newfangled EMC array in my office.  The similarities stop there.

There are substantial differences between desktop class hard drives and enterprise class hard drives.  Enterprise class drives are designed for longer duty cycles, higher throughput, and greater reliability.  Some key differences include:

  • Enterprise class drives are built with bigger, better, and faster components – from stronger actuator magnets and faster spindle speeds to increased platter counts, enterprise class drives are more capable of delivering more IOPS than desktop class drives.  This addresses the performance factor.
  • Enterprise class drives are rated for a much heavier duty cycle, capable of serving more data, more often.  This addresses the reliability and availability factors.
  • Enterprise class drives such carry longer warranties from the manufacturer and, better yet, are supported by your equipment vendor (EMC, NetApp, Dell, HP, etc.).  Supportability: check.
  • Enterprise class drives have more ‘intelligent’ firmware that is capable of identifying and recovering from corruption and other errors.  I’ve got your data integrity right here, buddy.
  • Enterprise class arrays running enterprise class drives have automatic RAID rebuild, proactive hot-sparing, and monitoring/alerting capabilities.  These all ease administrative burden – especially where the stuff hits the fan.

If you want more formal documentation, I recommend reading this whitepaper from Intel that covers some of the differences in enterprise and desktop class drives: https://download.intel.com/support/motherboards/server/sb/enterprise_class_versus_desktop_class_hard_drives_.pdf.  There are always exceptions – I’ve had brand new enterprise class drives show up dead on arrival, and my mom’s poor old 4.3 GB IDE drive is still crunching away.  Check out Google’s 2007 Hard Drive Study results if you want some real life metrics: https://labs.google.com/papers/disk_failures.pdf.  Interestingly enough, Google’s study specifically states that they are using consumer-grade hard drives….

Magnetic spinning media is not the only disk in the game – Solid State Disks’s are rapidly gaining in popularity in both consumer and enterprise applications.  Just like with traditional hard drives, the SSD space has differences between Enterprise Flash Drives (EFD) and consumer SSD’s. Multi Layer Cell (MLC) SSD’s are cheaper, slower, and more prone to errors, but are sufficient for consumer use.  Single Layer Cell SSD’s are typically more performant, reliable, and are often built with improved error-detection/correction abilities.  EFD’s are increasingly working their way into the mid-range storage appliances from several vendors.

The Array

If we think beyond the single disk to the array, the same enterprise vs. consumer class questions arise.  I’ll talk about best practices in choosing a storage array for your virtualization project in a later post, but there are a few items that compliment the consumer vs. enterprise class drive discussion that I want to present now.

First, if you’re building a bet-the-business solution, make it complete solution.  While that Drobo, Qnap, or Synology may be on the VMware HCL and is sporting more features than you can shake a stick at, those disks you got for cheap on Tiger Direct aren’t enterprise class, have a short manufacturer warranty, and won’t be supported by the array vendor when you need that support the most (usually 2am on a Saturday).  Nothing against Drobo, Qnap or Synology mind you – I’d love to have one of their NAS devices for my home lab – just suggesting that you think through your decisions and their impact on performance, reliability, availability, supportability, data integrity, and ease of administration.

Second, an array is more than a bunch of disks that go blink blink.  From advanced caching and vSphere integration, to seemingly simple things like rack mount design and vibration dampening, enterprise class storage arrays complex!  Don’t believe me?  Check out this video on shouting in the data center and the subsequent effects of the hard drive vibrations caused by the sound waves.

Extra Credit

Speaking of hard drive vibrations and reliability, there are several storage vendors getting creative.  Check out this write up on how Xiotech is innovating: https://www.virtualinsanity.com/index.php/2010/10/18/the-spinning-disks-stay-of-execution/

Need some extra ammunition to argue for that new SAN?  Check out this Hitachi Data Systems (HDS) whitepaper “Is DAS really cheaper than SAN? Storage Economic Methods help to differentiate Price and Cost“.

Bottom Line

If you are serious about building a virtualization platform or private cloud environment for your organization, choose components that are built to withstand your workload’s performance and availability requirements.

Keep Reading:

High CPU Ready, Poor Performance

by 16 Comments

**Update** January 2013: You’ve probably found your way here by doing a search for ‘High CPU Ready in VMware’ or something to that effect.  Welcome, I’m glad you’re here.  Give this article a read as it still has relevant content, then check out this newer post on CPU Ready, with some handy quick reference charts and more detailed info on the condition: https://vmtoday.com/2013/01/cpu-ready-revisted-quick-reference-charts/.  Thanks for reading! –Josh

I ran into an issue with a customer today where a VM was performing terribly.  From within the guest OS (a Windows 2003 application server running .NET in IIS which I will call BigBadServer) things appeared sluggish and CPU time was high.  The amount of time being spent on the kernel was notably high.  The VM in question had 4 vCPU’s and a good helping of memory.

high kernel time in perfmon

I don’t have access to the VMware client at this particular site – just some of the guests, so I was flying blind.  Gut feeling told me that I was dealing with a resource contention issue.  I had the VMstats provider running in the guest (https://vpivot.com/2009/09/17/using-perfmon-for-accurate-esx-performance-counters/) showed me that there was no ballooning or swapping going on, and that the vCPU’s were not limited and the CPU share value seemed to be at the default.

I strongly suspected that the physical server running VMware ESX was oversubscribed on physical CPU (pCPU) resources.  Essentially, the guest VM’s that are sharing the resources of the physical machine are demanding more resources than the machine can handle.  To verify this theory, I had the client check the ‘CPU Ready’ metric on BigBadServer and bingo!

CPU Ready is a measure of the amount of time that the guest VM is ready to run against the pCPU, but the VMware CPU Scheduler cannot find time to run the VM because other VM’s are competing for the same resources.

From the stats the customer provided on our phone call, the CPU Ready for any one of the 4 vCPU’s on the BigBadServer was on average 3723ms (min: 1269ms, max:8491ms).  (Update 8/25/2010 to clarify summation stat) The summation for the entire VM was around 12,000ms on average and peaked around 35,000.  The stats came from the real-time performance  graph/table in the vSphere client. The real-time stats in the vSphere Client update every 20 seconds, so the CPU Ready summation value should be divided by 20,000 to get a percentage of CPU ready for the 20 second time slice.  If I take the worst case scenario of 8491ms per vCPU, this VM spent nearly 43% (8491/20,000) of the 20 second time slice waiting for CPU resources.

Update: 10/10/2012: VMware published a KB article earlier this year that details the conversion of CPU ready from a summation to percentage.  Find the article here: https://kb.vmware.com/kb/2002181.

The CPU Ready summation in milliseconds counter in the vCenter Client is not always the most accurate or easy to interpret stat – to better quantify the problem it might be best to go to the ESX command line and run ESXTOP.  CPU Ready over 5% could be a sign of trouble, over 10% and there is a problem.  Running ESXTOP in batch mode and then analyzing the output using Windows Perfmon or Excel might be a good way to go on this to get a view over several hours rather than the realtime stats we were looking at.  I wrote a post a while back with more info on ESXTOP batch mode: https://vmtoday.com/2009/09/esxtop-batch-mode-windows-perfmon/

To help quantify the problem a bit more, the BigBadServer is on an ESX 4.0 server with about 10 other servers.  The physical blade has two dual-core CPU’s (AMD Opteron 2218HE’s which are not hyperthreaded).  The other VM’s on the blade have different vCPU and vMemory configurations.  3 VM’s (including BigBadServer) have 4 vCPU’s.  A couple have 2 vCPU’s, and the remainder are configured with 1 vCPU.  In ESX 4.x, the VMware console OS actually runs as a hidden VM, pegged to pCPU #1.

I generally recommend a pCPU:vCPU ration of 1:4 for mid-sized VMware deployments of single vCPU VM’s.  The blade we are running on is a 1:5 with several multi-vCPU VM’s.  The multi-vCPU’s start to skew the ratio recommendation and require some advanced design decisions.  VMware’s scheduler requires that all the vCPU’s on a VM run concurrently (even if the Guest OS is trying to execute a single thread).  Also, the VMware CPU Scheduler prefers to have all the vCPU’s from a VM run on the same pCPU.  As workloads are bounced around between pCPU’s, the benefits of CPU cache are lost.  This is one of those ‘more-is-less’ situations that you run into on virtualized environments.

What this CPU Scheduler nonsense means in this case is that the 4 vCPU’s on BigBadServer have to wait until all logical pCPU’s on the box are idle (including the one that runs ESX itself) before it can run.  If ESX can’t accomplish that (we are experiencing resource contention) it starts prioritizing workloads according to what it can best run.  It is much easier to schedule the smaller VM’s, so it tends to run those on pCPU more frequently.  The larger VM’s tend to suffer a bit more than the smaller ones.  We are competing with 2 other VM’s with 4 vCPU’s that use up all of the logical pCPU’s when they need to run, as well as with the smaller VM’s.

I suggested a few ways to fix this issue for the BigBadServer web server:

  1. Using Shares and/or Reservations on the VM.  This probably won’t work in our situation as the physical server is too over-subscribed.  We might see a slight improvement in BigBadServer (or we might not see any change), but possibly at the extreme expense of the other VM’s sharing the blade.
  2. Reduce the number of vCPU’s on BigBadServer AND the other multi-vCPU VM’s on the same physical server.  This would reduce resource contention and open up a whole bunch of scheduling options for the VMware CPU Scheduler.  This is the quickest/cheapest fix, but will not work if the VM’s really do need 4 vCPU’s.  A little workload analysis should determine which can be made smaller (the vCenter server graphs/stats should be enough for this).  For what it’s worth, by our analysis BigBadServer seems to be happier with 4 vCPU assuming we can run with a low CPU Ready on those 4.
  3. Move the BigBadServer VM to a physical ESX server with fewer multi-vCPU VM’s so there is less contention.
  4. Move the BigBadServer VM to a physical ESX server with quad-core pCPU’s (ideally two quad-cores or bigger).  This would give a lot more flexibility to the VMware CPU Scheduler and allow it to run quad-vCPU VM’s on the same pCPU for greater efficiency.
  5. Split BigBadServer into 2 smaller VM’s – The server currently runs a couple sites.  We could split them onto two servers – one for Project1 and one for Proejct2.  This configuration would take some design, testing, and time but could scale out better, give more flexibility and availability in the long run.

I’m not sure which way the customer will go on this one yet, but I feel good having armed them with enough knowledge and options to make an informed decision.

To avoid problems like this in the future, I recommend these rules of thumb:

  • Design your hosts for your guests.  Taking your Guest VM sizes into account when designing your environment and choosing physical hardware is crucial if you need bigger VM’s.
  • Don’t make your VM’s bigger than you have to.  It is always easier to add resources than take them away.  Hot Add of CPU and Memory in vSphere make adding incredibly easy.
  • Monitor your environment for CPU Ready, Swapping, and other metrics that can indicate an inefficient design.
  • Call for help when you can’t figure out what is going on (I’m happy to help!).  VMware is super powerful, but some things can be downright backwards when it comes to resource allocation on a fixed set of hardware.

If you are looking for some resources to help explain CPU Scheduling a bit more, I recommend:

(Updated 8/25/2010 to include a few additional reference links and corrected summation divided by time slice to get accurate values)