Now that I’ve settled in at Clearpath Solutions Group, I’ll be writing occasional blog posts on the company blog site.  My first article went live today – a fancy little piece on using Microsoft Key Management Services with VMware View to activate Windows 7.  Here’s a quick excerpt:

When deploying or recomposing a pool of Windows 7 (or Windows Vista) virtual desktops on VMware View, you may receive an error message stating: “composer agent initialization error (16): Failed to activate software license (waited 0 seconds).”  This error occurs when you use a Multiple Activation Key (MAK) to activate the parent VM.  View Composer QuickPrep, by default, uses Microsoft Key Management Services (KMS).  VMware KB Article 1026556 provides workarounds for activating MAK clients, or bypassing activation by QuickPrep so you can use a trial key.  The workarounds are appropriate for test/pilot cases, but will eventually cause problems such as MAK key exhaustion.  The proper way to license and activate a deployment of Windows 7 on VMware View is to use a KMS.

Read the rest of my post on how to setup and use a KMS for VMware View on the Clearpath Blog: http://clearpathsg.com/using-kms-vmware-view-windows-activation.

I recently ran into an issue when installing my first Windows Server 2008 R2 virtual machine.  The VM would hang/freeze randomly when used through the VMware vCenter Client’s console.  It turns out this is a known issue (see this VMware KB Article) with the SVGA driver that is installed as part of the default installation of VMware Tools.  While the article does not explain why you should disable the SVGA driver, it’s advice is correct if you want to avoid problems in your guest VM.  To correct my problem, I removed the SVGA driver from the Windows Device Manager and rebooted.  If you are having problems removing the SVGA driver before the VM hangs, use Remote Desktop to access the guest machine to perform the driver uninstall.  I have not observed hanging/freezing in the VM since removing the SVGA driver from my Windows 2008 R2 guest.  Note that this same issue is present in Windows 7.

I recently completed a VMware VI 3.5 to vSphere upgrade in a small environment (5 hosts, 80 VM’s).  Being a small environment, the upgrade was planned for one big overnight blitz.  Unfortunately, the size of the environment did not afford a test environment to uncover potential issues before the upgrade.  The upgrade to vSphere itself went swimmingly (the vCenter server had been upgraded a couple weeks earlier).  However, some things in the environment started to go wonky once the upgrade was complete.  Specifically, name resolution (DNS), DHCP, WINS, Group Policy, and really anything Microsoft Active Directory related just did not work.

Let me explain a bit about the environment so you can better understand what the problem was and how it was corrected. The environment was an all Microsoft shop, except for VMware of course. The company follows a virtualize-first policy and is about 90% virtualized, including the Active Directory Domain Controllers. The DC’s are Windows 2008 and serve up DHCP, DNS, and WINS in addition to their Directory Services roles.

The problems really began after I upgraded the virtual hardware version from v4 to v7 (check out page 97 of the vSphere Upgrade Guide for the upgrade procedure).  When a Windows server is upgrade from VMware Hardware Version 4 to 7, the VMware Upgrade Helper Service handles the reconfiguration of network adapters on the upgraded virtual machine.  The VMware Upgrade Helper Service is installed with VMware Tools and is one of the reasons, along with getting drivers installed for the new hardware, for upgrading VMware Tools before upgrading the hardware version.  If you review the Event Viewer Application log on an upgraded machine you will see several entries from VMUpgradeHelper (Source) with several different Event ID’s (26, 280, 272, 108, & 105).  An examination of these events will show that the VMware Upgrade Helper service 1.) Backed up the network configuration at OS shutdown, 2.) Started Automatically with the OS, 3.) Checks the device ID for the network adapter, 4.) If the device ID has changed (as a result of a hardware upgrade), the backed up configuration is restored and Event ID 269 is logged.

This behavior should be transparent for most configurations, with the exception of a slightly longer boot time following the upgrade.  However, I did notice a few problems with the NIC settings being restored under certain conditions.  First, on servers with a statically configured IPv4 stack, IP addresses and DNS server addresses were restored, but the WINS server addresses were not restored.  I suspect this is an oversight in the VMware Upgrade Helper service, but is probably not a major issue for many servers/environments as WINS is infrequently used.  However, for a WINS server itself to lose its configuration to use itself as a WINS server, bad things happen.  There are several ways to correct this – scripts, DHCP Options, etc.  In the end, this wasn’t really a show stopper for me in this small environment.

The second, and bigger issue for me, was that after the virtual hardware was upgraded and the VMware Upgrade Helper Service did it’s job my Active Directory and related services were not available.  DNS was not functioning, DHCP was not handing out addresses, and I couldn’t connect to AD using ADUC, GPMC or LDAP.  It took me a few minutes to figure out what was going on.  This seems to be what happened: the virtual hardware upgrade caused a new virtual network adapter to be installed in the VM and all of the settings, including the MAC, address to be restored.  The HW v4 NIC was removed from the machine, but Windows held onto the device as a ‘ghost NIC’ in Device Manager.  The core AD services, including DNS and DHCP, were still attempting to bind to the ghost NIC.  This behavior persisted through service restarts and reboots of the guest.  It wasn’t until I examined the IP configuration on the new NIC and clicked Apply (instead of canceling out) that I was prompted with a message indicating that there was more than one network interface configured with the same IP address, queuing me into the solution.

The error message should be familiar to anyone who has performed a Physical-to-Virtual migration (P2V) and is easily corrected by removing the old device through Windows Device Manager.  The device is hidden so you first have to expose it before deleting it.  Check http://support.microsoft.com/kb/315539 for details or simply follow my instructions below.  To expose the non-present NIC, open a command prompt and enter:

set devmgr_show_nonpresent_devices=1

You can then open Device Manager (enter devmgmt.msc at the command prompt to save some time).  In Device Manager, click View | Show Hidden Devices.  Expand Network Adapters and find the grayed-out entry for the old NIC as pictured below.

Select the ghost NIC and right-click | Uninstall to remove it.

The final gotcha for me on this is that the set devmgr_show_nonpresent_devices=1 command does not work on Windows 2008 (or Vista, Windows 7, or Windows 2008 R2).  To see and remove ghost NICs from Windows 2008, and environmental variable must be defined.  To set the variable, open Server Manager from the Windows Start Menu.  Highlight ‘Server Manager (%SERVERNAME%)’ in the left-side tree-view pane.  Click ‘Change System Properties’ in the right-hand pane.  Switch to the Advanced tab and click ‘Environment Variables.  Create a new System variable by clicking the New button.  The Variable name should be ‘devmgr_show_nonpresent_devices’ and the value should be ’1′ as pictured below.

Click OK to close out of any open Windows.  A reboot is not necessary for the variable to take effect, although you may have to close out of all open Device Manager Windows and then reopen devmgmt.msc.  Click View | Show Hidden Devices and remove the ghost NIC as described above.  A quick reboot after I removed the ghost NIC from the domain controllers and all Active Directory, DNS, DHCP, and WINS services immediately began operating normally.  This second issue is more of a Microsoft problem in my opinion, and has been around for some time.

Before you start getting all upset and the FUD starts flying (“this is Microsoft/VMware’s latest attempt to break VMware/Microsoft?”), it wasn’t really vSphere that broke Active Directory; It was me.  A little better planning and not rushing through the last wee hours of the upgrade Window could have saved some trouble.  If you are planning a similar upgrade, it would be best to upgrade your domain controllers/DNS servers one at a time and remediate the issues I have decribed before upgrading the next.  This will ensure continued availability of your Active Directory and other critical services during your upgrade.

Microsoft published a document named “Getting to Know Hyper-V: A Walkthrough from Initial Setup to Common Scenarios” last week.  According to Microsoft, “this guide provides detailed step-by-step walkthroughs for testing Hyper-V on a pre-production environment. You can use this guide to become familiar with Hyper-V and the process of creating and managing virtual machines. Also included in this guide are useful scenarios that you can test to better understand how Hyper-V can address the business goals of your organization.”  The document serves as a sort of evaluators guide for Hyper-V, stepping the reader through everything from enabling VT in BIOS through virtual networking.  It also includes some sections on using snapshots, base virtual machine templates, and managing Hyper-V based virtual machines remotely with Hyper-V Manager.  If you want more in-depth documentation on Hyper-V you can go through http://technet.microsoft.com.

As a side note, Microsoft has published the Microsoft Manual of Style for Technical Publications to help standardize technical documentation.  I have long been a fan of Microsoft’s technical documentation for its easy to read style, although it sometimes lacks the depth that I desire.

While we’re on the topic of virtualization documentation, I have also been quite pleased with VMware’s technical documentation over the years, and have found it to be continually increasing in quality, providing very specific technical guidance and references to additional resources.  I have also been pleased to see that VMware has improved delivery options for documentation.  VMware offers several formats for documentation delivery, including web-based and PDF’s.  Start with the Documentation Roadmap for a quick introduction to the available documentation, and where to find what you need.

You can find web-based vSphere documentation here: http://pubs.vmware.com/vsp40/.   The web-based documentation is great for running searches on.  All vSphere documentation can be accessed through this page: http://www.vmware.com/support/pubs/vs_pages/vsp_pubs_esx40_vc40.html.  If you want to do a full grab of all of VMware’s documentation for an in-house repository (e.g. SharePoint), check out xtravirt’s VMware Documentation Downloader script.

If you are looking for quick and easy evaluator guide-type documentation from VMware, check out these resources: ESXi Installable and vCenter Server Setup Guide and the Virtualization Kit (registration required) at http://www.vmware.com/resources/wp/virtualization101_register.html.

There is a ton of less formal VMware documentation in several places:

• Technical resources and case studies here: http://www.vmware.com/resources/techresources/
• Proven practices around Strategy, Applications, Security, Management, and Availability at VIOPS.
• Official VMware Blogs at http://www.vmware.com/vmtn/planet/vmware/.
• Community blogs aggregated by VMware at Planet v12n: http://www.vmware.com/vmtn/planet/v12n/
• VMworld Recorded Sessions & Labs (VMworld 2009 Sessions available as of today, September 14th) at http://vmworld.com.
• The VMware Community Forums: http://communities.vmware.com/
• And, 3rd party books like Scott Lowe’s Mastering VMware vSphere 4.

Do you have other sources of virtualization documentation or easy methods of searching documentation to find exactly what you need when you need it?  If so, leave a comment!

We picked up a few new 17″ MacBook Pro’s at work. We’re a Microsoft shop, so Mac’s aren’t part of the basic knowledge for our IT staff, myself included. I don’t want to be the Windows guy who says “I don’t do Macs” – part of being a technologist is serving the user base where they are at with the technologies they require to do their job (but please, included me in determining your requirements and technological solutions – a Mac might be really cool, but might not fit with the organizations needs or your IT group’s ability to support your solution). Really, that’s what Web 2.0 is all about – compatible, interchangeable tools that offer customized functionality for the users’ abilities and needs. Come to think of it, that’s what VMware is all about too – the right resources in the right place at the right time, independent of underlying hardware, application/OS agnostic, able to rise above local shortcomings by pushing to the cloud….

To be fair, I was issued a Mac at a previous company, but didn’t care much for it as the programs I had to run for my job were Windows based. I ran VMware Fusion, but it could only take me so far – funny things start to happen when you are in a VM, RDC’ing to a client server, opening the VI client and console’ing to a VM. Shortcut keys behave strangely, and one can only create so many alternate key mappings before going insane. It wasn’t the right tool for me and my job, but Macs do serve some purposes very well – graphic design and iPhone app development in my current case.

I didn’t have a requirement to do much customization the new Mac’s, but they did have to allow users to authenticate to the current Microsoft Windows Active Directory Domain. I hit a few snags as I went through the process, including making domain users local administrators and allowing domain users to log in to the Mac while off-line. Here is what I came up with for a final process in my environment – adjust according to your needs:

1.) Configure OS X to talk to the Active Directory

• Using Spotlight (LeftCommand+Space), open the ‘Directory Utility’
• Switch to the Services tab
• Tick the box next to Active Directory plug-in (Note: You may have to click the lock icon to make configuration changes).
• Highlight the Active Directory plug-in and click the Configure icon (pencil icon).
• Enter an Active Directory Domain, using the FQDN (example: mydomain.local)
• Enter a Computer ID.  This ID will be used to create a computer object in the AD.
• Expand Advanced Options:
  • On the User Experience Tab:
    • Check the box for ‘Create mobile account at login’.
    • Uncheck the box for ‘Require confirmation before creating a mobile account’.
    • Choose the ‘Use UNC path from Active Directory to derive network home location’ if your AD is set to map a user’s home location to a UNC and/or DFS path; if not, you may want to uncheck this option.
  • On the Administrative tab:
    • Check the box for ‘Allow Administration By:” and then Add the Active Directory ‘domain admins’ and ‘enterprise admins’ group
    • Check the box for ‘Allow Authentication from any domain in the forest’ if appropriate for your environment
• Click the Bind button and enter credentials for an account with permissions to join the domain on the Active Directory domain you are joining.  Note: The computer account may appear in the default AD ‘Computers’ container even if the redircmp utility was used on the domain to change the default Organizational Unit (OU) of new computers joining the domain.
• Click OK.
• Verify that the Active Directory Domain that you configured correctly appears with a green dot on the ‘Directory Servers’ tab of the Directory Utility.
• Close the Directory Utility.

2.) Configure basic login options

• Open the Accounts tool from Apple | System Preferences | Accounts
• Click Login Options (Note: you may have to click the lock icon to allow changes to be made).
• Configure the Login Options settings as follows:
  • Automatic Login: Disabled
  • Display login windows as: Name and Password
  • Check the box for Allow network users to login to this computer.
    • Click the Options button and configure all network users (i.e. – all Domain users) or only select users to have login permissions.
  • Configure other options as desired.
• Log out of the local Admin account

3.) Log in using a domain user account (with permissions to login to the server (see above) while connected to the network) using the AD user.name and password

• The first login may take several minutes to complete as a local account is being created.
• Open the Accounts tool from Apple | System Preferences | Accounts
• Highlight the logged-in user’s account.
  • Check the box for ‘Allow user to administer this computer’ as appropriate
  • Verify that the ‘Settings’ button for Mobile Account is grayed out – this means that an offline account has been created for the user.

4.) Test the config by removing network connectivity (disable AirPort and/or pull the network cable) and log in as the user you just configured.

5.) Buy VMware Fusion so you can run Windows on your Mac when all the stuff you were used to just ain’t there anymore 

VMware released version 1.5 of the VI Toolkit for Windows – the PowerShell management and reporting tool of choice for many VMware administrators. The new version carries build number 142961. You can download v1.5 here: http://blogs.vmware.com/vipowershell/. The update includes some 32 new cmdlets, including ones for getting/setting NTP settings on ESX, getting/setting Advanced configuration options on ESX, getting/setting ESX Firewall settings, and the ability to modify DRS rules using PowerShell. Existing cmdlets have also been updated with new parameters, and several fixes have been introduced. Check out the release notes here: http://www.vmware.com/support/developer/windowstoolkit/wintk15/windowstoolkit15-200901-releasenotes.html.

There are plenty of examples on the Internet to get you started with the VI Toolkit for Windows. Check out these sites to get started:

Start at the VMware Community site for the Windows Toolkit for great examples and a little help from some friends: http://communities.vmware.com/community/developer/windows_toolkit/
There are also some good example scripts and resources floating around, such as:

http://vmetc.com/2008/08/27/powershell-scripting-examples-for-vmware-virtual-infrastructure/
http://www.peetersonline.nl/
http://www.ivobeerens.nl/?p=106
http://www.vmguru.com/

Not a hard-core scripter? Grab this handy tool for a little GUI on your PowerShell, and extend it with the VMware Infrastructure PowerPack 2.0

What tools or examples are you using to extend the power of PowerShell into your Virtual Infrasturucture?  Leave a comment to share!

I wrote about a method for determining guest free disk space using a PowerShell script a couple weeks ago.  Scott Lowe picked up the post on his blog last week.  Since then I have had several other conversations with folks looking the best way to report on inefficiencies in their environments (it’s the economy, stupid) and mitigate those inefficiencies as budgets get tighter.

When it comes to reporting there are a ton of options available.  The solution you choose will be dependent on your environment and the tools you already have in place.  Small and Medium-sized Businesses (SMB’s) often do not have full blown, network-wide monitoring and management solutions, so VMware-specific solutions are often a great fit.  There are several examples beyond my simple script, and many are free.  The short list includes: Mightycare Solutions MCS StorageView 1.1, Rich Garsthagen’s VCplus, and Rob de Veij’s RVTools.

There are many other mid-tier solutions – both enterprise-wide and VMware specific – constantly emerging as the virtualization ecosystem matures.  Offerings from ManageIQ, Embotics, Veeam, V-Kernel, Zenoss, Hyperic, and others are increasingly able to provide fresh and relevant data on what is happening under the covers in your virtual environment.

Larger IT shops most likely have a systems monitoring solution easily capable of reporting this – think offerings from the likes of Microsoft, Altiris,  BMC, or CA.  The trick in these solutions is narrowing down the information to your virtualized resources and getting the information to the right teams.  Customized reports using fields such as the BIOS Vendor string can help show only servers running VMware, for example.  As a side note, the Vendor BIOS string can also come in handy when applying Group Policies (GPO), allowing you to filter policies for only virtualized resources (disabling screen savers on Windows guests through GPO is a good example of this).

And don’t forget, we’re not reporting for reporting sake.  We’re after relevant information that allows us to be more efficient and proactive in the overall goals of our environments.  Good reporting identifiies areas in need of improvement, and smart system administrators look for creative ways to improve their systems efficiency.

As the New Year rolls around you may find yourself looking for a new paying gig. Times are tough, but the virtualization space shows continued strength as companies seek to consolidate servers, reduce administrative burden, and hunt down cost savings in the data center. We will look at how you can land that new virtualization gig quickly.

Virtualization jobs come in all shapes and sizes, from the small shop IT guy who runs everything from the desktop to the data center (or data closet as the case may be) to large silo-structured enterprise environments where IT staff rarely deviate from their narrowly defined set of tasks.  Then there is the whole consulting side of the business, with small and large shop mentalities to go along with it.

Step 1 in getting a new job that will really satisfy (Snickers, anyone?) is to define what it is that you want to do.  Are you a big picture thinker who likes to pull in all of the pieces of the puzzle and assemble them over time – maybe a small IT shop is for you.  Do you like rapid-fire problem solving?  Check out a consulting gig where you can solve a variety of problems for customers.  Figure out what you want to do first, then start looking.

Step 2: Target your search.  Sure the big sites like Dice.com, Monster.com, and CareerBuilder.com have a ton of jobs listed, but to find that really great job you should look for other search resources.  Some of the new job search sites, including SimplyHired.com, Juju.com, and Indeed.com can turn up hits that you won’t find other places.  TheLadders.com also yeilds some great gems.  Social networking tools like Facebook and LinkedIn can get you closer to the hiring manager than search sites can.  Here in the Washington DC area we have a bunch of resources such as the Washington Post’s Job , Craigslist Jobs, The Northern Virginia Technology Council’s Job Site, and DCJobs.com.  A little Googling and personal networking will surely reveal similar resources in your area.  You can also get creative.  Use the VMware Partner Directory to identify companies in your area offering virtualization sales, services, hardware, etc.  Also be sure to check out job sites that focus on virtualization, such as VM People.

Step 3: Think outside the box.  Use search terms other than VMware.  Recruiters don’t always know the technology they are looking to staff.  Search for VM Vare, virtual, virtualization, ESX, HyperV.  Think of related related industries in the virtualization ecosystem (think storage vendors, software vendors, startups, systems integrators, etc.).  Visit the websites of those companies and find their ‘Careers’ section.

Step 4: Get your resume updated.  Show your skills and certifications.  Get your resume out there using social networking tools, leverage sites like VisualCV.com to build your portfolio and showcase your accomplishments.

Step 5: Give back to the community after you get that awesome new job.  Is your company hiring more staff?  Post those openings on VMtoday.com.  The stronger your team is, the better your chances are of being noticed for promtion and expanded responsibilities.

Do you have any great sources for finding virtualization jobs?  Leave your ideas in the comments!

Microsoft released version 2 of their Offline Virtual Machine Servicing Tool (OVMSTv2) yesterday.  The updated version of the tool, a component of Microsoft’s Solution Accelerator family of ‘best-practice-meets-tools’ adds integration with System Center Virtual Machine Manager 2008, and support for System Center Configuration Manager 2007 SP1, System Center Configuration Manager 2007 R2, and Windows Server® Update Services 3.0 SP1.  The tool works in much the same way as VMware’s Update Manager product updates Windows guest VM’s. Support for System Center Virtual Machine Manager 2008 obviously means that this product could supplement, replace or conflict with VMware’s Update Manager’s ability to patch Windows guests hosted on a VMware Virtual Infrastructure.

There is not yet much information on this version of the Offline Virtual Machine Servicing Tool.  A MSDN blog post from May 2008 includes some information on the beta version of the product.  The Getting Started guide included in the OVMSTv2 .zip download reveals a bit more about the requirements and footprint required to implement the solution.  The solution requires a good deal of Microsoft software, including (depending on your environment and requirements) Microsoft Active Directory, Windows Server Update Services (WSUS), Configuration Manager 2007, and Virtual Machine Manager 2007, and an appropriately designed Group Policy (GPO) environment.  Another requirement is that the target VM’s have DHCP assigned IP addresses.

I did not read anywhere in the documentation that the OVMSTv2 integrates snapshot capabilities to provide simple roll-back of guests that suffer problems on Patch Tuesday, as VMware Update Manager does.  I also did not see a reference to network isolation capabilites in OVMSTv2.  VMware Update Manager keeps offline VM’s offline during the patching process.  OVMSTv2 has to reach out to a WSUS or Configuration Manager server, so I suspect that the guest is not isolated during that patching process.

I have not had a chance to build up all of the requirements of this tool in the lab, so I don’t have any practical advise for using the tool.  Have any of you used the beta or release versions of the tool?  Post a comment to let everyone know your experiences.

The Microsoft Offline Virtual Machine Servicing Tool v2 could be a good tool to add to your Microsoft patch management arsenal, but the small footprint and simplicity of VMware Update Manager will most likely keep all but the most dedicated Microsoft shops from implementing OVMSTv2.