My Storage Basics series has been neglected for some time (sick kids, snow storms, VMware Upgrades, SAN implementations and some Cisco switch upgrades took all my free time), so let’s jump right in to Part V – Cache, Controllers, and Coalescing.   Between the alliteration and fancy words, it might seem like I am about to tell a tale of international espionage.  Unfortunately, my introductory treatment of these aspects of a storage system will probably not keep you on the edge of your seat – but I’ll try to keep it interesting.

Throughout this series, we’ve been working our way from the basic building block of any storage system – the disks – outwards towards the brains of the operation – the controller.  You’ll recall that in Part II I introduced IOPS and the math that goes into calculating the IOPS capacity of a disk array.  In Part III we considered a RAID implementation’s impact on performance and availability.  And most recently in Part IV we looked at the common interface types when dealing with storage arrays.  If we put the previous parts together we still don’t have a functional storage system.  The missing piece is the controller.  Simply put, the storage controller is the hardware adapter between the disks and the servers that connect to the storage.  The controller has a specific ‘interface‘ type, is responsible for RAID operations, and handles advanced storage functionality.  A controller can be as simple as the Dell PERC or HP Smart Array add-in card on your server, or as complex as the Storage Processor in an enterprise class Storage Area Network (SAN) such as an EMC CLARiiON or NetApp FAS.

Controllers

As we look at controllers and the advanced features they provide we’ll see that some of the earlier performance equations start to break down.  The simplest controllers take disk read/write commands from the operating system and send commands down to the disk(s) attached to be read or written. This gets data onto the disk, but often does not do so in an efficient or reliable manner.  RAID-capable controllers take on the added responsibility of configuring disks in the desired RAID level, calculating & writing parity data, and writing the data in disk-spanning stripes or mirrors depending on the RAID level.

Cache

To increase performance and improve reliability, storage vendors implement a caching system on their controllers.  Cache is memory that acts as a buffer for disk I/O, and is usually battery-backed to prevent data loss in the event of a power failure.  Because of the exponentially greater speed of RAM over spinning magnetic disks, cache can improve performance by orders of magnitude. Cache can operate on both reads and writes to disk.

When dealing with writes, the controller cache is typically used in one of two ways: write-through or write-back.  In write-through mode, data is written to volatile cache and then to disk, and only acknowledged as written once the data resides on the non-volatile disk.  Write-back mode allows the controller to acknowledge the data as having been written as soon as it is held in cache.  This allows the cache to buffer writes quickly and then write them to the slower disk when the disk has cycles to accept I/O.  The greater your cache size, the more data that can be buffered, ultimately resulting in better performance as measured in both IOPS and throughput.  This graph from my article on troubleshooting write performance on an IBM DS3300 iSCSI array shows how throughput increased and latency decreased when enabling write cache.  The extent to which cache increases performance is highly dependent on the workload characteristics (I/O size, randomness, and ratio of reads:writes).

Read-cache acts as a buffer for reads in a couple ways.  First, some controllers attempt to ‘read-ahead’, anticipating future read requests from the operating system and buffering what it expects to be the next blocks of desired data.  Some entry-level controllers simply buffer the next physical chunk of data and fill cache memory with it, while more advanced controllers may attempt to predict the right block of data based on previous requests (you just asked for 3 blocks in a row, I’m guessing you’ll come asking for the 4th next so I’ll just buffer it in fast cache for you now).  Secondly, read cache holds data that has been previously read, regardless of any pre-fetching the controller may have done.  This allows for much faster subsequent access of the same data because it is held in the faster cache, eliminating the need for the controller to go to disk for the data again.  Just like with write cache, the extent to which cache increases performance is highly dependent on the workload characteristics.

A given storage array controller only has so much cache to work with.  A Dell PERC5/E, for example, has 256MB of cache that can be used for both read and write.  While this may be enough for a direct-attached storage array, SAN’s serving multiple systems demand more cache.  In contrast, an EMC CLARiiON CX4-960 has 32GB.  Some storage vendors, such as NetApp, are getting creative with cache.  NetApp’s Performance Acceleration Module (PAM) is an add-in card that provides up to a whopping 512GB of Layer 2 cache to the storage system.

Caching mechanisms can dramatically influence performance under the right conditions.  With healthy cache in place, IOPS calculations become skewed.  However, cache can be exhausted or may not hold the data you are interested in.  If cache is insufficient to satisfy read requests, or has reached its high-water mark for writes, performance can drop off.  When cache is exhausted, the backing disk must be able to satisfy the I/O workload or performance will be unacceptable.  This is where the IOPS calculations kick in, and where having the right disk type and configuration really matters.

Queuing & Coalescing

Advanced storage systems introduce additional features to reduce I/O contention and improve cache utilization.  I won’t go into all of the features here because they vary by storage vendor.  However, I will point out two common techniques – queuing and coalescing.

Queuing refers to the ability of a storage system to queue storage commands for later processing.  Queuing can take place at various points in your storage environment, from the HBA to the storage processor/controller.  A little queuing may be OK depending on your workload, but too many outstanding I/Os can negatively impact performance (this is measured in latency).  Queue depths can be adjusted on many components in your storage and VMware landscape, but check with your vendor’s support group before you make changes to these settings.

Coalescing is performed by some storage systems to modify the character of the workload.  To better understand coalescing, picture a bunch of random write activity.  Without cache in place, the disk heads will be bouncing all over the platters trying to get the data on to disk.  A little write cache will allow the storage array to acknowledge the write for the OS, but the array still needs to de-stage that data from cache to disk quickly to prevent cache exhaustion.  The back-end disks will still be doing the chicken dance, bouncing around trying to write the random workload….  Now picture an intelligent system that re-orders the random writes that are held in cache and writes them to the disk in nice sequential stripes.  The disk heads will be less prone to jumping around the platter and the behavior will start to look more like a nice waltz than the funky chicken dance.  Coalescing is used for writes, not reads, so not all workloads benefit.

Wrap-up

With this article on Controllers, Cache, and Coalescing we’ll end our look at the basic building blocks of a storage array.  Before we end the Storage Basic series I am planning a few more articles on Storage Workload Characterization (which has been mentioned, but not directly addressed in this and previous articles), Identifying a Stressed Storage System, and Best Practices for Storage Performance in a VMware Environment.

If you are interested in more reading on Controllers, Cache, and Coalescing, I recommend the following:

Additional Reading:

• Impact of cache on the performance of the HP StorageWorks XP12000 Disk Array white paper
• Performance impact of controller cache: SQL Server read only workloads
• IOps? – Dig into the article’s comments for some great dialog between some people who really know their stuff!
• Storage Performance for SQL Server
• Storage Caching 101 – Chuck Hollis (EMC)
• Improving Performance with Interrupt Coalescing for Virtual Machine Disk IO in VMware ESX Server

We all know that virtualization allows us to do more with less.  Fewer servers and space-saving storage (talk about an oxymoron) help us put some green in the datacenter and back in the budget.  But with tight budgets demanding greater efficiency, virtualization pushing per-U-space utilization higher, and increasingly rack-dense equipment, proper planning of your physical plant remains an essential part of IT.  I argue that right-sizing your power, cooling, and floor-space is more critical now than it has ever been, and is a knowing how to do it is a darn good skill for a virtualization engineer to possess.

So along those lines… I was just doing some site-prep work for a new Clariion installation and noticed that the EMC Power Calculator has been updated.  It is now a pretty slick little web app that can be found on the PowerLink site (login required) here: https://powerlink.emc.com/nsepn/webapps/powercalculator/Main.aspx.

While I am at it, here are some links to other power consumption calculators.  Let me know if you have others and I will update this post:

• Dell: http://www.dell.com/calc
• IBM: http://www-03.ibm.com/systems/bladecenter/resources/powerconfig/index.html
• NetApp: Storage Efficiency Calculator here - http://www.secalc.com – it doesn’t calculate your consumption, just what you might save over a competitor’s offering.
• HP: http://h30099.www3.hp.com/configurator/powercalcs.asp
• Sun: http://www.sun.com/solutions/eco_innovation/powercalculators.jsp
• Hitachi/HDS: http://www.byhitachi.com/se/go/weight-and-power-calculator/
• APC: http://www.apc.com/prod_docs/results.cfm?DocType=Trade-Off%20Tool&Query_Type=10 and http://www.apcc.com/products/runtime_for_extendedruntime.cfm?upsfamily=165
• Emerson: Efficiency Calculator: http://www.emerson.com/edc/Calculator/default.aspx
• VMware ROI Calculator: http://vmware.com/go/calculator
• This site has a bunch of links to other calculators and resources: http://thegreenandvirtualdatacenter.com/calculator.html

There’s some fun and timely chatter happening right now on Twitter around power consumption and sizing – join in by following me at http://twitter.com/joshuatownsend/!

In Part I of this series, I discussed the important of storage performance in a virtual environment (really any environment, virtual or not, where you want acceptable performance), and introduced some of the basic measures of a storage environment.  In Part II, we will look more closely at what may be the most important storage design consideration in a VMware server-consolidation enviornments, many SQL environments, and VDI environments to name a few: IOPS.

If we stick with a single-disk-centric approach as we did in Part I, IOPS is quite simply a measure of how many read and write commands a disk can complete in a second.  IOPS is an important measure of performance in a shared storage environment (such as VMware) and in high-transaction-rate workloads like SQL.  Because hard drives are forced to abide by the laws of physics, the IOPS capabilities of a disk are consistent and predictable given a specific configuration.  The formula for calculating IOPS for a given disk is pretty straight forward (please show your work):

IOPS = 1000/(Seek Latency + Rotational Latency)

Exact latencies vary by disk type, quality, number of platters, etc.  You can look up the tech specs for most drives on the market.  As an example, I have randomly chosen the technical specifications of the Seagate Cheatah 15k.7 SAS drive.  This particular drive has the following performance characteristics:

- Average (rotational) latency: 2.0msec

- Average read seek (latency): 3.4msec

- Average write seek (latency): 3.9msec

Using the read latency number, the math works out like this:

1000
———- = 185 maximum read IOPS
2.0+3.4

The maximum write IOPS will be a bit less (~169IOPS) because of the higher write seek latency.  Writing is more ‘expensive’ than reading and therefore slower.

Fortunately, there are some widely accepted ‘working’ numbers, so you do not have to use this formula for each and every disk you might consider using.  Because rotational latency is based on the rotational speed, we can use the published Rotations Per Minute (RPM) rating of the drive to guess-timate the IOPS capabilities.  Typical spindle speeds (measured in RPM) and their equivalent IOPS are in the table below.

RPM………IOPS

7,200          80

10,000       130

15,000       180

SSD           2500 – 6000

While not a traditional spinning disk, I have also included Solid State Disks (SSD’s) for reference as SSD’s are starting to see increased market adoption.  I have seen a wide range of sizing IOPS for SSD depending on the technology, type (SLC, MLC, etc.)  Check out http://en.wikipedia.org/wiki/Solid-state_drive for an introduction, and ask your vendors for more in-depth technical information.

If you are brand-new to this (and you are still reading, congrats!), you can see how many IOPS your Windows computer is asking for by opening Performance Monitor and looking at the ‘Disk Transfers/sec’ counter under Physical Disk.  This is a sum of the ‘Disk Reads/sec’ and ‘Disk Writes/sec’ counters as you can see in the screenshot below:

If you are after some stats for your VMware ESX environment, check out esxtop and looking for CMDS/s in the output.  I published a couple articles on using esxtop here and here.  The numbers from PerfMon and esxtop get you pretty close but can be skewed by a few things we’ll discuss in later posts.

Now that was fun and all, but let’s get real: Single-disk configurations are uncommon in servers.  As such, we’ll part ways with our Simple Jack single disk approach to storage and begin to look at more real-world multi-disk enterprise-class storage configurations.  A discussion of IOPS in a multi-disk array is a great way to start.  From a very elementary perspective, you can combine multiple hard drives together to aggregate their performance capabilities.  For example, two 15k RPM disks working together to server a workload could provide a theoretical 360 IOPS (180 + 180). This also  scales out so ten 15k RPM disks could provide 1800 IOPS, and 100 15k RPM disks could provide 18,000 IOPS.

Designing your environment so that your storage can deliver sufficient IOPS to the requesting workload is of utmost importance.  If you are working on a storage design, arm yourself with data from perfmon, top, iostat, esxtop, and vscsiStats.  I typically gather at least 24 hours of performance data from systems under normal conditions (a few days to a week may be good if you have varying business cycles) and take the 95th percentile as a starting point.  So from a very simple approach, if your data and calculations show a 1800 IOPS demand at the 95th percentile, you ought to have at least ten 15k RPM disks (or twenty-three 7.2k RPM SATA disks) to achieve performance goals.  It’s amazing how some simple data and a pretty little Excel spreadsheet can help you understand and justify the right hardware for the job.

Now before you go and start filling out that PO form for a nice new storage system based on these numbers there are a few more things we ought to discuss.  RAID, cache, and advanced storage technologies will skew these numbers and need to be understood.  Stay tuned to future articles in this series for more on those topics and more.

Finally, there has been a bunch of activity in the VMware ecosystem of vendors, bloggers, and twittering-type-folks around storage performance.  As this here post sat in my drafts folder, Duncan Epping posted this gem of an article that pretty much included all of the content of this article, as well as future ones in my series: http://www.yellow-bricks.com/2009/12/23/iops/.  Do yourself a favor and read his post and the comments from his readers – both are filled with a ton of great information, including some vendor-specific implementations.
I was led to Duncan’s article by a post by Chad Sakac on his blog: http://virtualgeek.typepad.com/virtual_geek/2009/12/whats-what-in-vmware-view-and-vdi-land.html.  This is also a great read that covers some of the same information with a focus on VMware View/VDI and is also worth a few minutes of your time.  Also check out http://vpivot.com/2009/09/18/storage-is-the-problem/ for a rubber-meets-the-road post from Scott Drummonds on the importance of storage performance vis-a-vis IOPS in a VMware-virtualized SQL environment.

I am increasingly finding that both my SMB and Enterprise customers are uneducated on the fundamentals of storage sizing and performance.  As a result, storage is often overlooked as a performance bottleneck despite it being a vital component to consider in a virtualization implementation.  Storage will only increase in importance as hosts are getting bigger, data volumes increase, and more workloads are virtualized.  For some reason, most people can grasp the importance of CPU and memory performance constraints but storage performance is often overlooked and can be hard to explain to business users or executives.

Case in point – I have recently been called into some environments that were not performing well – these environments happened to be running Microsoft SQL, but could just have well been running any application or collection of virtual machines.  Fingers were being pointed in all directions: at applications, at the virtualization layer, at a lack of memory, and DBA’s were insisting that there were too few CPU’s.  The situation was getting political and emotional when I walked into it.  A few minutes with Windows Perfmon was all I needed to identify storage performance as the root cause of the firestorm that had been ignited.  Using a bit of data, I was able to turn the discussion from an emotional fight to a simple problem of physics and mathematics (and a bit of simple math could have avoided the problem in the first place).

I have seen this play out a few too many times and so decided to write-up this multi-part series on the basics of storage with a focus on storage performance.  That said, a little math and physics is where we will start as we look at the basic building block of a storage environment: a hard disk drive.  Wikipedia defines a hard disk drive as “a non-volatile storage device that stores digitally encoded data on rapidly rotating platters with magnetic surfaces.” Your computer, server, or VMware cluster uses hard drives to read and write data.  Wikipedia also covers the history and atomic structure of a hard drive pretty well.  For our purposes, the take away is that hard drives are physical objects, and as such, follow the laws of physics (duh) in the following measurable ways:

1.) Capacity, which is measured in bits or bytes and exponents there of (MB, GB, TB, PB).  This is how much data will fit on your disk, from simple text files to virtual disks, and everything in between.  For example, if you have a 500GB SQL database, you darn well better have a hard drive that has a capacity of at least 500GB.  This is a pretty simple concept, so I’ll leave it there for now.

2.) Performance, which is measured in a couple ways:

- at the disk itself in Input-Output Per Second (IOPS) – a measure of how many read and write commands a disk can complete in a second

- interface throughput, measured in MBps or Gbps – a measure of the peak rate that a volume of data can be read from or written to disk

- latency – the amount of time between when you ask a disk (or storage system if you want to read ahead) to do something and when it can actually do it, very closely related to IOPS as you’ll read in a forthcoming article in this series.

Each disk, array, and storage system has its own fixed set of measurements given a specific configuration.  Knowing the physical capabilities of your storage system as measured in the above ways, and your systems storage requirements will go a long way towards a successful design and implementation of your storage environment.  The remaining parts of this series will take a look at these performance characteristics a bit more in-depth and explain what happens as you introduce factors like RAID, cache, data reduction techniques such as snapshots and deduplication, and varying workloads.

Please keep in mind that while I have designed and implemented a variety of DAS, NAS, and SAN technologies from a host of vendors including Dell, EMC, IBM, and NetApp, I am by no means a storage expert.  The information I will provide is generalized, over-simplified, and does not consider varying approaches from different storage vendors.  Nonetheless, I hope you find this useful information if you are designing a solution, troubleshooting a performance issue or preparing to make a storage purchase.

Keep Reading:

Storage Basics – Part II: IOPS

Here are some bookmarks for resources that I have recently referenced:

• vCenter 4 and ESX 4 Now Use 10 Year Default SSL Certificate | VM /ETC – Rich Brambly has some guidance on installing a new SSL certificate on vCenter, with very useful links in his post to official VMware documentation and KB’s on the subject.
• VMware vSphere Client on Microsoft Windows 7! | Virtual Lifestyle – Heiko Verlande has found a way to run the VMware vSphere Client on Windows 7.
• Virtu-Al » PowerCLI: Daily Report V2 – Version two of a handy PowerShell based VMware Environment Daily Report from VMware vExpert and PowerShell guru Alan Renouf
  What’s new/Bug Fixes
  * Active VMs count
  * Inactive VMs count
  * DRS Migrations count and list
  * Correct NTP Server check for each host
  * VMs stored on local datastores
  * NTP Service check for each host
  * vmkernel warning messages for each host
  * VM CPU ready over x%
• VMware Self-Service- VMware Update Manager Plug-In fails to install -Troubleshooting steps for vCenter Plug-in install problems.
• Using VMware VDI and vmSight for Stronger and Sustainable HIPAA and PCI Compliance – Virtualization brings new options for protecting sensitive data by moving it from the desktop into the datacenter.
• Counter of the Week : Analyzing Storage Performance – The purpose of this article is to provide prescriptive guidance on how to troubleshoot logical and physical disk response times in regards to Windows performance analysis. Start with the following performance counters to analyze disk response…
• NetApp, Compellent, HP, Dell top the field in 12-product test – Network World – A terabyte isn’t what it used to be. Disks are slower than you think. And a Gigabit Ethernet is plenty of bandwidth for many storage applications.

We picked up a few new 17″ MacBook Pro’s at work. We’re a Microsoft shop, so Mac’s aren’t part of the basic knowledge for our IT staff, myself included. I don’t want to be the Windows guy who says “I don’t do Macs” – part of being a technologist is serving the user base where they are at with the technologies they require to do their job (but please, included me in determining your requirements and technological solutions – a Mac might be really cool, but might not fit with the organizations needs or your IT group’s ability to support your solution). Really, that’s what Web 2.0 is all about – compatible, interchangeable tools that offer customized functionality for the users’ abilities and needs. Come to think of it, that’s what VMware is all about too – the right resources in the right place at the right time, independent of underlying hardware, application/OS agnostic, able to rise above local shortcomings by pushing to the cloud….

To be fair, I was issued a Mac at a previous company, but didn’t care much for it as the programs I had to run for my job were Windows based. I ran VMware Fusion, but it could only take me so far – funny things start to happen when you are in a VM, RDC’ing to a client server, opening the VI client and console’ing to a VM. Shortcut keys behave strangely, and one can only create so many alternate key mappings before going insane. It wasn’t the right tool for me and my job, but Macs do serve some purposes very well – graphic design and iPhone app development in my current case.

I didn’t have a requirement to do much customization the new Mac’s, but they did have to allow users to authenticate to the current Microsoft Windows Active Directory Domain. I hit a few snags as I went through the process, including making domain users local administrators and allowing domain users to log in to the Mac while off-line. Here is what I came up with for a final process in my environment – adjust according to your needs:

1.) Configure OS X to talk to the Active Directory

• Using Spotlight (LeftCommand+Space), open the ‘Directory Utility’
• Switch to the Services tab
• Tick the box next to Active Directory plug-in (Note: You may have to click the lock icon to make configuration changes).
• Highlight the Active Directory plug-in and click the Configure icon (pencil icon).
• Enter an Active Directory Domain, using the FQDN (example: mydomain.local)
• Enter a Computer ID.  This ID will be used to create a computer object in the AD.
• Expand Advanced Options:
  • On the User Experience Tab:
    • Check the box for ‘Create mobile account at login’.
    • Uncheck the box for ‘Require confirmation before creating a mobile account’.
    • Choose the ‘Use UNC path from Active Directory to derive network home location’ if your AD is set to map a user’s home location to a UNC and/or DFS path; if not, you may want to uncheck this option.
  • On the Administrative tab:
    • Check the box for ‘Allow Administration By:” and then Add the Active Directory ‘domain admins’ and ‘enterprise admins’ group
    • Check the box for ‘Allow Authentication from any domain in the forest’ if appropriate for your environment
• Click the Bind button and enter credentials for an account with permissions to join the domain on the Active Directory domain you are joining.  Note: The computer account may appear in the default AD ‘Computers’ container even if the redircmp utility was used on the domain to change the default Organizational Unit (OU) of new computers joining the domain.
• Click OK.
• Verify that the Active Directory Domain that you configured correctly appears with a green dot on the ‘Directory Servers’ tab of the Directory Utility.
• Close the Directory Utility.

2.) Configure basic login options

• Open the Accounts tool from Apple | System Preferences | Accounts
• Click Login Options (Note: you may have to click the lock icon to allow changes to be made).
• Configure the Login Options settings as follows:
  • Automatic Login: Disabled
  • Display login windows as: Name and Password
  • Check the box for Allow network users to login to this computer.
    • Click the Options button and configure all network users (i.e. – all Domain users) or only select users to have login permissions.
  • Configure other options as desired.
• Log out of the local Admin account

3.) Log in using a domain user account (with permissions to login to the server (see above) while connected to the network) using the AD user.name and password

• The first login may take several minutes to complete as a local account is being created.
• Open the Accounts tool from Apple | System Preferences | Accounts
• Highlight the logged-in user’s account.
  • Check the box for ‘Allow user to administer this computer’ as appropriate
  • Verify that the ‘Settings’ button for Mobile Account is grayed out – this means that an offline account has been created for the user.

4.) Test the config by removing network connectivity (disable AirPort and/or pull the network cable) and log in as the user you just configured.

5.) Buy VMware Fusion so you can run Windows on your Mac when all the stuff you were used to just ain’t there anymore 

VMware released version 1.5 of the VI Toolkit for Windows – the PowerShell management and reporting tool of choice for many VMware administrators. The new version carries build number 142961. You can download v1.5 here: http://blogs.vmware.com/vipowershell/. The update includes some 32 new cmdlets, including ones for getting/setting NTP settings on ESX, getting/setting Advanced configuration options on ESX, getting/setting ESX Firewall settings, and the ability to modify DRS rules using PowerShell. Existing cmdlets have also been updated with new parameters, and several fixes have been introduced. Check out the release notes here: http://www.vmware.com/support/developer/windowstoolkit/wintk15/windowstoolkit15-200901-releasenotes.html.

There are plenty of examples on the Internet to get you started with the VI Toolkit for Windows. Check out these sites to get started:

Start at the VMware Community site for the Windows Toolkit for great examples and a little help from some friends: http://communities.vmware.com/community/developer/windows_toolkit/
There are also some good example scripts and resources floating around, such as:

http://vmetc.com/2008/08/27/powershell-scripting-examples-for-vmware-virtual-infrastructure/
http://www.peetersonline.nl/
http://www.ivobeerens.nl/?p=106
http://www.vmguru.com/

Not a hard-core scripter? Grab this handy tool for a little GUI on your PowerShell, and extend it with the VMware Infrastructure PowerPack 2.0

What tools or examples are you using to extend the power of PowerShell into your Virtual Infrasturucture?  Leave a comment to share!

Between budget cuts and New Year’s resolutions, improving your security posture is probably near the top of your to-do list.  Much has been made of security concerns in a virtual environment, but it is always good to re-visit your configurations and make sure they are still on par with recommended best practices.  I began re-reviewing VI security best practices after reading at post by Bob Plankers at The Lone SysAdmin (Bob has been on my reading list for years – he has a great style and always brings fresh insights) on why you would want a second super-user account on your ESX servers.

We certainly all have our own opinions and operations procedures when it comes to configuring and hardening our environments, but I decided to take a look at what the experts had to say on this particular subject and other basic build and hardening recommendations.  Here is what I found:

VMware Security Resources

VMware Security Utilities

VI3.5 Security Hardening Whitepaper

Defense Informaion Systems Agency (DISA) ESX Server Security Technical Implementation Guide

DISA ESX Server Checklist

As a side note, DISA publishes many STIG’s at http://iase.disa.mil/stigs/.  Your tax dollars paid for these, so you might as well check them out.

NSA VMware ESX  Server 3 Configuration Guide

There are also numerous tips and scripts for locking down your virtual infrastructure in the VMware Community Forums (Start here: http://communities.vmware.com/message/941372).

So back to the question of second super user accounts: It seems that best practices are to create a second user account with sufficient access to the console, granting that user SUDO privledges, and then disabling the default root account.

I wrote about a method for determining guest free disk space using a PowerShell script a couple weeks ago.  Scott Lowe picked up the post on his blog last week.  Since then I have had several other conversations with folks looking the best way to report on inefficiencies in their environments (it’s the economy, stupid) and mitigate those inefficiencies as budgets get tighter.

When it comes to reporting there are a ton of options available.  The solution you choose will be dependent on your environment and the tools you already have in place.  Small and Medium-sized Businesses (SMB’s) often do not have full blown, network-wide monitoring and management solutions, so VMware-specific solutions are often a great fit.  There are several examples beyond my simple script, and many are free.  The short list includes: Mightycare Solutions MCS StorageView 1.1, Rich Garsthagen’s VCplus, and Rob de Veij’s RVTools.

There are many other mid-tier solutions – both enterprise-wide and VMware specific – constantly emerging as the virtualization ecosystem matures.  Offerings from ManageIQ, Embotics, Veeam, V-Kernel, Zenoss, Hyperic, and others are increasingly able to provide fresh and relevant data on what is happening under the covers in your virtual environment.

Larger IT shops most likely have a systems monitoring solution easily capable of reporting this – think offerings from the likes of Microsoft, Altiris,  BMC, or CA.  The trick in these solutions is narrowing down the information to your virtualized resources and getting the information to the right teams.  Customized reports using fields such as the BIOS Vendor string can help show only servers running VMware, for example.  As a side note, the Vendor BIOS string can also come in handy when applying Group Policies (GPO), allowing you to filter policies for only virtualized resources (disabling screen savers on Windows guests through GPO is a good example of this).

And don’t forget, we’re not reporting for reporting sake.  We’re after relevant information that allows us to be more efficient and proactive in the overall goals of our environments.  Good reporting identifiies areas in need of improvement, and smart system administrators look for creative ways to improve their systems efficiency.

As the New Year rolls around you may find yourself looking for a new paying gig. Times are tough, but the virtualization space shows continued strength as companies seek to consolidate servers, reduce administrative burden, and hunt down cost savings in the data center. We will look at how you can land that new virtualization gig quickly.

Virtualization jobs come in all shapes and sizes, from the small shop IT guy who runs everything from the desktop to the data center (or data closet as the case may be) to large silo-structured enterprise environments where IT staff rarely deviate from their narrowly defined set of tasks.  Then there is the whole consulting side of the business, with small and large shop mentalities to go along with it.

Step 1 in getting a new job that will really satisfy (Snickers, anyone?) is to define what it is that you want to do.  Are you a big picture thinker who likes to pull in all of the pieces of the puzzle and assemble them over time – maybe a small IT shop is for you.  Do you like rapid-fire problem solving?  Check out a consulting gig where you can solve a variety of problems for customers.  Figure out what you want to do first, then start looking.

Step 2: Target your search.  Sure the big sites like Dice.com, Monster.com, and CareerBuilder.com have a ton of jobs listed, but to find that really great job you should look for other search resources.  Some of the new job search sites, including SimplyHired.com, Juju.com, and Indeed.com can turn up hits that you won’t find other places.  TheLadders.com also yeilds some great gems.  Social networking tools like Facebook and LinkedIn can get you closer to the hiring manager than search sites can.  Here in the Washington DC area we have a bunch of resources such as the Washington Post’s Job , Craigslist Jobs, The Northern Virginia Technology Council’s Job Site, and DCJobs.com.  A little Googling and personal networking will surely reveal similar resources in your area.  You can also get creative.  Use the VMware Partner Directory to identify companies in your area offering virtualization sales, services, hardware, etc.  Also be sure to check out job sites that focus on virtualization, such as VM People.

Step 3: Think outside the box.  Use search terms other than VMware.  Recruiters don’t always know the technology they are looking to staff.  Search for VM Vare, virtual, virtualization, ESX, HyperV.  Think of related related industries in the virtualization ecosystem (think storage vendors, software vendors, startups, systems integrators, etc.).  Visit the websites of those companies and find their ‘Careers’ section.

Step 4: Get your resume updated.  Show your skills and certifications.  Get your resume out there using social networking tools, leverage sites like VisualCV.com to build your portfolio and showcase your accomplishments.

Step 5: Give back to the community after you get that awesome new job.  Is your company hiring more staff?  Post those openings on VMtoday.com.  The stronger your team is, the better your chances are of being noticed for promtion and expanded responsibilities.

Do you have any great sources for finding virtualization jobs?  Leave your ideas in the comments!