Home Lab ESXi Install Problem on HP xw8600 with LSI1068

I recently finished a big basement remodel project (gutted it and started from scratch!), leaving me with a new, bigger office space.   The Townsend Family Director of Standards and Practices (a.k.a. She Who Must Be Obeyed, a.k.a. my wife) thinks she’s got a nice new space to do some work in, but I’ve got better plans! My VMware Home Lab reborn!  I started the rebuild tonight with a pair of HP xw8600 workstations.  They are a nice system for a home lab – dual-socket CPU (Intel Xeon 5400 quad-core series), up to 128GB RAM, 14 channels of on-board LSI 1068 SATA/SAS RAID Controller, PCIe x16, Dual Broadcom Gigabit networks adapters.  Oh yeah, and a Fusion-IO ioDrive 320GB card 😉

But….  when I went to install VMware ESXi 5.x I ran into problems.  The install went fine, or so I thought (when you do enough of them you stop paying attention and just start banging on the Enter and F11 keys until it’s done ;-)).  First step was to put the Fusion-IO VIB into ESXi so I could start taking advantage of all that crazy fast, nano-second latent, IOP screaming goodness.  A little SCP action to copy the VIB to /tmp on the first host failed with an out-of-space type error.  Odd.  Last I knew I had 700GB of SATA at my command, and ‘df -h’ agreed with me.  A bad install?  I didn’t check the MD5 of the ISO before I burned it…

I reinstalled, and this time was met with a Purple Screen of Death (PSOD).  Bad hardware? I installed Windows 7 on the same box and had no issues at all – all storage seen, no crashes, stable as stable as Windows can be.  I’ll take this to mean that all the hardware is functional, SATA disks correctly presented off of the SAS controllers a single RAID5 array.  I’m thinking a driver issue at this point.

Re-install ESXi (and just maybe pour myself a bourbon).  Another PSOD.  This time I actually paused to read the PSOD details before kicking the box.  A few lines caught my eye:

Logs are stored on non-persistent storage. Consult product documentation to configure a syslog server or a scratch partition.

No place on disk to dump data.

Interesting.  No scratch partition.  I thought those were included in the base options on this hot rod called ESXi.

Install, again.  This time, pay attention to the install routine.  I found that in my rush to get through the install, I ignored this screen:

On the ‘Select a Disk to Install or Upgrade’ screen of the VMware ESXi 5.0 installer, there were no (none) Local disks detected, and my internal RAID volume on the LSILOGIC controller was detected as a Remote disk.  A little LMGTFY action led me to this page in VMware’s vSphere 5.0 Install documentation:

For Serial ATA (SATA), a disk connected through supported SAS controllers or supported on-board SATA controllers. SATA disks will be considered remote, not local. These disks will not be used as a scratch partition by default because they are seen as remote.

So even through the LSI1068E controller is supported by ESXi, the SAS port attached SATA disks showed up as Remote, leaving my installation without a scratch partition and seemingly a bit unstable as a result.

My work-around is simple.  Install ESXi on a USB drive, then use the SATA and Fusion-IO for my VMFS datastores.  I’ve had good luck with the Verbatim TUFF ‘N’ TINY USB 2.0 Flash Drive – it is fast, good capacity, and tiny.  The small form factor makes it all but flush with the back of the server, keeping it from getting knocked out or broken off by the kids.

Hope this helps you if you are planning to install your VMware ESXi on an xw8600 or another system with an LSI1068E controller with SATA disks on a SAS controller!

And, as a bonus tip, PC2 5300F 555MHz RAM for the xw8600.  I was trying to cheap out and eBay some memory and got the wrong PC2-5300P DIMM’s.  Apparently there is a difference….  I now have some extra PC2-5300P 2GB DIMM’s for the highest bidder!