vCenter Virtual Appliance Network Settings Reverting

In one of my lab environments my vCenter Server Appliance (VCVA VCSA*) kept reverting its DNS server settings after I changed them in the VCSA web interface (https://:5480).  After changing the Preferred DNS Server and Alternate DNS Server settings to use my Active Directory Domain Controller IPs and then rebooting the appliance, the DNS settings were removed and the Preferred DNS Server setting was reverted to the setting present before I updated it.

Losing my DNS Server settings caused me all sorts of issues – my vCenter Server couldn’t find ESXi hosts by name, AD authentication failed, and general mayhem ensued.  I reviewed some logs on the appliance and poked around in /etc/sysconfig/network/config, but nothing jumped out at me as being wrong.  I even went so far as to manually update /etc/resolv.conf with the correct settings but after a reboot the DNS server settings were lost.  I needed some help to figure this one out, so I visited my Universal Troubleshooting Accelerator Dispensing Machine, poured a pint of my Rye Porter homebrew and took a step back to think outside the box (or outside the VCSA as it were).

If the problem is not coming from inside the appliance, it’s got to be coming from somewhere outside the appliance.  Either my incorrect DNS is getting set by DHCP or some other mechanism.  I’m using a Static IP so DHCP isn’t handing out a DNS server.  Think, dummy, think (absorb, beer, absorb)!  Ah ha!  I didn’t deploy the VCSA – it was deployed by the data center administrator to a separate management cluster before I had built an AD or DNS infrastructure on the hosts my VCSA was managing.  Because the administrator didn’t know about my non-existent DNS servers, he used a default setting when he set the properties of the vApp while deploying the OVF.  The OVF vApp settings were injected into the appliance and applied at each reboot, overwriting my settings.

I always like to dig a bit deeper when I run into a problem, so let’s look briefly at how vApp Options are applied to better understand this mechanism.  When a virtual appliance is imported via OVF, it either used VMware Tools to read the vApp Options through the vami service (see /opt/vmware/share/vami) or a virtual CD-ROM was mounted that contained an XML file with the vApp Options.  On the vCenter Virtual Appliance (based on SUSE Linux Enterprise 11), the vApp Options are written to /opt/vmware/etc/vami/ovfEnv.xml.  The vami service reads the settings in from the XML and applies them at subsequent boot (via /opt/vmware/share/vami/subsequentboot).  You can read a bit more about this process here: https://blogs.vmware.com/vapp/2009/07/selfconfiguration-and-the-ovf-environment.html.

Ultimately, the fix is an easy one – shut down the VM (the vApp Options can only be modified on a powered-off VM), update the vApp Options on the VM and start the VM.  In the vSphere Web Client, Edit Settings of the appliance, switch to the vApp Options tab, expand Networking Properties and update as desired.  Note that this issue is not unique to the vCenter Virtual Appliance (VCSA) – other virtual appliances that rely on the vApp Options properties will be impacted (Log Insight, vCenter Operations, vSphere Replication, vSphere Management Assistant (vMA) and other third party appliances).

As I explained in my primer on analytic troubleshooting, it’s not enough to identify the root cause and stop with a simple fix.  You need to stair-step the solution to figure out what else could be effected by the root cause and to prevent the problem from occurring again.  To prevent future problems, we could simply disable the vApp Options on a VM.  Quick fix, but doing so could have unanticipated consequences depending on how VMware or your ISV are using vApp Options.  Your virtual appliance vendor probably used VMware Studio to create the appliance with settings to pull vApp options into the appliance – you could break dependencies…. You’re probably better off updating your VMs vApp Options and making the task of updating your vApp Options a part of your change management process or run-book as a long-term strategy

If you want to save yourself some clicks, you can automate the process of changing vApp settings if you’re doing it often for several machines.  Official documentation for vApp PropertyInfo settings can be found here https://pubs.vmware.com/vsphere-55/topic/com.vmware.wssdk.apiref.doc/vim.vApp.PropertyInfo.html.  Here is an example of PowerCLI code for issuing a shutdown to a VM and waiting for it to be in a powered-off state, then updating vApp DNS Server settings on that VM, then powering it back on.  Note that this code is specific to my environment any probably will NOT work in yours as the property[] value and key value may be different for each VM.

function Wait-VMPowerState {
<# function from https://powershell.com/cs/media/p/17924.aspx #>
[CmdletBinding()]
Param(
        # The name of a VM
        [Parameter(Mandatory=$true,
                   ValueFromPipelineByPropertyName=$true,
                   Position=0)]
        $VMName,
        # The operation (up or down)
        [Parameter(Mandatory=$true,
                   ValueFromPipelineByPropertyName=$true,
                   Position=1)]
                   [ValidateSet("Up","Down")]
        $Operation
    )
begin{
    $vm = get-vm -Name $vmname
    }
process{
    switch ($operation) {
        down {
                if ($vm.PowerState -eq "PoweredOn") {
                    Write-Verbose "Shutting Down $vmname"
                    $vm | Shutdown-VMGuest -Confirm:$false
                    #Wait for Shutdown to complete
                    do {
                       #Wait 5 seconds
                       Start-Sleep -s 5
                       #Check the power status
                       $vm = Get-VM -Name $vmname
                       $status = $vm.PowerState
                    }until($status -eq "PoweredOff")
                }
                 elseif ($vm.PowerState -eq "PoweredOff") {
                    Write-Verbose "$vmname is powered down"
                }
            }
         up {
                if ($vm.PowerState -eq "PoweredOff") {
                    Write-Verbose "Starting VM $vmname"
                    $vm | Start-VM -Confirm:$false
                    #Wait for startup to complete
                    do {
                       #Wait 5 seconds
                       Start-Sleep -s 5
                       #Check the power status
                       $vm = Get-VM -Name $vmname
                       $status = $vm.PowerState
                    }until($status -eq "PoweredOn")
                }
                 elseif ($vm.PowerState -eq "PoweredOn") {
                    Write-Verbose "$vmname is powered up"
                }
            }
        }
    }
end{
    $vm = Get-VM -Name $vmname
    $vm
    }
}

$newValue = "10.152.190.20,10.152.190.21"
$MyvApp= get-VM -Name "VSOM-VCVA-5.5-01"
$vm = Wait-VMPowerState -VMName $MyvApp -Operation down
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.changeVersion = $MyvApp.ExtensionData.Config.ChangeVersion
$spec.vAppConfig = New-Object VMware.Vim.VmConfigSpec
$spec.vAppConfig.property = New-Object VMware.Vim.VAppPropertySpec[] (1)
$spec.vAppConfig.property[0] = New-Object VMware.Vim.VAppPropertySpec
$spec.vAppConfig.property[0].operation = "edit"
$spec.vAppConfig.property[0].info = New-Object VMware.Vim.VAppPropertyInfo
$spec.vAppConfig.property[0].info.key = 2
$spec.vAppConfig.property[0].info.value = $newValue
$Reconfig = $MyvApp.ExtensionData
$Configtask = $Reconfig.ReconfigVM_Task($spec)
start-sleep -s 5
start-vm -VM $MyvApp -confirm $false

Have anything to add? Know a better way to automate? Share it in the comments below!

*I have been rightly corrected on the proper acronym for the vCenter Server Appliance. Depending on where you read, you may see it called the by VCVA. This is not correct – it should be VCSA.