vCenter Virtual Appliance Network Settings Reverting

In one of my lab environments my vCenter Server Appliance (VCVA VCSA*) kept reverting its DNS server settings after I changed them in the VCSA web interface (https://:5480).  After changing the Preferred DNS Server and Alternate DNS Server settings to use my Active Directory Domain Controller IPs and then rebooting the appliance, the DNS settings were removed and the Preferred DNS Server setting was reverted to the setting present before I updated it.

vCenter Virtual Appliance Network Settings

Losing my DNS Server settings caused me all sorts of issues – my vCenter Server couldn’t find ESXi hosts by name, AD authentication failed, and general mayhem ensued.  I reviewed some logs on the appliance and poked around in /etc/sysconfig/network/config, but nothing jumped out at me as being wrong.  I even went so far as to manually update /etc/resolv.conf with the correct settings but after a reboot the DNS server settings were lost.  I needed some help to figure this one out, so I visited my Universal Troubleshooting Accelerator Dispensing Machine, poured a pint of my Rye Porter homebrew and took a step back to think outside the box (or outside the VCSA as it were).

Universal Troubleshooting Accelerator Machine - KegeratorIf the problem is not coming from inside the appliance, it’s got to be coming from somewhere outside the appliance.  Either my incorrect DNS is getting set by DHCP or some other mechanism.  I’m using a Static IP so DHCP isn’t handing out a DNS server.  Think, dummy, think (absorb, beer, absorb)!  Ah ha!  I didn’t deploy the VCSA – it was deployed by the data center administrator to a separate management cluster before I had built an AD or DNS infrastructure on the hosts my VCSA was managing.  Because the administrator didn’t know about my non-existent DNS servers, he used a default setting when he set the properties of the vApp while deploying the OVF.  The OVF vApp settings were injected into the appliance and applied at each reboot, overwriting my settings.

I always like to dig a bit deeper when I run into a problem, so let’s look briefly at how vApp Options are applied to better understand this mechanism.  When a virtual appliance is imported via OVF, it either used VMware Tools to read the vApp Options through the vami service (see /opt/vmware/share/vami) or a virtual CD-ROM was mounted that contained an XML file with the vApp Options.  On the vCenter Virtual Appliance (based on SUSE Linux Enterprise 11), the vApp Options are written to /opt/vmware/etc/vami/ovfEnv.xml.  The vami service reads the settings in from the XML and applies them at subsequent boot (via /opt/vmware/share/vami/subsequentboot).  You can read a bit more about this process here: http://blogs.vmware.com/vapp/2009/07/selfconfiguration-and-the-ovf-environment.html.

Ultimately, the fix is an easy one – shut down the VM (the vApp Options can only be modified on a powered-off VM), update the vApp Options on the VM and start the VM.  In the vSphere Web Client, Edit Settings of the appliance, switch to the vApp Options tab, expand Networking Properties and update as desired.  Note that this issue is not unique to the vCenter Virtual Appliance (VCSA) – other virtual appliances that rely on the vApp Options properties will be impacted (Log Insight, vCenter Operations, vSphere Replication, vSphere Management Assistant (vMA) and other third party appliances).

VMware vApp Options Networking Properties

As I explained in my primer on analytic troubleshooting, it’s not enough to identify the root cause and stop with a simple fix.  You need to stair-step the solution to figure out what else could be effected by the root cause and to prevent the problem from occurring again.  To prevent future problems, we could simply disable the vApp Options on a VM.  Quick fix, but doing so could have unanticipated consequences depending on how VMware or your ISV are using vApp Options.  Your virtual appliance vendor probably used VMware Studio to create the appliance with settings to pull vApp options into the appliance – you could break dependencies…. You’re probably better off updating your VMs vApp Options and making the task of updating your vApp Options a part of your change management process or run-book as a long-term strategy

If you want to save yourself some clicks, you can automate the process of changing vApp settings if you’re doing it often for several machines.  Official documentation for vApp PropertyInfo settings can be found here http://pubs.vmware.com/vsphere-55/topic/com.vmware.wssdk.apiref.doc/vim.vApp.PropertyInfo.html.  Here is an example of PowerCLI code for issuing a shutdown to a VM and waiting for it to be in a powered-off state, then updating vApp DNS Server settings on that VM, then powering it back on.  Note that this code is specific to my environment any probably will NOT work in yours as the property[] value and key value may be different for each VM.

function Wait-VMPowerState {
<# function from http://powershell.com/cs/media/p/17924.aspx #>
[CmdletBinding()]
Param(
        # The name of a VM
        [Parameter(Mandatory=$true,
                   ValueFromPipelineByPropertyName=$true,
                   Position=0)]
        $VMName,
        # The operation (up or down)
        [Parameter(Mandatory=$true,
                   ValueFromPipelineByPropertyName=$true,
                   Position=1)]
                   [ValidateSet("Up","Down")]
        $Operation
    )
begin{
    $vm = get-vm -Name $vmname
    }
process{
    switch ($operation) {
        down {
                if ($vm.PowerState -eq "PoweredOn") {
                    Write-Verbose "Shutting Down $vmname"
                    $vm | Shutdown-VMGuest -Confirm:$false
                    #Wait for Shutdown to complete
                    do {
                       #Wait 5 seconds
                       Start-Sleep -s 5
                       #Check the power status
                       $vm = Get-VM -Name $vmname
                       $status = $vm.PowerState
                    }until($status -eq "PoweredOff")
                }
                 elseif ($vm.PowerState -eq "PoweredOff") {
                    Write-Verbose "$vmname is powered down"
                }
            }
         up {
                if ($vm.PowerState -eq "PoweredOff") {
                    Write-Verbose "Starting VM $vmname"
                    $vm | Start-VM -Confirm:$false
                    #Wait for startup to complete
                    do {
                       #Wait 5 seconds
                       Start-Sleep -s 5
                       #Check the power status
                       $vm = Get-VM -Name $vmname
                       $status = $vm.PowerState
                    }until($status -eq "PoweredOn")
                }
                 elseif ($vm.PowerState -eq "PoweredOn") {
                    Write-Verbose "$vmname is powered up"
                }
            }
        }
    }
end{
    $vm = Get-VM -Name $vmname
    $vm
    }
}

$newValue = "10.152.190.20,10.152.190.21"
$MyvApp= get-VM -Name "VSOM-VCVA-5.5-01"
$vm = Wait-VMPowerState -VMName $MyvApp -Operation down
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.changeVersion = $MyvApp.ExtensionData.Config.ChangeVersion
$spec.vAppConfig = New-Object VMware.Vim.VmConfigSpec
$spec.vAppConfig.property = New-Object VMware.Vim.VAppPropertySpec[] (1)
$spec.vAppConfig.property[0] = New-Object VMware.Vim.VAppPropertySpec
$spec.vAppConfig.property[0].operation = "edit"
$spec.vAppConfig.property[0].info = New-Object VMware.Vim.VAppPropertyInfo
$spec.vAppConfig.property[0].info.key = 2
$spec.vAppConfig.property[0].info.value = $newValue
$Reconfig = $MyvApp.ExtensionData
$Configtask = $Reconfig.ReconfigVM_Task($spec)
start-sleep -s 5
start-vm -VM $MyvApp -confirm $false

Have anything to add? Know a better way to automate? Share it in the comments below!

*I have been rightly corrected on the proper acronym for the vCenter Server Appliance. Depending on where you read, you may see it called the by VCVA. This is not correct – it should be VCSA.

Comments

  1. Thanks for the post. We encountered a similar issue in our Lab.

  2. If you cannot update the vApp options with out the VCSA being off and in order to update the settings you need to be in the web client I don’t see how to accomplish the task. Once you shutdown the VCSA you can no longer access the web client. Can the settings be edited and then the VM shutdown and rebooted?

    • Hi Chris – sounds like a Catch-22. When you import the VCSA it is at VM version (hardware compatibility level) 7, so you can manage it with the older C# Windows client. You would connect directly to the ESXi host where the VCSA is currently running, shut the VCSA down, then make the changes in the old client and start VCSA back up. Hope this helps!

      • Actually, you can’t connect directly to the ESXi host and edit there, as the host is unaware of vApp properties. The properties are actually stored in the vCenter database. I just went through this myself, as our AD/DNS servers were migrated to a new IP range and needed to be updated in the vApp properties.

        The good news is that you CAN edit the vApp properties while the VM is running, but only with the web client.
        I just confirmed this by updating my vApp properties, and then rebooting the VCSA. To watch the boot I connected directly to the ESXi host via the desktop client. After it finished booting I verified the vApp
        properties in the VM settings, the network settings via the appliance management interface, and the actual
        configs on the VM itself. This was all with version 5.51b.

        What I haven’t figured out how to do yet, is to update the application version info the vApp reports. You can
        not edit that info via the web client, and the latest update to VCSA did not update it. The VCSA is now at
        version 5.5.0.10200 Build 1891314, but the vApp info still reports it as 5.5.0.10100 Build 1750781. The
        appliance management interface reports the correct version, so this is merely cosmetic.

  3. Mitch Lyman says:

    Just had a similar issue when I upgraded to VCSA 5.5.0.1b (build 1891314). Everything worked fine with VCSA 5.5.0.1a until the upgrade to 1b. Then, when trying to log in using the web client, I received a “Failed to connect to VMware Lookup Service ssl certificate verification failed”. Turns out the vApp application hostname did not match the VCSA FQDN. Corrected the vApp hostname and all is fine.

    • Mitch Lyman says:

      Clarification: The vApp hostname was corrected in the “vApp Options” dialog box within vCenter.

Trackbacks

  1. […] So I suggest you get the networking info correct when you deploy the vApp.    Read this for more info and what actually […]

Drop a comment below:

You may also like-

Performing a P2V on a Physical vCenter ServerPerforming a P2V on a Physical vCenter ServerBy Jason Shiplett Migrating a vCenter Server instance from physical to virtual can be done in a couple of different ways: Manually migrating vCenter ... vCenter Crashes After Applying ESXi Patch ESXi410-201010401-SGvCenter Crashes After Applying ESXi Patch ESXi410-201010401-SGMy last post described a problem I experienced with VMware HA after upgrading to vSphere 4.1.  Here is my experience with a similar issue ... Free VMware View Load Balancer Using SUSE Studio and HAProxyFree VMware View Load Balancer Using SUSE Studio and HAProxyI've had several customers ask me what options they have for providing redundancy / high availability / load balancing for their VMware View Connection Servers and ...
%d bloggers like this: