Update 1: This issue was fixed as of 3/15/2012 in ESXi 5.0 Update 1 per the original knowledge base article: VMware KB 2008144. To download ESXi 5.0/vCenter Server 5.0 Update 1, see the VMware Download Center.
Update 2 (Nov 7 2012): While VMware says the issue was fixed in vSphere 5.0 Update 1, I have not found this to be the case. I’ve had several environments where the bug was still present, and in some cases this nasty bug still appeared when implementing the Failback = NO setting on the vSwitch. Additionally, setting Failback = No may cause some other problems – see my update here for information from Chris Towles.
Update 3 (Nov 8 2012): I received an email from Nick Eggleston about this issue – Nick experienced this problem at a customer site and heard from VMware support that “There is a vswitch failback fix which may be relevant and has not yet shipped in ESXi 5.0 (it postdates 5.0 Update 1 and is currently targeted for fixing in Update 2). However this fix *has* already shipped in the ESXi 5.1 GA release.” Thanks, Nick!
Update 4 (Nov 9 2012): Once I knew what this bug looked like, I’ve found other symptoms that can help you diagnose the issue in your environment. One of the most telling is that the switches (assuming Cisco) carrying your iSCSI traffic may show high Output Queue Drops (OQD) counters when you run a ‘Show Interface Summary’. This is because the packets sent out the wrong vmnic have no business being on the switch they were routed too, so the switch drops the packet. I’ve also noticed that when this bug is causing issues the MAC address table (show mac address table) loses all entries for the iSCSI vmknic’s and in some cases the storage array’s front end ports.
I recently stumbled on two vSphere 5 ESXi networking bugs that I thought I would share. The issues are very similar from a cursory level, but have different symptoms, troubleshooting steps, and implications for your architecture, so I’m going to split the issues into two separate posts. Because troubleshooting these issues was a real pain, I’ll provide some details on how to identify these issues in your environments and wrap up with a third post on what I believe to be some best practices to avoid these same problems and achieve greater redundancy and resiliency in your vSphere environments.
The Problem
Today, we’ll look at an ESXi 5 networking issue that caused massive iSCSI latency, lost iSCSI sessions, and lost network connectivity. I’ve been able to reproduce this issue in several environments, on different hardware configurations. Here’s the background information on how all this started: I upgraded an ESXi 4.1 host to ESXi 5 using vSphere Update Manager (VUM). Note that I did use the host upgrade image that contained the ESXi500-201109001 iSCSI fixes – if you are upgrading to vSphere 5 and have iSCSI in your environment, use this image. Here’s a quick look at how the networking was configured on this host:
The iSCSI networking was configured in a very typical setup, and per best practices, as outline in VMware’s documentation, as well as from many vendors (see EMC’s Chad Sakac’s ‘A Multivendor Post on using iSCSI with VMware vSphere’), with two vmnic uplinks, two vmknics, with one active adapter on the correct layer-2/layer-3 network, and the other unused.
After the upgrade, the standard vSwitch with two vmnics for uplinks (Broadcom NetXtreme II BCM5709 1000Base-T) and two vmknics that serviced the software iSCSI adapter failed to pass traffic (vmkping to the iSCSI targets failed) and could not mount ANY iSCSI LUN’s. VM network, management, and vMotion ports were not affected.
If I let the host sit long enough, it *might* find a couple paths to the storage, but even then performance was deteriorated per the vmkernel.log:
WARNING: ScsiDeviceIO: 1218: Device naa.60026b90003dcebb000003ca4af95792 performance has deteriorated. I/O latency increased from average value of 5619 microseconds to 495292 microseconds.
Troubleshooting
I’m going to dump a whole bunch of my troubleshooting steps on you – hopefully they not only help folks dealing with this particular bug, but help with general network and configuration troubleshooting in VMware vSphere. During troubleshooting, I removed the vmk binding for these two on the iSCSI adapter, removed the software iSCSI Adapter itself, removed the vmknics on the vSwitch, and removed the vSwitch itself. I then recreated the vSwitch, set vSwitch MTU to 9000, recreated two vmk ports, set 9000MTU, assigned IP, and set failover order for multipath iSCSI. I then re-created the software iSCSI adapter and bound the two vmk ports. I was able to pass vmk traffic and mount iSCSI LUN’s. Great – problem solved!?!?! Not so much – I rebooted the host and the problem returned.
Here are my next troubleshooting steps:
- I repeated the procedure above and re-gained connectivity, but the problem returns on subsequent reboots. I can verifiably recreate the problem.
- I verified end-to-end connectivity for other hosts on the same Layer 1, Layer 2, and Layer 3 network as the iSCSI initiator and iSCSI targets.
- I verified the ESXi host’s networking configuration using the vSphere client, double-checking the vSwitch, vmnic uplinks, and vmknic configurations. Everything looked good so I canceled out.
- I then reinstalled ESXi from scratch (maybe something was left over from 4.1 that a clean install would weed out), built up the same configuration, and was again able to re-create the problem.
- I poured over logs (vmkernel.log, syslog.log and storagerm.log primarily). I could see an intermittent loss of storage connectivity, failure to log into the storage targets (duh – there is no connectivity, no vmkping) and high storage latency on hosts where I had rebuilt the iSCSI stack and run a few VM’s.
- I switched out the Broadcom NIC for an Intel NIC (the Broadcom had hardware iSCSI capabilities – I wanted to be sure the hardware iSCSI was not interfering).
- I verified TOE was enabled.
The ‘Ah-Ha’ Moment
Next, I verified the ESXi host’s networking configuration using the vSphere client one more time – the properties of the vSwitch, the properties of the vmkernel (vmk) ports, the manual NIC teaming overrides, IP addressing, etc. Everything looked correct – I MADE NO CHANGES – but when I clicked OK (last time I canceled) to close the vSwitch properties and was greeted with this warning:
Wait a second… I didn’t change anything, why am I being prompted with a you’re ‘Changing an iSCSI Initiator Port Group’ warning? I like to live dangerously, and wanted to see what would happen, so I said ‘Yes’.
Much to my surprise, after only viewing and closing the vSwitch and iSCSI vmk port group settings, I was able to complete a vmkping on the iSCSI-bound vmk’s. And moreover, I completed a Rescan of all storage adapters and my iSCSI LUN’s were found, mounted, and ready for use. Problem solved? Nope. The same ugly issue re-appeared after a reboot.
While the problem wasn’t solved, I now had something to work with. My go-to troubleshooting question “What Changed?” could maybe be answered. Even though I didn’t change anything in the vSwitch Properties GUI, something changed. To see what changed in the background, I compared the output of the following ESXi Shell (or vCLI, or PowerCLI) commands before and after making ‘the change’ happen (by viewing the properties of the vSwitch/vmk ports), but found no changes.
- esxcfg-vswitch -l
- esxcfg-vmknic -l
- esxcfg-nics -l
Then, I made backup copy of esx.conf
cp /etc/vmware/esx.conf /etc/vmware/esx.conf.bak
Then I caused ‘the change’ and then compared checksums using md5sum, but found no differences:
md5sum /etc/vmware/esx.conf /etc/vmware/esx.conf.bak
I compared the running .conf and the backup .conf, but found no differences:
diff /etc/vmware/esx.conf /etc/vmware/esx.conf.bak
Call in Air Support
At this point, I was out of ideas so I called for help: “Hello, 1-866-4VMWARE, option 4, option 2 – help!”
After repeating many of the same troubleshooting steps, the support engineer decided that I had hit on a known, and not yet patched, bug. The details of the bug are included in KB 2008144: Incorrect NIC failback occurs when an unused uplink is present. That’s right – my iSCSI traffic, vmkpings, etc were being sent down the wrong NIC – the UNUSED NIC. Ouch. The bug caused the networking stack to behave in a very unpredictable way, making my troubleshooting steps next to useless, and any other advanced troubleshooting ideas I had (sniffing, logs, etc.)
Once I knew what the issue was, I could see a bit of evidence in the logs:
WARNING: VMW_SATP_LSI: satp_lsi_pathIsUsingPreferredController:714:Failed to get volume access control data for path "vmhba33:C0:T0:L4": No connection
NMP: nmp_DeviceUpdatePathStates:547: Activated path "NULL" for NMP device "naa.60026b90003dcebb0000c7454d5cc946".
WARNING: ScsiPath: 3576: Path vmhba33:C0:T0:L4 is being removed
Notice the NULL path – the path can’t be interpreted correctly when being sent down the wrong (unsued) vmnic that is on a different subnet and VLAN. The gotcha on this issue is that I had followed best practices where applicable, and accepted default settings on the vSwitch and vmknics.
The Quick Fix
VMware KB 2008144 offers two workaround for this bug. The quick fix for the problem is to simply change the Failback setting on either the vSwitch running the software iSCSI vmknic’s to “No” (default is yes), or to change the setting on the vmknic itself if you have other port groups on the vSwitch (such as a VM Network port group to give your guest VM’s access to the iSCSI network).
Changing Failback = No on the iSCSI vmknics and then rescanning the storage adapters fix the glitch immediately.
Architecture Changes
The second workaround from VMware is “Do not have any unused NICs present in the team.”. This translates to a slightly different architecture than that described in many documents. To achieve this workaround, the configuration would have to change to two vSwitches, each with a single vmnic uplink and a single vmk port, bound to the iSCSI adapter. This change does not impact redundancy or availability when compared with the single-vSwitch:two-vmk configuration that I was running with as one of the vmnics was set to unused anyway. This workaround does add a bit more complexity, as there are a few more elements to configure, monitor, manage, and document.
This problem seems to only present itself on vSphere Standard Switches (vSwitch), although I could not get confirmation of this (please post a comment if you know!). Assuming this is true, a vDistributed Switch (vDS) could be used for Software iSCSI traffic. Mike Foley has a write-up on how to migrate iSCSI from a vSwitch to a vDS on his blog here: https://www.yelof.com/?p=72.
A Couple More Notes
My troubleshooting fix of viewing the vSwitch settings and clicking ok seemed to temporarily resolve the issues because it triggered an up/down event on the vmk of the unused uplink. This caused the network stack to re-evaluate paths and start using the correct, Active, uplink.
Note that this problem can occur outside of my iSCSI use case – any vSwitch, Port Group, or VMKNIC with an unused adapter set in the NIC Teaming Failover Order are susceptible to this bug, so watch for it on redundant vMotion networks (vMotion randomly fails), VM Network networks (sudden loss of guest connectivity), or even your management network (hosts fall out of manageability from vCenter, and can’t be contacted via SSH, vSphere client, etc.
Leave a comment if you’ve experienced this bug – your notes on the problem may help others find and fix the issue until VMware releases a fix. I understand that a fix for this particular bug is not due out until at least vSphere 5 Update 1.
I’ll have another (shorter) writeup on the 2nd networking bug I found in ESXi 5 later in the week – check back here for a link once it is published.
Another victim of this bug! 🙁
I blogged about this issue back in October, which spawned KB article 2008144.
https://wahlnetwork.com/2011/10/11/explicit-failover-shenanigans-when-upgrading-to-esxi-5-x/
Hi Josh,
You’re right. Setting one vmnic to unused for each iSCSI VMkernel port group is a standard and recommended practice. This is how we’ve implemented over a dozen datacenters recently. One difference I see in your config and ours is that you’re running iSCSI traffic over two different subnets/VLANs while our iSCSI config, while exactly the same as what you first described above, differs in one (perhaps not) small way: our iSCSI traffic is over a single subnet/VLAN. So if this happens in our environments, we may lose half the paths to our storage but still keep connectivity because traffic still traverses the same subnet even though it’s using a different vmknic. Does this sound about right to you?
All the best,
Mike Brown
https://VirtuallyMikeBrown.com
Hi Josh,
You’re right. Setting one vmnic to unused for each iSCSI VMkernel port group is a standard and recommended practice. This is how we’ve implemented over a dozen datacenters recently. One difference I see in your config and ours is that you’re running iSCSI traffic over two different subnets/VLANs while our iSCSI config, while exactly the same as what you first described above, differs in one (perhaps not) small way: our iSCSI traffic is over a single subnet/VLAN. So if this happens in our environments, we may lose half the paths to our storage but still keep connectivity because traffic still traverses the same subnet even though it’s using a different vmknic. Does this sound about right to you?
All the best,
Mike Brown
https://VirtuallyMikeBrown.com
Great writeup 🙂
Great writeup 🙂
Hi Josh,
i found a different workaround for a problem similar to this, i couldn’t ping any vmkping after reboot and in the event log i found errors like this:
Login to iSCSI target iqn.1986-03.com.hp:storage.p2000g3.1130132c47 on vmhba39 @ vmk2 failed. The iSCSI initiator could not establish a network connection to the target.
I noticed that if you make 2 vmkernel on 2 vswitch with different subnets these errors are logged again only if you manually bind the 2 vmk in the software iscsi adapter. If you don’t make the bind all works fine and the errors disappeared (it’s not the best practice but the bind seems to make itself automatically).
I’ll be waiting for an answer to this solution from vmware support.
I’ll let you know!
Thanks!
Hi Josh,
i found a different workaround for a problem similar to this, i couldn’t ping any vmkping after reboot and in the event log i found errors like this:
Login to iSCSI target iqn.1986-03.com.hp:storage.p2000g3.1130132c47 on vmhba39 @ vmk2 failed. The iSCSI initiator could not establish a network connection to the target.
I noticed that if you make 2 vmkernel on 2 vswitch with different subnets these errors are logged again only if you manually bind the 2 vmk in the software iscsi adapter. If you don’t make the bind all works fine and the errors disappeared (it’s not the best practice but the bind seems to make itself automatically).
I’ll be waiting for an answer to this solution from vmware support.
I’ll let you know!
Thanks!
Thanks for this write up. Will have to keep in mind as I’m planning my environment upgrade now. Though, like Mike Brown, my two vmkernels are on the same subnet, where yours are on different ones. Wonder if that affects it.
Thanks for this write up. Will have to keep in mind as I’m planning my environment upgrade now. Though, like Mike Brown, my two vmkernels are on the same subnet, where yours are on different ones. Wonder if that affects it.
Out of curiosity, are your 2 iSCSI VMK ports redundant links to the same storage? If so, I would wonder why they’re in two different /24 subnets… If so, why the need for the subnetting? Unless you’re pushing insane amounts of iSCSI traffic, I couldn’t see why you would need/want to segregate your iSCSI traffic from itself (to two different controllers on the same SAN perhaps, but again, why?).
I have this exact same config, minus the different subnets on my storage (2 hosts going to a single Dell EQL SAN), and haven’t run into this issue at all…
Scott,
Equallogic is configured as a single subnet because you cluster shelf’s together not typical of an iSCSI SAN.
The lower end Dell PowerVault is configured to have each port on a different subnet.
Out of curiosity, are your 2 iSCSI VMK ports redundant links to the same storage? If so, I would wonder why they’re in two different /24 subnets… If so, why the need for the subnetting? Unless you’re pushing insane amounts of iSCSI traffic, I couldn’t see why you would need/want to segregate your iSCSI traffic from itself (to two different controllers on the same SAN perhaps, but again, why?).
I have this exact same config, minus the different subnets on my storage (2 hosts going to a single Dell EQL SAN), and haven’t run into this issue at all…
Scott,
Equallogic is configured as a single subnet because you cluster shelf’s together not typical of an iSCSI SAN.
The lower end Dell PowerVault is configured to have each port on a different subnet.
Same shit here:
Two DELL R415s w. BCM 5709 dual port on a DELL MD3200i. Each of the 4 ports of the storage has a separate subnet; 192.168.130|131|132|133.101. DELL support told me that was necessary…I still wonder why, but didn’t ask (no time and the DELL guy wasn’t quite enjoying the talk). First I had a single vSwitch on each host with a separate VMk for each port of the BCM. I matched the subnets to those on the storage (I bound both vmnics to each VMk, one active, one unused, as it should work out the box):
ESX0,PORT0 > 192.168.130.100 > 192.168.130.101
ESX0,PORT1 > 192.168.131.100 > 192.168.131.101
ESX1,PORT0 > 192.168.132.100 > 192.168.132.101
ESX1,PORT1 > 192.168.133.100 > 192.168.133.101
I bound the VMks to the corresponding ports 0/1 of the iSCSI HBA (no automatic binding happened):
ESX0, vmhba32 > VMk ESX0,PORT0
ESX0, vmhba33 > VMk ESX0,PORT1
ESX1, vmhba32 > VMk ESX0,PORT0
ESX1, vmhba33 > VMk ESX0,PORT1
Then I configured the Dynamic discovery on each hba port only for the corresponding subnet:
ESX0, vmhba32 > 192.168.130.101
ESX0, vmhba33 > 192.168.131.101
ESX1, vmhba32 > 192.168.132.101
ESX1, vmhba33 > 192.168.133.101
Result: One path to each LUN the host was allowed to discover. After a restart *boom* it took ESX1 about 20 minutes to boot, logging the same “no network connection” errors as stated here. Also the storage was lost and couldnt be rediscovered.
After searching my head bloody, I found this…and had to discover that the “Quick Fix” wouldn’t work for me. So I changed my config to have a separate vSwitch with single VMks for every BCM port / storage subnet. The binding of the VMks happened automatically when opening the HBA’s properties. After configuring the dynamic discovery at least all the LUNs and pathes were found almost immediately. At the moment I’m checking reboots…okay: it’s taking 8 minutes instead of more than 20.
Thank you guys: great article, great comments!
You need to bind your iSCSI vmkernels to your iscsi init… Yes – all four.
esxcli swiscsi nic add -n vmk# -d vmhba##
You’ll need to SSH into the server to do that, and su – (change to roots home as that is the only place you can issue that command)
The reason they are on separate subnets is because that is simply the way the MD’s handle multipathing. no more, no less.
Also, dynamic discovery does what is called a iSCSI send request. You technically only need to put in one IP – your long boot times could be attributed to this. The storage array will report back all available iSCSI ports. Put in one from each controller on separate NICs (or subnets if you have one nic for iSCSI)
Just an opinion – using broadcoms as HBA’s is horrible. Lots of problems. Use the software iSCSI init.
For you equallogic guys that need to change the network binding order so you have unused nics, you should also change the lowest vmk# port to be a storage heartbeat and have access to both nics. Then create another “real” VMK for iSCSI… storage heartbeat has to be the lowest numbered vmk.
Same shit here:
Two DELL R415s w. BCM 5709 dual port on a DELL MD3200i. Each of the 4 ports of the storage has a separate subnet; 192.168.130|131|132|133.101. DELL support told me that was necessary…I still wonder why, but didn’t ask (no time and the DELL guy wasn’t quite enjoying the talk). First I had a single vSwitch on each host with a separate VMk for each port of the BCM. I matched the subnets to those on the storage (I bound both vmnics to each VMk, one active, one unused, as it should work out the box):
ESX0,PORT0 > 192.168.130.100 > 192.168.130.101
ESX0,PORT1 > 192.168.131.100 > 192.168.131.101
ESX1,PORT0 > 192.168.132.100 > 192.168.132.101
ESX1,PORT1 > 192.168.133.100 > 192.168.133.101
I bound the VMks to the corresponding ports 0/1 of the iSCSI HBA (no automatic binding happened):
ESX0, vmhba32 > VMk ESX0,PORT0
ESX0, vmhba33 > VMk ESX0,PORT1
ESX1, vmhba32 > VMk ESX0,PORT0
ESX1, vmhba33 > VMk ESX0,PORT1
Then I configured the Dynamic discovery on each hba port only for the corresponding subnet:
ESX0, vmhba32 > 192.168.130.101
ESX0, vmhba33 > 192.168.131.101
ESX1, vmhba32 > 192.168.132.101
ESX1, vmhba33 > 192.168.133.101
Result: One path to each LUN the host was allowed to discover. After a restart *boom* it took ESX1 about 20 minutes to boot, logging the same “no network connection” errors as stated here. Also the storage was lost and couldnt be rediscovered.
After searching my head bloody, I found this…and had to discover that the “Quick Fix” wouldn’t work for me. So I changed my config to have a separate vSwitch with single VMks for every BCM port / storage subnet. The binding of the VMks happened automatically when opening the HBA’s properties. After configuring the dynamic discovery at least all the LUNs and pathes were found almost immediately. At the moment I’m checking reboots…okay: it’s taking 8 minutes instead of more than 20.
Thank you guys: great article, great comments!
You need to bind your iSCSI vmkernels to your iscsi init… Yes – all four.
esxcli swiscsi nic add -n vmk# -d vmhba##
You’ll need to SSH into the server to do that, and su – (change to roots home as that is the only place you can issue that command)
The reason they are on separate subnets is because that is simply the way the MD’s handle multipathing. no more, no less.
Also, dynamic discovery does what is called a iSCSI send request. You technically only need to put in one IP – your long boot times could be attributed to this. The storage array will report back all available iSCSI ports. Put in one from each controller on separate NICs (or subnets if you have one nic for iSCSI)
Just an opinion – using broadcoms as HBA’s is horrible. Lots of problems. Use the software iSCSI init.
For you equallogic guys that need to change the network binding order so you have unused nics, you should also change the lowest vmk# port to be a storage heartbeat and have access to both nics. Then create another “real” VMK for iSCSI… storage heartbeat has to be the lowest numbered vmk.
Postscriptum:
I’m turning to the software initiator with Jumbo Frames because cloning a 100GB VM takes about an hour. The host CPU performance this will cost is negligible as there will ever run no more than 4 VMs on the cluster.
Consequence of this “practice”: Will never again buy dependent iSCSI HBAs like the BCM 5709s.
Postscriptum:
I’m turning to the software initiator with Jumbo Frames because cloning a 100GB VM takes about an hour. The host CPU performance this will cost is negligible as there will ever run no more than 4 VMs on the cluster.
Consequence of this “practice”: Will never again buy dependent iSCSI HBAs like the BCM 5709s.
Thanks Christoph, exactly what i was looking for.
For me the quick workaround the author suggested was working for reestablishing the iSCSI connection, but deep in my guts i got a feeling that i still don’t have real multipathing working. The initiator never used the second path as i would have liked to in my RR-path selection policy (was only seeing the Controllerport from the first subnet).
I went on creating another vSwitch with the second vmk Port and removed the initiator bindings under the software initiator completely – et voila – i see both different initiator IPs in path selection policy and can use RR again.
So can we conclude this? –
If you use different subnets for each storage controller port and previously have bound them with multiple vmk’s under the same vswitch with nic teaming (that is under vsphere 4, maybe earlier), you need to seperate your vmk Ports to different vSwitches and get rid of nic teaming in order to make them properly work under vsphere 5.
Thanks Christoph, exactly what i was looking for.
For me the quick workaround the author suggested was working for reestablishing the iSCSI connection, but deep in my guts i got a feeling that i still don’t have real multipathing working. The initiator never used the second path as i would have liked to in my RR-path selection policy (was only seeing the Controllerport from the first subnet).
I went on creating another vSwitch with the second vmk Port and removed the initiator bindings under the software initiator completely – et voila – i see both different initiator IPs in path selection policy and can use RR again.
So can we conclude this? –
If you use different subnets for each storage controller port and previously have bound them with multiple vmk’s under the same vswitch with nic teaming (that is under vsphere 4, maybe earlier), you need to seperate your vmk Ports to different vSwitches and get rid of nic teaming in order to make them properly work under vsphere 5.
Ouch…just hit this myself, after updating the Broadcom network drivers on a set of 4 HP hosts. Funny thing is, I had read your blog post about a week ago and thought to myself “funny, I haven’t run into this since our ESXi 5 upgrades…”. Caused crazy latency and trespasses of the VMFS LUNs on our SAN, such that the storage processors started rebooting (ack!). I managed to set the Failover policy to No and restart the ESXi management services, and everything stabilized. Luckily this was in our DR site so the net effect was minimal. Thanks for documenting this!
Excellent write up! I pulled out most of my hair trying to figure this out over many weeks. I eventually created two vSwitches and everything worked so I left it that way. Glad to find this article though.
Excellent write up! I pulled out most of my hair trying to figure this out over many weeks. I eventually created two vSwitches and everything worked so I left it that way. Glad to find this article though.
Thanks, just what i needed!
I am having a very simalar problem at work. My error is lost access to Volume due to connectivity issues. We have been having this issue since we installed esxi 5. Although I don’t have a unused vmnic listed, I am still going to try this work around. If it works, I will let you all know.
I am having a very simalar problem at work. My error is lost access to Volume due to connectivity issues. We have been having this issue since we installed esxi 5. Although I don’t have a unused vmnic listed, I am still going to try this work around. If it works, I will let you all know.
My issue was due to a routing issue. 🙂
My issue was due to a routing issue. 🙂
I have need running this on ESX4.1 on single vswitch, two kernel ports with one active, one unused and NO failover all along but on 5.0 each iscsi rescan was producing failures. Tried reconfiguring with two vswitches, one active NIC each and no unused but its doing the same thing. The only way I am able to clear these is to remove the ACL with iscsi initiators off each volume on the storage array. Ultimately, all the volumes are connected with or without the ACL but i would really like to find a proper way to clear rescan errors. No chap used. Vmware support from HP are saying there’s nothing they can do since from it’s technically not broke. I’m checking with the array vendor. Array and iscsi adapters are all up-to-date firmware and drivers wise and are on vmware HCL. Perhaps its time to roll back to 4.1 🙂
Zhanna – drop down to Layers 1, 2 and 3. Make sure that basics like spanning tree root bridge elections are done and correct, flow control is enabled on physical switches, etc. You might also consider increasing the login timeout on the software iSCSI adapter (look in advanced settings) to give the array a bit more time to respond.
I have need running this on ESX4.1 on single vswitch, two kernel ports with one active, one unused and NO failover all along but on 5.0 each iscsi rescan was producing failures. Tried reconfiguring with two vswitches, one active NIC each and no unused but its doing the same thing. The only way I am able to clear these is to remove the ACL with iscsi initiators off each volume on the storage array. Ultimately, all the volumes are connected with or without the ACL but i would really like to find a proper way to clear rescan errors. No chap used. Vmware support from HP are saying there’s nothing they can do since from it’s technically not broke. I’m checking with the array vendor. Array and iscsi adapters are all up-to-date firmware and drivers wise and are on vmware HCL. Perhaps its time to roll back to 4.1 🙂
Zhanna – drop down to Layers 1, 2 and 3. Make sure that basics like spanning tree root bridge elections are done and correct, flow control is enabled on physical switches, etc. You might also consider increasing the login timeout on the software iSCSI adapter (look in advanced settings) to give the array a bit more time to respond.
Fixed yesterday (3/15/2012) !! Hooray!
https://kb.vmware.com/kb/2008144
Thanks, Robert! I’ve updated the post to reflect the patch.
Fixed yesterday (3/15/2012) !! Hooray!
https://kb.vmware.com/kb/2008144
Thanks, Robert! I’ve updated the post to reflect the patch.
Strange, I did not have this problem until applying patch 5.01. In the syslog I can see iscsi bindings on my mgt nics which of course fail because mgt is not on the iscsi vlan. Trying to get back up and running. esxcli, df are failing as well on the local console.
Josh, great article. But I did not see these issues UNTIL going to 5.01. My lab is down and the iSCSI is not connecting. I see in the log where ISCSID is trying to connect to iSCSI over my mgt NIC. Trying to figure out what happened. Also /scratch has been relocated to a VMFS volume to preserve log files – thought that was a best practice, but in hind sight maybe not so go. esxcli is failing, df is failing. See: https://communities.vmware.com/message/2020956#2020956
Thanks for the article
Josh, great article. But I did not see these issues UNTIL going to 5.01. My lab is down and the iSCSI is not connecting. I see in the log where ISCSID is trying to connect to iSCSI over my mgt NIC. Trying to figure out what happened. Also /scratch has been relocated to a VMFS volume to preserve log files – thought that was a best practice, but in hind sight maybe not so go. esxcli is failing, df is failing. See: https://communities.vmware.com/message/2020956#2020956
Thanks for the article
Josh, great article, but we had the same problems at some sites with different storage vendors, DELL and HP. Both where having the same problems: WARNING: …… performance has deteriorated. I/O latency increased from average value of …. microseconds to …. microseconds. Also the problem is on ESX 5 and ESX 5 U1. I solved the problem by disabling DelayedAck. See https://communities.vmware.com/thread/328081?tstart=0. Does someone tried this?
Regards Robert
Thanks for the comment, Robert. The DelayedAck issue is separate (but no less important), and tends to affect 10Gb iSCSI connections. The issue is discussed in this VMware KB article: https://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1002598&sliceId=2&docTypeID=DT_KB_1_1&dialogID=324174407&stateId=0%200%20321023763
Josh, great article, but we had the same problems at some sites with different storage vendors, DELL and HP. Both where having the same problems: WARNING: …… performance has deteriorated. I/O latency increased from average value of …. microseconds to …. microseconds. Also the problem is on ESX 5 and ESX 5 U1. I solved the problem by disabling DelayedAck. See https://communities.vmware.com/thread/328081?tstart=0. Does someone tried this?
Regards Robert
Thanks for the comment, Robert. The DelayedAck issue is separate (but no less important), and tends to affect 10Gb iSCSI connections. The issue is discussed in this VMware KB article: https://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1002598&sliceId=2&docTypeID=DT_KB_1_1&dialogID=324174407&stateId=0%200%20321023763
Joshua, that is correct, but only when i disable DelayedAck the issue is resolved. It was tested in the following configurations: DELL R610 Host (ESX 5.0 U1) HP P4500 MultiSite SAN, HP DL380 G7 (ESX 5.0) DELL EQL P6000. This issue is not solved with U1.
Joshua, that is correct, but only when i disable DelayedAck the issue is resolved. It was tested in the following configurations: DELL R610 Host (ESX 5.0 U1) HP P4500 MultiSite SAN, HP DL380 G7 (ESX 5.0) DELL EQL P6000. This issue is not solved with U1.
Have you try to make 2 different vswitch for iscsi network, 1 only active vnic assigned to 1 only vswitch?
Have you try to make 2 different vswitch for iscsi network, 1 only active vnic assigned to 1 only vswitch?
Hi all,
great job and happy to find a site with so many expert.
First of all, I apologize for my bad English, i try to explain..
I just set up a site for my customare. 3 Esxi 5.01 host with all recent update loaded.
Netapp Fas2240 cluster, used with iscsi. Network for iscsi dedicated build on HTC5500 (hp4800) dual unit switch in stack. Vmk is built on 4 vswitch each one with a single adapter and single vmk. Storage side, network is a 4gbps link with ip balancing.
With Esxtop i have confirmation of correct network load distribution on all the 4 ESXi uplink, i’ve also followed best practice from Netapp regasrding aggretates, volume and implementation with vmware.
…All works fine, all seems to be ok, but randomly i got the warning on esxi host “performance has deteriorated. I/O latency increased from average value of …. microseconds to …. microseconds”…
I’ve double check everithing, nothing appear to be wrong or misconfigured..
Last my test is about disabling DelayedAck.
i keep informed about results.
Thanks to all.
Max
Hello MaxZana,
Did “DelayedAck” fix your problem ? We have very similar issue with HP P2000 G3 iscsci en VMware ESXi 5.0 Update 1…
Thank you very much,
Florian
DelayedAck is certainly an issue in 1gb iSCSI environments, but was not related to this specific issue.
Hi Florian,
after i have disabled DelayedAck, errors appears less frequently, but not disappeared at all…
Better, yes, but not solved.
Max
Hi all,
great job and happy to find a site with so many expert.
First of all, I apologize for my bad English, i try to explain..
I just set up a site for my customare. 3 Esxi 5.01 host with all recent update loaded.
Netapp Fas2240 cluster, used with iscsi. Network for iscsi dedicated build on HTC5500 (hp4800) dual unit switch in stack. Vmk is built on 4 vswitch each one with a single adapter and single vmk. Storage side, network is a 4gbps link with ip balancing.
With Esxtop i have confirmation of correct network load distribution on all the 4 ESXi uplink, i’ve also followed best practice from Netapp regasrding aggretates, volume and implementation with vmware.
…All works fine, all seems to be ok, but randomly i got the warning on esxi host “performance has deteriorated. I/O latency increased from average value of …. microseconds to …. microseconds”…
I’ve double check everithing, nothing appear to be wrong or misconfigured..
Last my test is about disabling DelayedAck.
i keep informed about results.
Thanks to all.
Max
Hello MaxZana,
Did “DelayedAck” fix your problem ? We have very similar issue with HP P2000 G3 iscsci en VMware ESXi 5.0 Update 1…
Thank you very much,
Florian
DelayedAck is certainly an issue in 1gb iSCSI environments, but was not related to this specific issue.
Hi Florian,
after i have disabled DelayedAck, errors appears less frequently, but not disappeared at all…
Better, yes, but not solved.
Max
I have the same issue. I’m using separate links from the host to the iscsi target that go in different switches. One VLAN on each switch with a subnet off /24 dedicated on each VLAN for iscsi traffic. Rebooted the server, could not see any storage. Tried to add it again and i had to wait 5 minutes to see the disks again. I must say i use shared storage ( a computer running Centos 6.2 and 5x2TB Seagate in RAID5).
Another problem is the performance of the ESXi 5 host. VM copy data on their virtual HDD with 30-35MB/s. The performance is abysmal. This is my home lab on which i do tests.
I have the same issue. I’m using separate links from the host to the iscsi target that go in different switches. One VLAN on each switch with a subnet off /24 dedicated on each VLAN for iscsi traffic. Rebooted the server, could not see any storage. Tried to add it again and i had to wait 5 minutes to see the disks again. I must say i use shared storage ( a computer running Centos 6.2 and 5x2TB Seagate in RAID5).
Another problem is the performance of the ESXi 5 host. VM copy data on their virtual HDD with 30-35MB/s. The performance is abysmal. This is my home lab on which i do tests.
Hi Guys
Has anyone requested a patch for this new KB from Vmware?
https://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2019936&sliceId=1&docTypeID=DT_KB_1_1&dialogID=389996029&stateId=1 0 389994495
Regards
Hi Guys
Has anyone requested a patch for this new KB from Vmware?
https://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2019936&sliceId=1&docTypeID=DT_KB_1_1&dialogID=389996029&stateId=1 0 389994495
Regards
Thanks so much for this. I had a rock solid ESX 5.0 server running for months (with VNXe storage as the back-end) and had to reboot a few weeks ago and have been puzziling over where my data stores had gone. Some with with EMC support and a reboot and they were back, making me falsely believe the problem was sorted. Coming back from vacation to find one of the images offline was a shock.
Mind if I link to this from the spiceworks vmware forum?
Thanks so much for this. I had a rock solid ESX 5.0 server running for months (with VNXe storage as the back-end) and had to reboot a few weeks ago and have been puzziling over where my data stores had gone. Some with with EMC support and a reboot and they were back, making me falsely believe the problem was sorted. Coming back from vacation to find one of the images offline was a shock.
Mind if I link to this from the spiceworks vmware forum?
Just wanted follow up and let everyone know to if they still have failback set to no that it causes the problem of ESX using the Unused Nic.
Vmware ESXi 5.0 update 1 Sending Traffic out unused VMNIC because of Failback
https://www.christowles.com/2012/09/vmware-esxi-50-update-1-sending-traffic.html
Also if you setup your Equallogic with MEM 1.1.1 or 1.1.0 it still sets the Failback causing the issue.
Good work, Chris. Back and forth on this one, with no clear guidance from VMware. Thanks for taking the time to write up your findings and sharing the update here!
Thank you Josha, without your post i’d still be here scratching my head. Here’s hoping VMware resolves this soon.
Just wanted follow up and let everyone know to if they still have failback set to no that it causes the problem of ESX using the Unused Nic.
Vmware ESXi 5.0 update 1 Sending Traffic out unused VMNIC because of Failback
https://www.christowles.com/2012/09/vmware-esxi-50-update-1-sending-traffic.html
Also if you setup your Equallogic with MEM 1.1.1 or 1.1.0 it still sets the Failback causing the issue.
Good work, Chris. Back and forth on this one, with no clear guidance from VMware. Thanks for taking the time to write up your findings and sharing the update here!
Thank you Josha, without your post i’d still be here scratching my head. Here’s hoping VMware resolves this soon.
I’m a bit confused … this is an old issue, but I still get it with ESX 5.1 and vcenter 5.1 …
shouldn’t it be corrected ?
I get plenty of these, every 30 sec.
naa.60060160b060280068e4a585c895e111
performance has deteriorated. I/O latency
increased from average value of 17798
microseconds to 357022 microseconds.
Strange thing, it’s a clean install, and the only VM is my vcenter.
I did the “failback => no” trick, didn’t work :/
Any idea ?
Jack – it is possible that you’ve got a different issue going on if you’ve implemented the workarounds. Check your multipathing config and your storage array health. Make sure you have enough spindles to satisfy your workload IOPS requirement. Feel free to reach out to me at Clearpath if you need help troubleshooting the problem further.
I’m a bit confused … this is an old issue, but I still get it with ESX 5.1 and vcenter 5.1 …
shouldn’t it be corrected ?
I get plenty of these, every 30 sec.
naa.60060160b060280068e4a585c895e111
performance has deteriorated. I/O latency
increased from average value of 17798
microseconds to 357022 microseconds.
Strange thing, it’s a clean install, and the only VM is my vcenter.
I did the “failback => no” trick, didn’t work :/
Any idea ?
Jack – it is possible that you’ve got a different issue going on if you’ve implemented the workarounds. Check your multipathing config and your storage array health. Make sure you have enough spindles to satisfy your workload IOPS requirement. Feel free to reach out to me at Clearpath if you need help troubleshooting the problem further.