In one of our datacentres we have suddenly started having issues with Vmotion. When we migrate any VM from one host to another, it drops off the network for up to 2m20s. Sometimes it works absolutely fine, sometimes the dropout is not as long.
The configuration of the hosts is 2x 10Gb NICs connected to a standard VSwitch, to cover MGMT, Vmotion and VM Traffics. Each of the 10Gb NICs on the host connects to a separate Nexus2000 switch FEX’d out of a NEXUS5000 switch, connected to Cisco 4500 cores. I have used esxtop to check which vmnic the VM is registered to, and which one it moves to on the target host. Testing with two hosts the VM eventually can ping out on all 4 NICs (2 NICs on 2 hosts) so I know the VLAN is trunking fine. Additionally at the same time VMs on the same VLAN on the target host can still ping out, so I know the ports aren’t flapping. The same behaviour is observed whether the vmnic chosen on VMotion happens to be on the same switch, or whether it registers to the opposite switch.
I got the ARP table on the physical switches when Vmotioning and when we get the failure the MAC is not re-registering on the new host for the same time (up to 2m20s)
There are no errors in vmkernel log either, but I suspect it’s a switch issue. I got a physical VMware host and tested moving the cable from one NIC to the other and again observed the MAC taking 2m20s to register on the new port.
We are using ESXi 5.0 Update 3
Happens on various VMs, running Windows Server 2008 and 2012 using either E1000 or VMXNET3.
It doesn't happen in another datacentre with the same build and configuration of VMware.
Any ideas?