Monday, December 5, 2011

ESX(i) 4.x Corrupt/Bad switch port causes the vmnic to drop packets

Hardware Configuration:
Server Model: DL 380 G5
Storage :
Network : 2 onboard nics+3 add on cards 2 ports per card
total=8 ports
procurve 2810

Problem description/symptoms/errors:
vmnic0, vmnic1 and vmnic6 are assigned to vswitch0.
The default nic teaming policy is to keep the vmnic1 as standby but by doing so he gets an alert stating that the network redundancy is lost.
System board has been replaced for the onboard nic issue.

Actions taken:
There were no VMs in the host.
There were 3 nics for the vswitch 2 meant for vm traffic.
Moved one non critical VM to the problematic host1.
Removed all the nics from the vswitch2.
Added vmnic0 from vswitch0 to vswitch2 and the VM in it was able to ping other VMs.
Moved the vmnic0 back to vswitch0 and the vswitch2 became empty.
Added vmnic6 from vswitch0 to vswitch2 and the VM in it was able to ping other VMs.
Moved the vmnic6 back to vswitch0 and the vswitch2 became empty.
Added vmnic1 from vswitch0 to vswitch2 and the VM in it was able to ping other VMs but it was timing out intermittently with considerable amount of packet loss.
When added another known good vmnic it was able to ping continuously.
Updated the driver
but no go.
When added vmnic1 only to the vswitch2 with the VM in it the VM was still timing out intermittently when pingig other VMs and it continued doing the same even with another vmnic2 being added.
Removed vmnic1 and retained vmnic2 only on vswitch2 with vm and the vm pinged continuously.
Issue with vmnic1.
The system board has been replaced for the onboard nic issue yesterday.
Suggested caller to try new cables for both the onboard nics but no go.
Told customer to switch the cables of vmnic0 and vmnic1 and the VM in the vswitch2 started pinging continuously.
Issue followed the port of the cable(switch port on which the cables are connected to the vmnic0 and vmnic1).
Now the switchport connecting to the vmnic0 is faulty.
Redirected customer to procurve team.