Hal Rosenberg has released a document to the VMware Communities titled “Performance Troubleshooting for VMware vSphere 4”. Hal’s document utilizes a flow chart approach in order to identify specific performance issues. For each problem covered, it includes a discussion of the possible root-causes and solutions. It is a must read for anyone administering a vSphere environment. Thanks Hal. You can find his post and download link here

     My team was deploying a new SharePoint portal for a customer and were seeing some strange behavior from some of our VMs. Our environment consisted of 3 BL685 servers running ESX and 9 VMs ( I know this is overkill, but our customer was very wary of the portal performance in a virtual environment, and our hope was to introduce a platform that could lead them into further virtualization and server consolidation). We were receiving some rather “interesting” errors while adding domain users to the local administrator group of some of the machines. While doing a “check name”, AD would return the account name along with some non-printable characters ( blocks, smiley face, etc…). We were also having issues copying files to the VM through UNC. Some would copy, others would not stating “source no longer available”. We were not seeing this behavior across all the VMs, which was disturbing as they were all deployed from the same template. We did manage to narrow it down to guests running on just one of the hosts.

     My initial thought was the possibility of bad memory installed on this host. After running some memory tests and the eventual replacement, our errors continued. We then replaced the host with another blade and swapped the hard drives so we wouldn’t need to reinstall ESX. After powering up the host and migrating a couple of VMs back, we were astonished that the issue still remained. It was then that we reinstalled ESX as we thought that was the only thing left.

   After reinstalling ESX, it was then that we found some additional problems. We added the host to the cluster and were attempting to configure HA. This would fail at random intervals with often different errors including, path not found, SSL errors, and others. It was at this point my attention turned to the network. This server was configured with 2 NICs with all network trunked and vlan tagged. I disabled one of the NICs and all traffic continued to work. I switched over to the remaining NIC and everything came to a halt. Finally, it looked like we were narrowing down the cause. When the blade was swapped out, the NICs were moved from the original. We swapped out the NIC and everything worked as advertised.

Although the initial errors did not point to the network, after all other troubleshooting did not result in success the NICs were the only thing left.