ESXi 6.0 and network performance issue
I have had a few customers that had a performance issue, when running in certain combination:
- ESXi 6.0
- Windows 2012 / Windows 8 or later
- Virtual hardware 11 / Compatibility 6.0 or later
- VMXNET3 network card 1.6.6.0 (this is part of newer VMware Tools) or later
There is VMware KB on this, that have a solution: https://kb.vmware.com/kb/2129176
I have used the alternative solution:
You can disable LRO/RSC for all virtual machines on an ESXi host using:
esxcli system settings advanced set -o /Net/Vmxnet3SwLRO -i 0 esxcli system settings advanced set -o /Net/Vmxnet3HwLRO -i 0
Note:This will disable LRO for all virtual machines on the ESXi host. Virtual machines will need to be powered off and back on or vmotioned between hosts for the changes to take effect.
Notes (March 6, 2017):
The Article says that this has to do with Jumbo frames, I have see this in configuration without jumbo frames.
It also says SQL, I have see this with other application.
It also says it solved in 6.0 Update 2, But I have see the problem after upgrading to this, with Veeam Proxy server take backup thru NBD. The solution is the same.
Notes (February 23 , 2018):
I have had some customers that is running 6.5 Update 1, that is seeing this problem again, the solution is the same, here is a script for setting this setting on all hosts i a vCenter.
$vcenter = Connect-viserver <vCenter server>
$vmhosts = get-vmhost
foreach ($vmhost in $vmhosts) {
Get-AdvancedSetting -Entity $vmhost -Name "Net.Vmxnet3SwLRO" | Set-AdvancedSetting -Value "0" -Confirm:$false
Get-AdvancedSetting -Entity $vmhost -Name "Net.Vmxnet3HwLRO" | Set-AdvancedSetting -Value "0" -Confirm:$false
}
Disconnect-viserver $vcenter
Use of this at your own risk. remember that the VM’s need to be rebooted or vMotioned to another host to take affect.
open an SR?
No not yet, I’m going to do it when I get time.
Hi Allan,
I came across this issue to.
I have tried disabling LRO on per machine basis with mixed results. From my research it only effects Broadcom cards (of which there are many).
The commands you suggest:
esxcli system settings advanced set -o /Net/Vmxnet3SwLRO -i 0
esxcli system settings advanced set -o /Net/Vmxnet3HwLRO –i 0
(reboot vm)
Do they 100% work for resolving this issue? Is performance much effected?
Loving your blog, so much useful info here.
Dan
I haven’t checked if it’s only effects Broadcom. But I don’t see at all my customers, and they run a kinds of hardware.
The command have solved different network performance problems, and i can give a big jump en performance.
The Veeam Proxy went fra 900 Kb/s to 180 Mb/s,
Hi Allan
I seem to have all these conditions in my environment but im not sure if i have a performance issue or not, is there a way to check easy?
Regards
Thomas
of the customers where I have seen the problem, users complained about poor performance of their ERP / CRM systems
Allright, i checked the original KB article and saw that 6.0 Update 2 customer should not be affected so i should be in the clear
Thanks for a good blog 🙂
That is an impressive change in speed.
You don’t need to reboot the host for those commands to take effect. Just rebooting VM is enough?
No, just set the advanced settings on all hosts in the cluster, and do a vMotion of the affected VMs.