XenServer Issues with SMB over PV NIC
-
@DustinB3403 said:
It's an issue specific to the VM. As jshiers stated he doesn't have the same issue on other VM's using the same pNIC.
Which points away from the PV driver rather than towards it, I think.
-
Yes the XenTools are up to date with the latest and greatest after all the host updates were applied. Based on my testing I am thinking that my issue is specific to Server 2008R2 and its interaction with the PV NIC but I was hoping to get some direction on how to figure that out specifically. The VM I had previously been having issues with was taking 6-11 hours to copy 1.5 GB from my IBM Mainframe SMB share to itself and then processing that into a SQL database. After turning off NIC Offloading that VM has run that process in less than 10 min. The software package I am have trouble with now also use SMB and SQL but is sending data to the clients via SQL. When watching Processor Monitor you can see SMB crank up as part of the system process and get north of 500 meg but SQL never gets above 50kbps. Same VMs turn Offloading back on and SQL cranks up north of 100 meg but SMB never gets over 70kbps.
I am nearing a point where I am about ready to run AD Prep and move the forest up to a 2012R2 functional level and then run an in place upgrade on one or more of these servers and see if that helps at all.
-
Did you have these issues prior to updating the host(s)?
-
Yes with the VM that would not copy at a reasonable rate. That is why I moved to 6.5SP1 from 6.2 base. It seemed like everything was ok when we first installed but over time (I am guessing with windows updates) it just kept getting slower and slower. I traced thru everything I could think of from an IO perspective moved the hosts from RAID 6 to RAID 10, on the VMs upped the vCPU counts, and upped the RAM. When none of that worked I purchased Citrix support and they came up with the NIC offloading.
-
Have you attempted a 2008R2 server as a clean install and see if these issues persist?
You mentioned that you upgraded from 6.2 to 6.5SP1 because of these issues, but these issues could have been caused because of the 6.2 PV drivers.
-
@jshiers said:
Yes with the VM that would not copy at a reasonable rate. That is why I moved to 6.5SP1 from 6.2 base. It seemed like everything was ok when we first installed but over time (I am guessing with windows updates) it just kept getting slower and slower. I traced thru everything I could think of from an IO perspective moved the hosts from RAID 6 to RAID 10, on the VMs upped the vCPU counts, and upped the RAM. When none of that worked I purchased Citrix support and they came up with the NIC offloading.
If it is tied to a Windows update that means it is less likely to be driver related.
-
What happens if you switch that 2008R2 VM from a PV NIC to a Non-PV Nic? Can you do that in XenServer?
-
I can't prove the performance issue is tied to Windows updates. What I can prove is how the VMs are preforming today and relate that back to the fact that 6-12 months ago I didn't seem to have these issues with the same software packages. Performance started going down gradually as we didn't really notice issues until it "suddenly" occurred to us that things were not running well. In the overnight I have spun up a 2012R2 VM to do some testing with. I am also wondering if I need to flash the BIOS on the underlying hardware as if the software layer was updated and is expecting the NIC to perform a specific way but the BIOS isn't doing that then maybe my issue is there?
-
@jshiers the BIOS would effect every system you have, so I would update (to update) but don't expect there to be a performance change. As your other VM's aren't have any issues.
With your 2012R2 VM monitor it's performance and see if there is a difference. It's not Apples to Apples, as this issue is occurring with your 2008R2 server.
Do you have a VM backup that you can light up of the 2008R2 server from several months back and see if the issue persist?
-
@dafyre You'd have to install dedicated hardware drivers to the VM rather than the xentool.iso drivers.
Which I'm not certain what would happen with it in this case.
Additionally I don't think it would address the issue. The VM changed, and performance was degraded.
Speaking of, how much free space does the vHDD have on it? I've seen windows servers come to a crawl when the drive is full. Worth a look.
-
I have made sure all the VMs have at least 20 gig of free vHDD space to eliminate any issue with drive space. We also have used a Fluke MicroScanner to check out our physical cables and found a couple of suspect cables that we replaced as well.
Here are some test results, SMB file copy from a 2008R2 VM that has NIC offloading turned on to the new 2012R2 box with no PV tools so its using the base Windows Realtek driver and got 24 Mbps. Not great but much better than the 64-72 Kbps I get between two 2008R2 boxes with offloading turned on. Same test from 2008R2 VM with NIC offloading turned off and I get the same 24 Mbps. All of these VMs are on the same host.
Ok installed the PV tools on the 2012R2 server. Copy from a 2008R2 VM with NIC Offloading ON and the 350 meg file copied in a flash. Resource Monitor showed 2 Gbps. Copy from the VM with NIC Offloading OFF and the copied at 225 Mbps.
Maybe I should just start do inplace upgrades on my servers!!!!
-
Upgrading to the newest release of the Server OS never really hurts, so long as your systems are capable of running on that OS.
I'm glad you were able to remove trouble patch cables, and your testing with 2012R2 seems to have resolved the issue.
Are you able to perform an inline upgrade in a test environment so you're not effecting your production systems?
-
@jshiers said:
I can't prove the performance issue is tied to Windows updates. What I can prove is how the VMs are preforming today and relate that back to the fact that 6-12 months ago I didn't seem to have these issues with the same software packages. Performance started going down gradually as we didn't really notice issues until it "suddenly" occurred to us that things were not running well. In the overnight I have spun up a 2012R2 VM to do some testing with. I am also wondering if I need to flash the BIOS on the underlying hardware as if the software layer was updated and is expecting the NIC to perform a specific way but the BIOS isn't doing that then maybe my issue is there?
Unlikely that you need to flash the BIOS but that does not mean that it is not a good idea, but unlikely to help.
That performance has gone down slowly means that drivers are also unlikely to be involved.
-
@jshiers said:
Maybe I should just start do inplace upgrades on my servers!!!!
That's an option for sure. Snap them, upgrade, test and see.
