Need to Improve Disk Utilization on XenServer 7.2
-
Crossposting on behalf of @krisleslie
I’m using a Dell Precision 490
Two Xeon CPUs quad core
32 GB RAM
1 TB 7200 RPMXenServer is installed on the same SR with thin provisioning enabled.
If I run more than 2 vms at same time the io is horrible.
I have two identical Windows 2016 VMs setup with 2 vcpu and 2 gb ram and 32 GB HDD
-
Is it only a single 7200 RPM drive? No RAID?
-
If IOPS drop when there are more workloads added, and none of the workloads are specifically IOPS intensive, this tells us that the source of the issue is contention. When multiple workloads share a single storage pool, not only do they have to split the available resources in half (same as with CPU or RAM), but with mechanical disks we also face contention as each asks the drive to do something that is not in the interest of the other workload - moving the read arm into a position far away from the data of the other workload. So IOPS don't just get sliced in half from the sharing, but the drives become less efficient overall.
This is compated in modern storage through cache mechanisms that are used to absord the contention and queue reads and writes to mitigate that effect while supplying some data via the cache for increased speed overall.
-
If there is only a single disk and no RAID, remember that a 7200 RPM drive is very slow even compared to laptop standards today. SSDs are the norm for desktop and laptop usage, and they have tens of thousands of IOPS, compared to the 100-125 IOPS of a 7200 RPM drive. With two Windows VMs trying to share that single drive, and if contention was zero, then you'd expect each to get the equivalent of a 3600 RPM drive (slower than anything made in the last decade or two, I've never seen one so slow.) Add in contention and it could cut that in half again, easily.
-
What kind of cache do you have on the storage system?
-
Yes I could pop in a second drive.
-
@krisleslie said in Need to Improve Disk Utilization on XenServer 7.2:
Yes I could pop in a second drive.
Going to RAID 1, that would double your read IOPS (RIOPS) but keep your write IOPS (wiops) steady the same as they are now. There are always reads, normally way more reads than writes, so this would certainly help and possibly fix the problem completely. Also staggering boot ups can help, give one VM time to get up and running before the other kicks off.
Without any RAID layer, I assume there is no cache mechanism as nearly all systems from hardware RAID to software RAID put their cache systems into their RAID and adding it separately is not trivial or obvious. So having a single slow mechanical drive without the contention absorbtion cache would be expected to cause IOPS to plummet with multiple VMs.
-
I assume that this is just a lab box? It might be worth getting a consumer SSD for this if the disk speed is an issue at all.
-
Talking offline, @krisleslie has SSD available. This, combined with moving to RAID 1, should fix the problems
- SSD typically have 10K - 100K IOPS per drive. 7200 RPM SATA have ~100 IOPS per drive. So this is a 100x - 1,000x time improvement in starting IOPS.
- SSD are electrical, not mechanical, and do not have a reduction in total available IOPS from contention. So instead of 100 IOPS dropping to 50-80 total IOPS split in two, it would remain the same total.
- RAID 1 would double the Read IOPS so most operations would have twice the available IOPS as a single SSD and write IOPS would be less likely to wait on Read operations.
- My adding a RAID layer, like MD RAID, there is the chance to add RAM cache on top of the RAID subsystem to absorb contention and improve IOPS for items in the cache.
-
With the recent news that Amazon is abandoning Xen, the concensus is that its time is really over. Citrix has really been crippling XenServer for about a year now. So a lot of people are moving to KVM. KVM is getting all of the love and attention these days. It might be worth switching now as this is a new install.
-
KVM also fully supports MD RAID, as does Xen, but XenServer officially doesn't support software RAID at all, which is stupid and ridiculous. You can make it work just fine, but they try to make it hard.
-
Fedora 26 makes a perfect platform for KVM. And Fedora 27 is releasing in just a couple of days.
-
@scottalanmiller wow if Amazon is moving away that’s HUGE. What’s that going to do to XO?!?!
-
@krisleslie said in Need to Improve Disk Utilization on XenServer 7.2:
@scottalanmiller wow if Amazon is moving away that’s HUGE. What’s that going to do to XO?!?!
XO was on here this week discussing that a lot. They are mostly still focused on XS, but definitely looking into how they can hook into KVM.
-
@scottalanmiller
So it sounds like I would do an install of Fedora as a server not desktop, to strip away the GUI. Then install kvm? What about management tools? -
@krisleslie said in Need to Improve Disk Utilization on XenServer 7.2:
@scottalanmiller
So it sounds like I would do an install of Fedora as a server not desktop, to strip away the GUI. Then install kvm? What about management tools?You can also install the GUI on the server and have local management tools. Obviously managing purely remotely is better. But as this is a desktop anyway, local management tools are not out of the question and you can switch later once you are comfortable with it. There is no lock in to your GUI or tools choices like with Hyper-V.
-
Check out this thread...
https://mangolassi.it/topic/14944/manage-kvm-through-cockpit/
-
-
-
Just more reason I like open source I think I’m gonna switch to kvm at home
