KVM Poor Man Replication HA
-
So I did ask around before posting this and I did receive warnings, but I have been testing this and it is very simple and dumb yet works (tested only with Linux VM Distro). sharing it here so you guys can bash me about my method, as well as enlighten me.
So you have your 2 KVM hosts okay + 1 Fedora machine with Virt Manager
You create all the VMs on the KVM and call it primary, let us call it KVM 1
the second KVM 2 will be just the slave incase KVM 1 drops.So what we do is schedule a script to :
-
(optional )Stop Software VM activity like DB or WebServer sometime when it is not being utilized at midnight for example, systemctl stop httpd ..etc
-
Freeze the FileSystem (guest agent needs to be installed.)
virsh domfsfreeze VMNAME -
Rsync the VM image to another KVM host (KVM 2) periodically each night.
rsync --progress --inplace -h
or
rsync --progress --inplace -h -W (safer) -
Prepare previously in KVM 2 the VM xml (RAM/CPU)and make it same as the one in KVM1, also copy the MAC address. but keep VM2 shutdown.
-
You can script Fedora Machine to keep pinging KVM 1 or VM 1 and if it fails to receive thus it is down, so if that happens KVM 2 will virsh start VM2 , which will have the same everything even the same internal IP.
What do you think ? Too cheezy ? Does it show how much I am afraid to play with GLusterFS?
btw the tags are acting weird, whenever I type one it removes itself.
-
-
@emad-r said in KVM Poor Man Replication HA:
What do you think ? Too cheezy ? Does it show how much I am afraid to play with GLusterFS?
Yes, it does. GlusterFS is rock solid after you get the initial configuration done.
What you described will work, and would be a good way to make backups. GlusterFS isn't really going to be a backup.
-
@emad-r said in KVM Poor Man Replication HA:
So I did ask around before posting this and I did receive warnings, but I have been testing this and it is very simple and dumb yet works (tested only with Linux VM Distro). sharing it here so you guys can bash me about my method, as well as enlighten me.
So you have your 2 KVM hosts okay + 1 Fedora machine with Virt Manager
You create all the VMs on the KVM and call it primary, let us call it KVM 1
the second KVM 2 will be just the slave incase KVM 1 drops.So what we do is schedule a script to :
-
(optional )Stop Software VM activity like DB or WebServer sometime when it is not being utilized at midnight for example, systemctl stop httpd ..etc
-
Freeze the FileSystem (guest agent needs to be installed.)
virsh domfsfreeze VMNAME -
Rsync the VM image to another KVM host (KVM 2) periodically each night.
rsync --progress --inplace -h
or
rsync --progress --inplace -h -W (safer) -
Prepare previously in KVM 2 the VM xml (RAM/CPU)and make it same as the one in KVM1, also copy the MAC address. but keep VM2 shutdown.
-
You can script Fedora Machine to keep pinging KVM 1 or VM 1 and if it fails to receive thus it is down, so if that happens KVM 2 will virsh start VM2 , which will have the same everything even the same internal IP.
What do you think ? Too cheezy ? Does it show how much I am afraid to play with GLusterFS?
btw the tags are acting weird, whenever I type one it removes itself.
So the issue with rsyncing for anything other than a snapshot is the data will be different.
For example:
you rsync VM1 at 5AM to KVM2. At 3PM KVM1 goes down. VM1 will start up on KVM2, but any data written between 5AM and 3PM is lost. I wouldn't recommend rsync for anything other than copying disks as a backup or snapshot.
Gluster will do the real time replication you need for this.
-
-
gluster has been choosen upstream and by read hat for their hyperconverged stuff, so it should be ok, even if use it without ovirt seems a bit complex. But it is just a feeling, I never used it, just had a look to the how-tos
-
@matteo-nunziati said in KVM Poor Man Replication HA:
gluster has been choosen upstream and by read hat for their hyperconverged stuff, so it should be ok, even if use it without ovirt seems a bit complex. But it is just a feeling, I never used it, just had a look to the how-tos
Yeah, if you don't break it down it can look like a bear, but it's really not that difficult. I haven't spun up a cluster using it in years, otherwise I'd do a how-to here.
-
@travisdh1 said in KVM Poor Man Replication HA:
@matteo-nunziati said in KVM Poor Man Replication HA:
gluster has been choosen upstream and by read hat for their hyperconverged stuff, so it should be ok, even if use it without ovirt seems a bit complex. But it is just a feeling, I never used it, just had a look to the how-tos
Yeah, if you don't break it down it can look like a bear, but it's really not that difficult. I haven't spun up a cluster using it in years, otherwise I'd do a how-to here.
I think that I have one.
-
@scottalanmiller said in KVM Poor Man Replication HA:
@travisdh1 said in KVM Poor Man Replication HA:
@matteo-nunziati said in KVM Poor Man Replication HA:
gluster has been choosen upstream and by read hat for their hyperconverged stuff, so it should be ok, even if use it without ovirt seems a bit complex. But it is just a feeling, I never used it, just had a look to the how-tos
Yeah, if you don't break it down it can look like a bear, but it's really not that difficult. I haven't spun up a cluster using it in years, otherwise I'd do a how-to here.
I think that I have one.
https://mangolassi.it/topic/8619/installing-gluster-on-centos-7
Yes you do, and I read it. And looking forward to playing with it. But the thing is my simple method works if you can afford some time lost between both servers, some small workloads can especially if there are separate file\DB backups.Have fun all at MangoCon
What I really want to say and express, its very fun to have full fledged OS to play with that have an hypervisor role, is it very hard for me to go back to ESXi standalone, having a full OS (Centos + KVM) makes a world of possibilities.
-
@travisdh1 said in KVM Poor Man Replication HA:
@matteo-nunziati said in KVM Poor Man Replication HA:
gluster has been choosen upstream and by read hat for their hyperconverged stuff, so it should be ok, even if use it without ovirt seems a bit complex. But it is just a feeling, I never used it, just had a look to the how-tos
Yeah, if you don't break it down it can look like a bear, but it's really not that difficult. I haven't spun up a cluster using it in years, otherwise I'd do a how-to here.
what made me think about complaxity is that they suggest to build only a single brick on top of only a single thin provisioned LVM volume on top of only a single thin provisioned volume.
I mean: everytime you need a brick you need to create a lot of stuff... maybe a shell script can make it more streamlined but if you use plain LVM as virtual disk for your VM it means a lot of stuff everytime you create a VM. boring
-
I deployed it for an HCI cluster (well tried). The issue was it locked read only on files on re-sync operations and certain things (which blew them up). I'm sure it's some ways, but Gluster wasn't designed for VM storage (Neither was Ceph, but it was designed to perform better at writes than gluster). The original core use cases were scaled out file services to get around limitations of scaling of XFS/EXT4 etc and do a little DRAM caching for read/metadata on nodes. It was a poor man's Isilon more than a low latency transactional filer, or general purpose block platform.
The other thing is to be careful with Gluster and quorum. It's REALLY easy to setup. It's also REALLY easy to setup something like what this guy had that was working till it didn't.
-
@matteo-nunziati RedHat's building a GUI for this stuff was my understanding. The other thing to be aware of is running bare metal requires you deal with things like Driver/Firmware lifecycle for Drives, Controller Backplanes and work with the OEM on issues like SAS buffering incompatibility, liberal interpretations of the T10 SPEC, or good old SATA tunneling protocol locking an entire PHY because of a SMART poll and crashing the controller. There be dragons in running storage devices in pass thru mode (especially modern flash devices in dense configs). If you go down this route I'd want to have RedHat or SuSE backing me from a support stance.
-
@aaronstuder said in KVM Poor Man Replication HA:
Ceph would be a better option
I think Redhat knows why they aren't using it. Its core feature set was high-speed transactional object and not block (the Block have was explicitly not production ready when they acquired it). While it is architecturally better for the block, It's dynamic failure handling is non-existent (Gluster is a bit better here because of the layering it tends to sit on and use of Local RAID which was what RedHat was packaging for HCI). Ceph is the #1 reason I've seen for OpenStack deployments to crash and burn in a disastrous way. With Careful testing, validation, lifecycle control, and prescriptive deployments it's powerful. It's just... Kinda a glass cannon, and you'll see it openly cursed at OpenStack conferences and presentations (with people talking about using CINDER + something commercial like SolidFire etc).
-
@john-nicholson said in KVM Poor Man Replication HA:
@aaronstuder said in KVM Poor Man Replication HA:
Ceph would be a better option
I think Redhat knows why they aren't using it. Its core feature set was high-speed transactional object and not block (the Block have was explicitly not production ready when they acquired it). While it is architecturally better for the block, It's dynamic failure handling is non-existent (Gluster is a bit better here because of the layering it tends to sit on and use of Local RAID which was what RedHat was packaging for HCI). Ceph is the #1 reason I've seen for OpenStack deployments to crash and burn in a disastrous way. With Careful testing, validation, lifecycle control, and prescriptive deployments it's powerful. It's just... Kinda a glass cannon, and you'll see it openly cursed at OpenStack conferences and presentations (with people talking about using CINDER + something commercial like SolidFire etc).
And CephFS still isn't production ready. Plus it requires whole separate metadata servers. RedHats hyperconverged stuff and storage stuff both use Gluster. It's easier to maintain and less prone to explode in your face.
Plus Ansible has Gluster modules
-
@john-nicholson said in KVM Poor Man Replication HA:
I deployed it for an HCI cluster (well tried). The issue was it locked read only on files on re-sync operations and certain things (which blew them up). I'm sure it's some ways, but Gluster wasn't designed for VM storage (Neither was Ceph, but it was designed to perform better at writes than gluster). The original core use cases were scaled out file services to get around limitations of scaling of XFS/EXT4 etc and do a little DRAM caching for read/metadata on nodes. It was a poor man's Isilon more than a low latency transactional filer, or general purpose block platform.
The other thing is to be careful with Gluster and quorum. It's REALLY easy to setup. It's also REALLY easy to setup something like what this guy had that was working till it didn't.
I initially tried Gluster with VM replication and found it kind of slow. What I switched to was having (some of) the VMs do replication themselves with Gluster. That way it's easy to add replicates or distribute the load instead of messing with the physical storage. It's worked pretty well so far (as exactly a poor man's Isilon).
Gluster doesn't perform super well with a ton of tiny files (at least didn't used to) and putting it on RAID 5 was not an awesome idea. Writes would have been painfully slow.
-
http://www.phoronix.com/scan.php?page=news_item&px=oVirt-4.1.5-Released
Good timing for you on this.
-
Thread resurrected !!!
I see, interesting . Regarding Ovirt + GLusterFs my update on this is that did learn glustering and it was easy to perform. I didnt apply it in production or VMs. I did use Ovirt a month ago and it was very slow web ui experience.
I should write a thread about my and Gluster.
