offline, air-gapped backups / backup rotation (looking for hardware & ideas)
-
@dave247 You'll have a hard time doing a RAID 1 replication of a 16TB drive in less than 24 hours. You have to beat 185 MB/s continuously.
Worm tape sounds like the best option. Then you can just do a full backup and increments. Nobody can overwrite the data or erase it. It's standard LTO tape drives but with worm tapes.
Dell PowerVault TL1000 with LTO8 drive and LTO8 worm media will get the job done. Impossible to ransomware. No need to remove any tapes or do anything until they are full. One tape = 12TB of data or ~30TB of compressed data. Tape drive will compress and encrypt in real-time.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
Whatever you do, mucking with the RAID is not what you want to be doing. That is something that would guarantee data loss due to someone entering a setting wrong.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
SSD is faster, and that helps, for sure. But the real issue is the physical connections and the RAID mechanism, not the drives themselves. Physical drives are a perfectly valid media for your use case. It's RAID being used as an archival mechanism rather than as a disaster avoidance mechanism that causes the problems both in software and in hardware.
-
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
Whatever you do, mucking with the RAID is not what you want to be doing. That is something that would guarantee data loss due to someone entering a setting wrong.
I wouldn't call it mucking with RAID. Its just drive rotations.
-
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
SSD is faster, and that helps, for sure. But the real issue is the physical connections and the RAID mechanism, not the drives themselves. Physical drives are a perfectly valid media for your use case. It's RAID being used as an archival mechanism rather than as a disaster avoidance mechanism that causes the problems both in software and in hardware.
Maybe I will just have to set up a network repository and simply plug the network cable in to let backup file copy to sync, then disconnect. That would probably be the easiest way to be honest.
I just wanted some mechanism that forced us to always have a full backup of data sitting offline/air-gapped... but F it lol
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
Whatever you do, mucking with the RAID is not what you want to be doing. That is something that would guarantee data loss due to someone entering a setting wrong.
I wouldn't call it mucking with RAID. Its just drive rotations.
Each time you add/remove a drive from the array, you chance clicking the wrong drive, the wrong action, etc. I'd call it unnecessary, mucking about where you shouldn't, and very risky as a few of the more friendly terms.
Why do you feel like the removable drives need to be a part of the RAID array?
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
SSD is faster, and that helps, for sure. But the real issue is the physical connections and the RAID mechanism, not the drives themselves. Physical drives are a perfectly valid media for your use case. It's RAID being used as an archival mechanism rather than as a disaster avoidance mechanism that causes the problems both in software and in hardware.
Maybe I will just have to set up a network repository and simply plug the network cable in to let backup file copy to sync, then disconnect. That would probably be the easiest way to be honest.
I just wanted some mechanism that forced us to always have a full backup of data sitting offline/air-gapped... but F it lol
Yeah, it’s called tape. And it’s $8k price tag.
-
One of my clients does what the OP wants.
They bought 5 single drive NAS boxes... the backup software writes to the designated drive each night.
In the morning, they unplug it and take it home...Not great but it is cheap, In comparison
-
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
Whatever you do, mucking with the RAID is not what you want to be doing. That is something that would guarantee data loss due to someone entering a setting wrong.
I wouldn't call it mucking with RAID. Its just drive rotations.
Each time you add/remove a drive from the array, you chance clicking the wrong drive, the wrong action, etc. I'd call it unnecessary, mucking about where you shouldn't, and very risky as a few of the more friendly terms.
Not if you follow the correct steps and know what you are doing. Its not anything terribly difficult.
Why do you feel like the removable drives need to be a part of the RAID array?
Because the RAID1 mirror would write the changes to the swapped disk.
The idea is to have Drive 1 and Drive X in a RAID1 mirror, where Drive X = both drives 2 and 3 which would be swapped out daily.
-
Nothing is quite as simple as a usb drive that has last week's backups....
/Sarcasm
-
@dashrender said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
One of my clients does what the OP wants.
They bought 5 single drive NAS boxes... the backup software writes to the designated drive each night.
In the morning, they unplug it and take it home...Not great but it is cheap, In comparison
That's a little different, right? Not using the RAID, but abusing the hot swap bays.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
The idea is to have Drive 1 and Drive X in a RAID1 mirror, where Drive X = both drives 2 and 3 which would be swapped out daily.
The idea makes sense, just none of the parts are designed to be used in that fashion.
-
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
Each time you add/remove a drive from the array, you chance clicking the wrong drive, the wrong action, etc. I'd call it unnecessary, mucking about where you shouldn't, and very risky as a few of the more friendly terms.
That too, it does depend on fencing against human error a bit more than a normal system. There are better, more efficient, more reliable ways to handle this where that isn't the case.
-
To do basically the same thing, what you want is a NAS with local storage (with or without RAID, in this case you are without RAID even though you are using RAID, so no need to have RAID at all) and having a hot swap drive in a mechanism meant to handle this, like a USB style drive, and a script that does a file copy of just the backup, not a block mirror of the drives, to copy the backup to the second drive.
-
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
To do basically the same thing, what you want is a NAS with local storage (with or without RAID, in this case you are without RAID even though you are using RAID, so no need to have RAID at all) and having a hot swap drive in a mechanism meant to handle this, like a USB style drive, and a script that does a file copy of just the backup, not a block mirror of the drives, to copy the backup to the second drive.
Actually, I just remembered that with the Highly Reliable system, they had Windows software RAID 1 which did a good job in this kind of setup. Yes, its not perfect or ideal, but given that I have stated that I already have thorough backups and am only seeking to add offline/air-gapped copies as an added precaution, I don't think its that big an issue.
-
@dashrender said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
SSD is faster, and that helps, for sure. But the real issue is the physical connections and the RAID mechanism, not the drives themselves. Physical drives are a perfectly valid media for your use case. It's RAID being used as an archival mechanism rather than as a disaster avoidance mechanism that causes the problems both in software and in hardware.
Maybe I will just have to set up a network repository and simply plug the network cable in to let backup file copy to sync, then disconnect. That would probably be the easiest way to be honest.
I just wanted some mechanism that forced us to always have a full backup of data sitting offline/air-gapped... but F it lol
Yeah, it’s called tape. And it’s $8k price tag.
Yeah I'm not doing tape and I think the alternate mechanism I proposed is roughly fine, depending on how its approached.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
Actually, I just remembered that with the Highly Reliable system, they had Windows software RAID 1 which did a good job in this kind of setup.
RAID 1 has to do a block copy of a disk in use and track changes. It works, but isn't an efficient way to do this kind of workload. And most RAID has to mirror the entire drive, not just the portions with data. So in some cases it can be pretty dramatically slower than alternative methods.
ANd it's not like you want a mirror in the end. You just want a file copy. So the overhead of the RAID system doesn't provide benefits.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
Yes, its not perfect or ideal, but given that I have stated that I already have thorough backups and am only seeking to add offline/air-gapped copies as an added precaution, I don't think its that big an issue.
The biggest issue is the hardware. How do you plan to connect and reconnect drives because no business class system that does RAID is meant for this to happen. So you either use business class devices that get abused and aren't expected to remain reliable. Or you use consumer gear to get the hotswap portion but don't have overall good hardware.
It can be done, everyone suggests doing it, and there is a reason that it's considered a horrible idea that should never be done. Trust me, there are simple, better ways to do something similar, rule this out and never think about it again. RAID is close to, but not the actual correct tool. The idea of copying the data to another drive is good, but RAID isn't a file copy and that's the underlying problem... this is triggering a disaster recovery mechanism designed for something totally different.
-
Don't know if still viable but Dell used to have and RDX drive that took hard drive medium instead of tape. Was OEM from either Quantum or Tandberg can't remember who and Google Fu isn't working for me
-
