@scottalanmiller said in Incorporating Ransomware Protection into Backup Plan:
D2D2T
Appreciate all of the input. This is the solution I've been leaning towards over the last week. Had an infrastructure hiccup & haven't been able to spend any time on this. But I will utilize my existing backup device for the backup disk & incorporate standard LTO-8 drive library with a rotating weekly offsite storage.
Several years ago Scott gave me some solid advice on arrays & I thought it was worth checking in on what the people here do for incorporating ransomware protection into their backups when online backup services aren’t an option. In the last 3 months we’ve had two customers get hit by ransomware. This is my nightmare scenario & it’s time to spend some resources building a more comprehensive protection plan.
We have two servers. A production server that has 30TB / 30,000,000 files with 1 - 2 TB that rotates on / off weekly basis. This server also has a 30GB production SQL Database.
The second server is our internal company resources, a domain controller / DNS, SVN etc. It's on a VM.
I have a local backup plan - it’s not high tech & runs on a weekly basis. But we can survive any type of hardware failure. I have intentionally avoided daily / hourly backups due to ransomware concerns.
I don’t fully understand or trust incremental / incremental + block level backups. We do use them for local computers on remote services (Cloudberry + Backblaze) which is fine. But a local machine has a lot fewer changes than what I’m dealing with.
My understanding is the processing time of calculating differences is likely to be difficult for this size & this many files - as well as the restore. I like being able to look at / open / copy files.
From my limited research - I’ve come to four broad options that are something like
Large incremental backup array with an airgap only coming online for backups (Possibly manually done on a weekly basis). Phrases like “incremental backup decreases recovery reliability.” terrify me because I’ve been the brunt of “it should have worked but didn’t” in this exact scenario (although this was years ago).
I know that if the network doesn’t have authorization - ransomware shouldn’t be able to touch it - but I like 'can't' more than 'shouldn't' - (Tape Drives or smaller self contained arrays that have even less connection time to the network).
I’m also skeptical about the processing time for 30 mil small files. (Maybe I’m wrong about this?) This would have to include a software solution (Veeam, Cloudberry, HP StoreEasy has free Carbonite software)
$6-8K hardware / software ?
Tape drives
I’ve been trying to read on these. Some people have said IT moved away from them because they weren’t reliable, slow to retrieve individual files etc. This seems like it might be a workable solution but I need to know if it’s going to handle 30mil files on a weekly basis - & if not then it may be a dead end solution already.
$8K-12K hardware / software / tapes
4 Small(er) weekly backup arrays. Weekly rotating backup that involves physically connecting to the network, starting the backup & unplugging when done. Configuration would probably be 4 DL380 G8s + 48 used/refurbished HGST He8 drives in a small cabinet with a KVM & an actual person weekly backup task (only 1 computer would touch the network per week)
$8K hardware + software
Other solutions - OneBlox? I know people over at Spiceworks were big fans of OneBlox which seems to have morphed into StorageCraft. I think OneXafe is what I’d be looking at which seems to be a Hardware + Software in a box solution.
OneXafe $15K - $40K
My biggest concern is investing a substantial sum of money & time in a solution I’m sold that tries to do more complex things than I need (realtime protection, instant restore, cloud backup etc), requires a steeper learning curve & then winds up not working out when it’s needed the most.
I don’t mind spending what’s needed for a safe solution. But I bend towards simplicity as only 20% of my time is spent on infrastructure & I’d prefer to keep it that way. Given all this - in my shoes what direction would you look for backups that incorporate ransomware safety & why?
I've actually been playing OpenRA - real blast from the past. I've been impressed with how well they did on re-creating it.
@scottalanmiller I somehow missed this reply. This is the answer I was looking for. The great news is that my hardware will likely stay (almost) the same when I need to upgrade.
@PhlipElder said in Safe to have a 48TB Windows volume?:
What's the air-gap to protect against an encryption event if any?
What's the air-gap to protect against an encryption event if any?
My backup server has access to the rest of the network - but it pulls the backups to itself vs backups being pushed. The rest of the network can't directly write to it. My backups happen weekly - so my (hope) is that I would recognize what was happening to my live network before it was backed up.
I have been contemplating doubling my backup storage space to make sure I have enough space to store older file revisions in a ransomware situation.
@scottalanmiller said in Safe to have a 48TB Windows volume?:
There are no cases where you can't use hardware RAID.
Yea - for some reason I was thinking I would need to use ZFS. I'd prefer to stick to the enterprise hardware as it's caused 0 issues for me.
Before Crashplan turned their service off - they updated with a kill switch that made it to where you couldn't access data that had been backed up unless you had a computer that had been offline & not updated versions (& even then it was hackish - had to find a guy who made a "Plan B" tool to grab your decryption key etc). IMO insanely shady - I wouldn't touch them again.
We moved to Backblaze + Duplicati & moved the backup offsite instead of in house. Setup is much more of a pain but speeds are fast & pricing is super reasonable.
@scottalanmiller said in Safe to have a 48TB Windows volume?:
You can go to Linux and XFS without changing your RAID in any way.
Ah perfect. So I wouldn't need to move to software raid to move away from NTFS. I'm not convinced I need to yet. But if after more research I find out I do - Is it likely I'm going to run into issues using something like SAMBA + XFS as a windows shop network share?
@travisdh1 said in Safe to have a 48TB Windows volume?:
The triple mirror means that you will have increased read speed. If you don't need the increased read speed, then that's just a waste of drives.
It does (sortof) decrease my risk as I would need 3 drives out of any set of 3 to fail. I understand this looks like overkill. It also helps on read speed. Prior to this array I was using 36 600GB 15K SCSI. My goal was similar speed + safer setup + bigger volume. The difference in cost between using raid 10 & 10 ADM using 3TB drives is only about $2,000.
@scottalanmiller said in Safe to have a 48TB Windows volume?:
But it is darn close when using triple mirroring!
FWIW - you're the reason I migrated to Raid 10 off of my Raid 6 / 36 drive setup. Lots of yelling at me on Spiceworks a few years ago about how raid 6 isn't safe for huge arrays 
At a previous company I was over the dept I'm doing IT for now. I was told "we couldn't afford a backup" 'cuz I kept yelling at them about it (it was raid 6 across 25 1TB SATA drives in a fly by night company SuperMicro type box + 1 spare).
At one point IT did an array expansion adding drives & were unfamiliar with the array card. It corrupted our data (hundreds of thousands of files randomly re-assigned to different folders, 10s of thousands of corrupted files etc) (HP supports live expansion but this array controller did not). We were down for months, the fallout for not finding everything followed us for years. It almost destroyed us.
I am fine being down for a full week - or two weeks if I have to restore (haven't in 5 years). But offsite backup = insurance policy. I don't trust array controllers or a single server setup any more than I do a hard drive.
It seems like I remember Scott Miller talking about combining enterprise hardware + SAS/SATA Controller + Linux for storage requirements vs proprietary hardware raid controller.
@Donahue - Yes. I have a similar setup offsite backup several miles away for disaster recovery / hardware failure etc. I know raid != backups.
I'm currently using an 34 HP 3TB drives with an array configured for Raid 10 ADM (every drive set = 3 redundant drives) + 1 spare. Drives are sitting in 3 d32600 DAS boxes connected to a P822 controller in an HP DL360 g8 running Windows Server 2012 with a single 30TB share. This volume has about 30 million files that are accessed daily for a period of 1 - 3 months by 25 people before the data is archived to offline storage. Server is a stand alone bare metal installation.
It's important to be cheap, reliable & easy to manage as possible if I get hit by a bus. The only way I've found to do this is ride 1 - 2 generation old enterprise equipment (HP is what has worked well for me). Everything I've purchased is used (except for drives).
I am happy with my Windows + HP situation & 48TB would be fine for the next 2 - 3 years. Unless there's a big risk I'm just not factoring in. When I was asking for advice before & moving from an inherited 18TB / Raid 6 setup on Spiceworks a few years ago several people mentioned a possible concern about large NTFS volumes. Have any of you used 48TB Windows volumes? Any resources on risk analysis vs ZFS?