I'm currently using an 34 HP 3TB drives with an array configured for Raid 10 ADM (every drive set = 3 redundant drives) + 1 spare. Drives are sitting in 3 d32600 DAS boxes connected to a P822 controller in an HP DL360 g8 running Windows Server 2012 with a single 30TB share. This volume has about 30 million files that are accessed daily for a period of 1 - 3 months by 25 people before the data is archived to offline storage. Server is a stand alone bare metal installation.

It's important to be cheap, reliable & easy to manage as possible if I get hit by a bus. The only way I've found to do this is ride 1 - 2 generation old enterprise equipment (HP is what has worked well for me). Everything I've purchased is used (except for drives).

I am happy with my Windows + HP situation & 48TB would be fine for the next 2 - 3 years. Unless there's a big risk I'm just not factoring in. When I was asking for advice before & moving from an inherited 18TB / Raid 6 setup on Spiceworks a few years ago several people mentioned a possible concern about large NTFS volumes. Have any of you used 48TB Windows volumes? Any resources on risk analysis vs ZFS?

It seems like I remember Scott Miller talking about combining enterprise hardware + SAS/SATA Controller + Linux for storage requirements vs proprietary hardware raid controller.

@Donahue - Yes. I have a similar setup offsite backup several miles away for disaster recovery / hardware failure etc. I know raid != backups.

@scottalanmiller said in Incorporating Ransomware Protection into Backup Plan:

D2D2T

Appreciate all of the input. This is the solution I've been leaning towards over the last week. Had an infrastructure hiccup & haven't been able to spend any time on this. But I will utilize my existing backup device for the backup disk & incorporate standard LTO-8 drive library with a rotating weekly offsite storage.

@travisdh1 said in Safe to have a 48TB Windows volume?:

The triple mirror means that you will have increased read speed. If you don't need the increased read speed, then that's just a waste of drives.

It does (sortof) decrease my risk as I would need 3 drives out of any set of 3 to fail. I understand this looks like overkill. It also helps on read speed. Prior to this array I was using 36 600GB 15K SCSI. My goal was similar speed + safer setup + bigger volume. The difference in cost between using raid 10 & 10 ADM using 3TB drives is only about $2,000.

@scottalanmiller said in Safe to have a 48TB Windows volume?:

But it is darn close when using triple mirroring!

FWIW - you're the reason I migrated to Raid 10 off of my Raid 6 / 36 drive setup. Lots of yelling at me on Spiceworks a few years ago about how raid 6 isn't safe for huge arrays

@PhlipElder said in Safe to have a 48TB Windows volume?:

What's the air-gap to protect against an encryption event if any?

What's the air-gap to protect against an encryption event if any?

My backup server has access to the rest of the network - but it pulls the backups to itself vs backups being pushed. The rest of the network can't directly write to it. My backups happen weekly - so my (hope) is that I would recognize what was happening to my live network before it was backed up.

I have been contemplating doubling my backup storage space to make sure I have enough space to store older file revisions in a ransomware situation.

@scottalanmiller Really appreciate the input. Didn't know if it was possible for chkdsk to not pick up corruption issues or it was something mid flight. Intel drivers seems to make the most sense.

@jaredbusch From one folder on the share to another folder on the share from a client computer. Was reproducible on specific directories before restart. Copying same files down to a networked computer locally - then back up to the share resulted in legitimate copy.

@jaredbusch said in File Corruption on Copy Issue:

updates get applied and the reboot was pending,

Updates were applied and reboot was pending on the network driver. Starting to think this was the issue, even though I don't understand how or why it would have caused it.

These were the updates installed - after reboot issue went away & hasn't come back

2021-07 Security Monthly Quality Rollup for Windows Server 2012 for x64-based Systems KB5004956
Windows Malicious Software Removal Tool X64 - v5.91 KB890830
2021-07 Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.2, 4.7, 4.7.1, 4.8 for Windows Server 2012 for x64 KB5004230
2021-04 Servicing Stack Update for Windows Server 2012 for x64-based Systems KB5001401
Update for Windows Server 2012 KB3102429
Intel - LAN (Server), Other hardware - Intel(R) 10 Gigabit CX4 Dual Port Server Adapter
2021-01 Security Update for Windows Server 2012 for X64-based Systems KB4535680

@scottalanmiller May spin up Ubuntu VM & try this again, last time I did I ran into issues & got lost down rabbit hole of trying to figure out what the problem was.

@scottalanmiller said in 45TB-60TB / 45-60 mil file volume recommendations?:

use your own hardware and just install Linux on it

I get this - but as stated I spend 10-15% of my time on Hardware. Based on my personal interactions with linux on personal offtime nothing is officially supported and there are 21 different bash scripts needed to run to enable standard stuff while everyone claims it is consumer friendly.

This machine also runs SQL Server w 40GB database so moving to linux for file store would require different hardware for the Windows SQL Server (SQL for Linux I have no interest in doing)

@dashrender Yes - Raid 10 ADM was over cautious / overbuilt. Went from Raid 60 -> Raid 10 ADM after reading some of @scottalanmiller stuff & understanding the tradeoffs of rebuilds for 60 vs 10. After 3 years of not losing a single drive, I'm comfortable moving to simple Raid 10 with a hot swap.

My reason for considering NAS or something like Exablox was because as I understand it some linux filesystems (I'm not familiar with which ones) are safer than NTFS for large volumes with tons of files. This latest hiccup of file corruption is what alarmed me. I bend towards not preferring this as it makes my life more difficult / less flexible have a 40GB SQL Database etc on Windows Server. Only spend about 10-15% of my time on hardware stuff so much more interested in things that just work.

Hope I'm not out of line to create two topics. I have DL360 G8 / 3 D2600s / 36 3TB HP MB3000FBUCN HDs / Raid 10ADM / Windows Server 2012 that hosts about 30 mil files used by 10-12 people daily. As jobs are processed, these files are moved off the server into cold storage.

I need to increase our storage to 45-60TB. My plan was to just migrate to Raid 10 instead of Raid 10 ADM (I haven't lost a 3TB drive since moving off of raid 60 to Raid 10ADM in 3 years). I'm a little alarmed after a recent file corruption issue described here - https://mangolassi.it/topic/23278/file-corruption-on-copy-issue

I've been a hardcore HP fanboy because I've had very very few problems with their stuff even buying used. Their continued aggressiveness at putting drivers behind paywalls + proprietary HD cost has me bent out of shape pretty hard. I'm looking at two possible solutions.

1 - Switch from Raid 10ADM -> Raid 10 / 30TB -> 45TB. Upgrade DL360 Gen8 to Windows Server 2019 (possible new Windows versions make NTFS volume of this size safer?)

2 - Switch to NAS / Exablox - StorageCraft type solution (I know this used to be shilled pretty hard - is it still top notch set & forget type solution? Will speed be comparable to 36 spindles on Raid 10?)

*I currently have a backup solution using VEEAM + DL380 w 12 drives + tapes for offline storage.

I've got an HP DL360 G8 hooked up to 3 D2600s / 30TB Raid 10 / P822 / 30 mil files / Windows Server 2012.

Last week where someone copied TIF images - the source & destination had same file size / counts but the newly made copy was corrupt / not viewable. (Verified this with a binary editor actual files didn't match / but file properties matched.). When I tried to do this exact copy - it reproduced the exact same situation (corrupt copy / file properties matched). 2 people also claimed they had copied directories that had pulled different directories. I have only seen a situation like these once & it had to do with IT staff expanding an array that didn't support expanding live.

I ran a chkdsk that didn't find any issues on the array, monitored files early in the AM to make sure we weren't being ransom wared, dug thru log files to see if I could find anything - everything appeared to be normal. After the Windows Update -> Reboot I could no longer produce the problems. The day people complained to me 4 people had issues, after the windows update / reboot - no one has complained since.

I'm pretty alarmed & have paused our 10 day rolling backups, have additional storage arriving tomorrow to create a snapshot backup restore before resuming. Have any of you ever seen something like this where both the array controller said the array was fine & chkdsk didn't find anything?

@pete-s said in Getting fed up with HP Active Warranty / Support Agreement Requirements:

You could comfort yourself with the fact that it's only small companies that pay the steep markup. Enterprise customers doesn't.

Grumbles well I have been running my entire server stack off of used enterprise equipment via eBay so full circle of life I guess