Data storage solution

[SiliconWizard]

I've built a NAS - entirely custom - almost 10 years ago. Runs on CentOS (6), small case, 4TB in RAID-5 (so that's 3x2TB HDDs.)
It has run fine (had to replace 1 drive a while back) ever since, but its power supply suddenly died (in a small explosion noise with nasty smell - a cap exploded.)

Since I was already thinking of replacing this NAS with something even smaller and lower power for a while, I'm thinking this is the right opportunity instead of just fixing/replacing the PSU.

What would you guys choose for storing ~3TB of data (4TB capacity is still alright with me)? The use case is: frequent reads, but relatively infrequent writes. A LOT of files of various size up to a few GB each. Reliability a factor. Cost too...
(I was thinking of actually replacing that with just SSDs - no RAID, but for 4TB total, it's still pretty expensive.)

[chriva]

I'd stick to the raid solution. Running a slightly similar setup myself but with Debian and ZFS. Way cheaper and probably more reliable than the consumer crap

While raid is not really a "backup", it's still safer than a single drive so I'd not upgrade to ssd unless you stick to the same arrangement but with solid state drives.


Specs for the heck of it:
Running a phenom 965 with custom CnC settings (lowest state is way, way down there in terms of voltage. HT and NB are both underclocked/undervoltaged at all times), dd3l-ecc ram (Low voltage is "not supported" but you can easily make it work any way). Getting around 8-10W idle when all the drives are spun down. Don't mind if it takes a few secs to get going if I haven't accessed it in a while.
4 x 4tb split in 2 software partitions and a few loose drivers for temporary gunk I don't care about

[SiliconWizard]

Well. Yeah. The point of this NAS was for a zero-backup solution. (Among other things, it actually served as a backup solution for other computers itself.) It has served well for this. I had one drive failure in 10 years, and nothing lost at all thanks to RAID-5. Of course, theoretically, people will tell me RAID is no substitute for backups, but if you monitor it on a regular basis, it's not that risky. It was actually backed up (partially, critical files) on other media, but pretty infrequently.

This small server was also to act as a small server for other tasks, but I can largely do without that now.

Things have evolved though, and my views on this as well. I'm thinking that provided that I duplicate my data on a regular basis, the probability of loosing it is relatively small. I have years of experience with this, and never lost anything. I now tend to favor data replication (on a regular basis) rather than a "no-fail" approach (yeah, I know, anything can fail anyway.)

As to power draw, my NAS was about 30W on idle. Not too bad, but I'd like less.

[Wuerstchenhund]

For my data (approx 20TB give or take) I tend to stick with Enterprise stuff. For hardware that means servers from HP or Dell, and for the OS that means derivates of the two major Linux vendors which are Red Hat and SUSE.

For NAS applications I've been using HP MicroServers since the day of the N36L (HP's first one) back in 2011 or so. At home I currently run a number of MicroServer Gen8 with XEON E3-1260L v2 CPUs, 16GB RAM and Smart Array P222 and P420 SAS/SATA Hardware RAID controllers. The MS G8 is great, it's small but still comes with server-class hardware including its own remote management processor with HP ILO4. We use lots of them at work as workgroup storage and they have proven rock solid, as have the ones I use at home. They are also inexpensive for what they offer.

My primary NAS is a MS G8 with four HGST UltraStar 6TB SATA hard drives in RAID5 for data and a single Micron 5100 200GB SATA SSD in the optical bay for the OS (openSUSE 15.1). Both the OS and the data rest on BTRFS file systems.

However, i do have to say that I do make backups. One is online into encrypted cloud storage, the other is on tape (LTO5). Simply because as reliable as the HP hardware normally is, RAID is not a backup, and if you don't have a backup then your data isn't important, period. RAID's purpose is to maintain availability of the storage subsystem in case a drive fails, but it's not a replacement for backups.

What would you guys choose for storing ~3TB of data (4TB capacity is still alright with me)? The use case is: frequent reads, but relatively infrequent writes. A LOT of files of various size up to a few GB each. Reliability a factor. Cost too...
(I was thinking of actually replacing that with just SSDs - no RAID, but for 4TB total, it's still pretty expensive.)

For your use case I'd be tempted to get a MS G8 (used and pretty cheap) or one of the later ones (HPE MS G10, MS G10+, depending if you need remote management) and two Enterprise-grade 4TB SATA drives and run them in RAID1 (i.e. as mdmraid, or under openSUSE as BTRFS RAID1). With less than 4TB of data there is little use to setup a RAID5 with smaller drives today.

[fordem]

Of course, theoretically, people will tell me RAID is no substitute for backups, but if you monitor it on a regular basis, it's not that risky.

There's no theory about it - the purpose of RAID is not to prevent data loss due to drive failure, but to reduce down time in the event of a drive failure - it allows you to schedule a maintenance window at your convenience.

A couple of decades ago the local branch of a big Canadian bank was running their banking platform on an IBM mid range system with mirrored (RAID1) disks, they experienced a failure that took out both disks in one of their mirrors - as you would expect, the data was backed up to tape, so there was no loss of data, but, for several years their management never missed an opportunity to remind me of the failure.

The other fact of RAID you need to be aware of is that with increased drive capacity comes increased rebuild times and an increase in the probability of a second drive failure caused by increased stress during the rebuild - none of the major storage manufacturers recommend RAID5 any more.

[PKTKS]

Similar path myself but  somewhat different

- RAID mostly for the sake of speed
- RAID is very costly so HDs are usually better bit/dollar
- RAID does not need power stuff usually a 500W is too much
- RAID needs cooling and fast ethernet
- those specs usually helps to select MOBO/CPU/MEM an PSU
- RAID WILL DEFINITELY REQUIRE A DECENT UPS
- PFC PSUs will double the cost requiring true sine UPS
- under 500W my RAID uses affordable non PFC under 500W
- with much cheaper quasi-sinus UPS (more than half price)

As stated above - RAID is not BACKUP.
RAID  fits mostly speedy storage.

For BACKUP an entirely extern unit is used which
typically is also very costly by itself but runs standalone
in external BACKUP units (managed by periodic services)

Both solution match nicely a decent cost/effective ownership

Paul
 

[SiliconWizard]

To make things a bit clearer if they were not.
This is mostly for "personal" use. Not for an enterprise-wide solution.

As I said, I have a lot of data, but most of it doesn't change frequently. So frequent backups are not necessary IMHO (and experience so far has proven me right.)
A lot of this data is not ultra-critical. Loosing it would suck, but would not be a disaster. My critical data is of course backed up on a regular basis (but takes a lot less storage space!)

Any possible down-time would be annoying. That's mainly why I chose RAID-5 at the time. As I said, despite an HDD failure, I never lost any data and never had any significant down time (to be honest, during reconstruction, data throughput is of course significantly altered, but it's still accessible.)

What my thread kind of implies is that I'm looking for a simpler solution, that would still reasonably fit my requirements.

As to reliability, RAID-5 will not only limit downtime in case of HDD failure, but it also lowers the probability of losing data. Of course, I know this probability is not going to be 0. But I was mainly looking for a solution that was more reliable than a single drive (single point of failure), and would make backing it up less frequently an option. Sure it's all in the risk taking area. I just want less maintenance, but with a reasonable risk. The above solution has worked well for me for that. I'm just looking for something that would have a similar failure probability and be simpler/less power hungry.

[M0HZH]

Running Synology DS713+ with 2x4TB WD Red as a home storage / development machine / small web server for 4+ years now. The hardware is OK but the operating system is where it really shines; by far the best option for home and SMB use. Higher cost of entry but extremely easy to use, pretty powerful and requires no maintenance. We use Synology boxes for storage and systems backup at work as well (medium sized business).

[PKTKS]

To make things a bit clearer if they were not.
This is mostly for "personal" use. Not for an enterprise-wide solution.
(..)

That is simpler

Go to a simpler redundant HD RAID PC with
a cheap simple MOBO with enough PCIex slots to house
2 or 3 ethernet gigabit addins.

use a cheap CPU really -  CPU only matters if
the SATA controller is too lame. Memory won't matter
the lesser the better.

Such a system barely exceeds 250W may be 300 if the
CPU TDP is high. Use 120mm fans on the HDs and
most important a decent UPS (do not be cheap here).

If your switch supports port trunking just configure
your *NIX for 802.3ad port trunking and ifconfig a port
trunking bond attached interface.

If not (no 802.3ad) just use a bond balance mode.

Cheap affordable setup - Try DEVUAN or may be
https://systemd-free.artixlinux.org/

You don't need interfaces at all - use plain NFS
over your multi gigabit setup. Way faster.

Paul

[Marco]

As to reliability, RAID-5 will not only limit downtime in case of HDD failure, but it also lowers the probability of losing data.

Performance is severely degraded during a rebuild, which lasts so long and trashes the array so hard the danger of a second drive dying is non negligible.

SSD caching can replace striping and simple mirroring can replace parity. Less things which can go wrong, saves power too.

PS. also though it's not relevant here, you can just expand storage 2 drives at a time without it affecting the rest of the storage.

[Monkeh]

Go to a simpler redundant HD RAID PC with
a cheap simple MOBO with enough PCIex slots to house
2 or 3 ethernet gigabit addins.

If your switch supports port trunking just configure
your *NIX for 802.3ad port trunking and ifconfig a port
trunking bond attached interface.

If not (no 802.3ad) just use a bond balance mode.

All of which will achieve being limited to 1Gbps for a single stream as if you only had one NIC.

[rdl]

I have FreeNAS running on a HP Microserver with a pair of mirrored drives (2x 2TB Blue and 2x 4TB Red). The two mirrors are currently synced, but eventually the Blues won't be big enough for that. I don't really like having both mirrors in the same machine, so I'm also looking for a new system of some kind.

[bdunham7]

RAID 5 is, I believe, statistically less safe then just a single drive.  It doesn't sound like uptime is an issue for you either. 

I use two 8TB Seagate 5400RPM (slow) SMR drives harvested from the external HDDs that they sell at Costco and so on.  You could do the same with the 2.5" 4TB external units if you wanted small, quiet, cool and low power.  You could run RAID 1 if you want a sort of auto-backup (RAID 1 is still not a replacement for backup!) What I do is just run them as two drives and if I'm especially concerned about a new file before it is compressed and backed up, I can keep a copy on each.  The SMRs can slow down a bit on writes and they need to be on a lot to do their reshuffling thing, but mine have been quiet and reliable this way.  And they were cheap.

[Wuerstchenhund]

The other fact of RAID you need to be aware of is that with increased drive capacity comes increased rebuild times and an increase in the probability of a second drive failure caused by increased stress during the rebuild - none of the major storage manufacturers recommend RAID5 any more.

Actually, this isn't entirely correct. The whole "RAID5 should no longer be used'" idea came along because of a series of ZDNet articles which used a very theoretical approach and basically assumed that a drive doesn't tell you when it encounters an unrecoverable error (which is what many SATA consumer hard drives do, but neither NAS nor Enterprise class SAS or SATA hard drives, i.e. the type you'd typically find in a real server).

In addition, the often made claim that RAID5 is too risky because of the supposedly much longer time needed to resilver an array isn't entirely correct either, at least as far as HW RAID is concerned, because back in the old days it took longer to recover an array of four 18GB SCSI drives connected to a fast Mylex DAC960 HW RAID controller than it takes to do the same for an array of four 8TB SATA NAS drives connected to a modern controller like a HPE P440ar. And I've seen a lot more problems during resilvering back then than today.

RAID5 is perfectly fine for some uses cases as long as you remember that (as you wrote) RAID is not a backup, and that the reliability of a storage solution lives and dies with the quality of its components.

[Wuerstchenhund]

To make things a bit clearer if they were not.
This is mostly for "personal" use. Not for an enterprise-wide solution.

Doesn't matter much. In any case the aim is to prevent data loss, right?

As to reliability, RAID-5 will not only limit downtime in case of HDD failure, but it also lowers the probability of losing data.

Not really, no. Because hard drive failures are not the only cause of data loss, nor are they the most common (far from it, actually). There is a wide range of other reasons why you can lose data, from simple user error or malware to silent data corruption due to a hardware problem (or even just because the RAM doesn't have ECC). RAID provides *zero* protection against any of them.

Besides, backups don't need to be complicated. Cloud backups for example can run automatically in the background. Just make sure you test your recovery procedure.

[Wuerstchenhund]

Go to a simpler redundant HD RAID PC with
a cheap simple MOBO with enough PCIex slots to house
2 or 3 ethernet gigabit addins.

use a cheap CPU really -  CPU only matters if
the SATA controller is too lame. Memory won't matter
the lesser the better.

Such a system barely exceeds 250W may be 300 if the
CPU TDP is high. Use 120mm fans on the HDs and
most important a decent UPS (do not be cheap here).

300W? 

The Dell PowerEdge T320 in my living room draws less than 110W (it mostly idles around 89-95W), and that is with a 10-core Xeon E5-2470v2 processor, 96GB RAM, two SAS 12G SSDs and three 6TB SATA drives, a PERC H730P HW RAID controller and a dual 10GBit Ethernet adapter

Don't know what the MicroServer Gen8 use but I recon it's less than 60W, with four hard drives and a SSD.

250 or more Watts for a simple NAS is madness.

If your switch supports port trunking just configure
your *NIX for 802.3ad port trunking and ifconfig a port
trunking bond attached interface.

If not (no 802.3ad) just use a bond balance mode.

Cheap affordable setup - Try DEVUAN or may be
https://systemd-free.artixlinux.org/

You don't need interfaces at all - use plain NFS
over your multi gigabit setup. Way faster.

That may make sense in Brazil where you can't get modern stuff at reasonable prices but here in Europe a 10Gbit network adapter can be had for $30 or so.

If network was actually a problem, that is.

[Marco]

Actually, this isn't entirely correct. The whole "RAID5 should no longer be used'" idea came along because of a series of ZDNet articles which used a very theoretical approach and basically assumed that a drive doesn't tell you when it encounters an unrecoverable error (which is what many SATA consumer hard drives do, but neither NAS nor Enterprise class SAS or SATA hard drives, i.e. the type you'd typically find in a real server).

Baarf is a lot older than that.

It's just become an even worse option along the way. Storage was always too cheap to deal with all the headaches, now moreso.

[Wuerstchenhund]

Actually, this isn't entirely correct. The whole "RAID5 should no longer be used'" idea came along because of a series of ZDNet articles which used a very theoretical approach and basically assumed that a drive doesn't tell you when it encounters an unrecoverable error (which is what many SATA consumer hard drives do, but neither NAS nor Enterprise class SAS or SATA hard drives, i.e. the type you'd typically find in a real server).

Baarf is a lot older than that.

True, but the reliability and performance of the RAID hardware back then wasn't exactly stellar, plus all the sensitivities with the cabling (parallel SCSI and the issues with termination). Resilvering failures were a lot more common back then.

It's just become an even worse option along the way. Storage was always too cheap to deal with all the headaches, now moreso.

RAID, including RAID5, works just fine for millions of applications around the world. It's a simple fact.

Where it doesn't work is when people abuse it because they don't understand what RAID is for (which isn't backup), or when it's used in combination with cheap and nasty consumer-grade hardware while expecting enterprise-grade reliability.

[PKTKS]

Such a system barely exceeds 250W may be 300 if the
CPU TDP is high. Use 120mm fans on the HDs and
most important a decent UPS (do not be cheap here).

(..)

250 or more Watts for a simple NAS is madness.

You (and that Monkhey guy) sound devoted of making
 FUN and MOCKERY whatever post fits a twisted vision of yours...
much like young whiny juveniles  rants..

Please READ THAT FUCKING THING again where is "exceeds.."
is a proper ceiling word which fits my last 20y using NAS.

FYI  my first options were those "ready made consumer things"
which you promote as being "better"..

Well those things were replaced along last years with
solutions FAR better where I can control software and
hardware and scale them as needed.

If your switch supports port trunking just configure
your *NIX for 802.3ad port trunking and ifconfig a port
trunking bond attached interface.

If not (no 802.3ad) just use a bond balance mode.

Cheap affordable setup - Try DEVUAN or may be
https://systemd-free.artixlinux.org/

You don't need interfaces at all - use plain NFS
over your multi gigabit setup. Way faster.

That may make sense in Brazil where you can't get modern stuff at reasonable prices but here in Europe a 10Gbit network adapter can be had for $30 or so.

If network was actually a problem, that is.

There is nothing related to Brazil taxes here - and here comes again
such stupid twisted vision...

I ditched ready made solutions 2 decades ago.

Replacing them with  my own solution includes a custom Kernel
with all the proper  file systems and security measures in place.

One of them is bounded to the proper interfaces whre
REDUNDANT (fault tolerance) and a port trunking solution
fits as no other "new crappy shit" you can buy on your
local store.

Period..  I could mock around your "new hardware" all day long
but honestly ...

I have no sense of humor.

My solutions are in place for decades
They replaced your claimed "better" with custom ones

Please go have fun with someone else..

Paul

[SiliconWizard]

RAID, including RAID5, works just fine for millions of applications around the world. It's a simple fact.

Where it doesn't work is when people abuse it because they don't understand what RAID is for (which isn't backup), or when it's used in combination with cheap and nasty consumer-grade hardware while expecting enterprise-grade reliability.

Yup. And it's worked fine for me as I explained. Again of course, it's not 100% fail safe.

But yes, the probability of data loss, if correctly set up and not abused as you said, is still much lower than using a single drive. I'm not taking user mistakes into account either here. That's also what back-ups are for. As I said, my use case is specific, as the largest chunk of my data (in size) is almost read-only, so the probability of messing it up, if not from a hardware issue, is pretty low. Of course I back up frequently all my work files (but it's a lot smaller overall, certainly not TBytes of data.)

In over ten years of use, I never had to recover from a back-up, despite again a drive failure. I'm not advocating NOT doing any back-up. What I'm saying here though, is that for a limited use case such as mine, you can definitely back up a lot less frequently, and still expect your data to be reasonaly safe. I do back up critical data frequently, so that's pretty much all a matter of data criticity and probability of loss. I want a low probability, but I'm ready to accept the risk. I wouldn't do that for a company, nor again for my critical data. But even if I'm ready to accept some risk for my less criticical data, I still don't want a single point of failure.

Now it's also all related to the practicality of frequently backing up several TB of data. I for one don't want to use cloud backup, and my DSL connection would make that not very practical anyway (it would probably take days for a full backup.)

But getting back to RAID, as I said, that's what I built like 10 years ago. Things have changed. Hardware has evolved, my views as well. So I'm ready to take another approach, but having significant experience with handling data now, I'm even ready to take a much simpler approach. Just curious what others typically do. These days, I'm thinking it may make more sense (and would draw less power) not to use RAID, use a single drive, and replicate it on a regular basis (that would look like backing up), swapping it every time. I dont really mind having to do this manually, as it's a forced opportunity to check drive health, data integrity, etc. (Again I would not advocate doing that for a larger organization, that wouldn't scale up.)

As I said earlier, my goal in terms of hardware/power consumption is also very spartan. My current NAS is drawing ~30W on idle (and usually not much more than 60W when streaming data over the network), and I'm now looking for less. So anything in the order of 250W or over is out of the question.

[PKTKS]

(..)
As I said earlier, my goal in terms of hardware/power consumption is also very spartan. My current NAS is drawing ~30W on idle (and usually not much more than 60W when streaming data over the network), and I'm now looking for less. So anything in the order of 250W or over is out of the question.

Mine goals at first where that simple...
They changed over the years as things (and the internet, wifi) changed

I was using ready made solutions with such really low
grade power ratings. - but they don't scale and they
became obsolete (file systems and security...etc)

In rough numbers if you want to run a custom kernel
with proper firewall, access control and all good file systems..

You will need a MOBO + CPU + SATA drives  - being roughly:
= 10W per each modern SATA  (7W motor 3W logic)
= 15W to 35W a iGPU integrated MOBO
= 45W to  65W a decent low TDP CPU

about 4 to 8 SATA drives will give you around 150W
or more depending on PSU, NICs  and COOLER solution

For some time those really small gigs served me pretty well
till a CPU upgrade and more drives forces a full ATX PSU in place

Otherwise I would  not be able to setup  a fault tolerant
multi gigabyte wifi capable and with proper recent firewall
system as it run today.  For 4 SATA a 150W expected
as it grows as 8 SATA expect 200W

Paul

[Monkeh]

You will need a MOBO + CPU + SATA drives  - being roughly:
= 10W per each modern SATA  (7W motor 3W logic)
= 15W to 35W a iGPU integrated MOBO
= 45W to  65W a decent low TDP CPU

All worst-case load consumption, not idle. If you're going to give (bad) advice, at least speak the same language as everyone else.

Now, the easy pickings done..

You're going to find it hard to get a multi-TB system much below 30W idle without spending a lot of money on SSDs or trying to wear drives out by spinning them down when idle. At least with any redundancy applied.

[bdunham7]

RAID, including RAID5, works just fine for millions of applications around the world. It's a simple fact.

Where it doesn't work is when people abuse it because they don't understand what RAID is for (which isn't backup), or when it's used in combination with cheap and nasty consumer-grade hardware while expecting enterprise-grade reliability.

Well, I've abused  vigorously used a lot of 'consumer' hardware with little or no reliability issues for years, with the exception of some truly cheap and nasty PSUs.  Your much-vaunted 'enterprise grade' doesn't necessarily have all that much to do with reliability. 

But, what I really wanted to point out is that the whole attraction for RAID5 in the first place was that it was the cheapest possible solution in that it used the least number of hard drives with the least capacity loss due to the room used for parity.  It sucked from every other possible perspective.  Is there even a plausible use case for it these days?  At any given desired capacity can you get a better, faster more reliable deal from a different configuration at a lower price point?  I think you can.

[SiliconWizard]

But, what I really wanted to point out is that the whole attraction for RAID5 in the first place was that it was the cheapest possible solution in that it used the least number of hard drives with the least capacity loss due to the room used for parity.  It sucked from every other possible perspective.

That is mostly right, and yes, one of the reasons I chose RAID-5 at the time (several TB HDDs were still not that cheap 10 years ago!)
As I said, it has never failed me, but I'm definitely considering something else this time.

Anecdotically, the HDD failure I experienced once was not due to wear due to RAID-5 itself... but at the time I bought WD green HDDs (they were very low power compared to competition), but they came with a firmware that would park heads every few seconds if the drive was not accessed (and there was nothing you could do to change that), which happened all the time of course for a typical "personal" use. This infamous "feature" gave the Green line a very bad rep and would usually kill them in a matter of a couple years.

[bdunham7]

If you used an Atom-based (or even an older Celeron-based) mini-itx board (one that runs on a 12V wall wart) and a Seagate/Samsung 4TB 2.5" (15mm--no laptops) SMR drive, you'd meet your power requirements.  Add one more drive for either backup and let the OS do an incremental backup every night or week or whatever.