Yet another ... which Linux file systems ?

[BravoV]

Planning to replace the aging swmbo desktop, and she practically used it just for casual documents work like word processing, spreadsheet and browsing. And also she does a lot of photos & videos backup from her phone into the desktop storage. And casually plays old windows games.

And no, no cloud storage discussion please.

Me my self and her are Linux noob, but decades of Windows experiences.

Planned configuration :

- Quad core ryzen
- Memory 16 GB (ECC or non ECC, subject to final decision) ... done deal, ECC it is.
- Boot drive NVME 1TB
- Storage 3 x 8TB (Raid 5 or RAID-Z1)

OS , will try forcing luring her into Linux and probably with Windows VM. 

Regarding the RAID, I'm fully aware as this is just to reduce down times, we have an air gapped external HD 16TB which we do routine monthly backup.

Also regarding desktop RAID, I'm fully familiar since early days of fake raid using Intel chip-set up to dedicated  LSI Megaraid, its just I'm getting tired having to rely on the controller as single point of failure, beside its only for casual desktop usage, not high end enterprise grade server, CMIIW.

Questions :

This regarding the file system :

Option 1
Boot drive : EXT4
3 x 8TB : On MDADM

Option 2
Boot drive : EXT4
3 x 8TB : ZFS RAID-Z1

Option 3
Boot drive : ZFS
3 x 8TB : ZFS RAID-Z1  ... option 1 is decided.

Linux distro decision ... latest Ubuntu LTS ? 

Regarding physical drives nvme + 3 HD decision, its already fixed and purchased.

Or .. any other suggestions , critics maybe on my plan ?

[greenpossum]

Unless you are skilled and know it, go with the filesystem your preferred distro uses, you'll have less hassles that way. Performance isn't going be an issue for your workload.

[rdl]

I recently set up a Linux system to be my daily use replacement for a Windows 7 machine. I went with whatever Debian uses by default and never gave it a second thought.

I would consider using those 3 8TB drives as basis for a FreeNAS set up, but that would mean getting another computer suitable for that purpose.

[RoGeorge]

ZFS can do many things that ext4 can't, and by design is more reliable, definitely ZFS.  I will go for Option 3, or maybe an Option 4:  Ubuntu 20.04 LTS + 1TB (ZFS OS) + 8TB ZFS + 2x8TB ZFS RAID1 powered only during backups.

For the enumerated applications, there should be no need to use Windows VMs.

[0db]

Ext4 in my experiences was pure crap.

[Karel]

In the past I used Reiserfs because that was the default with SuSE Linux.
After mr. Reiser murdered his wife, they switched to ext3 and then to ext4.
Now they use btrfs as default but I don't trust it and continue to use ext4.
The reason is, ext4 has never caused me any dataloss ever, and we have quiet some power interruptions here 
I have been thinking about to buy an ups, but my impression so far is that ext4 is very robust.
I'm using a mix of traditional harddrives and solid state (Samsung Pro).

[0db]

I do trust xfs v4, and I really like its checking/repairing tools.

[madires]

ext4 is fine for a desktop system.

[PKTKS]

Option 1  Boot drive : EXT4 3 x 8TB : On MDADM
Option 2  Boot drive : EXT4 3 x 8TB : ZFS RAID-Z1
Option 3 Boot drive : ZFS 3 x 8TB : ZFS RAID-Z1

Linux distro decision ... latest Ubuntu LTS ? 
Regarding physical drives nvme + 3 HD decision, its already fixed and purchased.
Or .. any other suggestions , critics maybe on my plan ?

I am a long term *NIX user (include linux BSD and older stuff HPUX AIX SCO..)
But recent hardware is your biggest concern - not the file system.

I will not setup a  BOOT drive based on NVM sticks nowadays - to be safe
I would even consider problems of SWAP on SSD devices as a big issue.

So I would setup using regular HD (cheap TB available on SATA 6G)
That is unlike to have a single issue.

IMHO - ?  use a plain good SATA HD for your boot device
with regular  ext3 - SAFE , FAST, BUG proof and very reliable.

The rest options has too many issues to discuss here..
But as a DISTRIBUTION ?  I would radically consider to be

** FAR AWAY AS POSSIBLE OF ANY systemd DISTRO **
systemd is a total crap security hazard

Try  any ARCH based feature rich for newbies like
https://endeavouros.com/

IF POSSIBLE a DEVUAN  (DEBIAN without systemd)
order of magnitude better than any *BUNTU
which mostly is kinda goonie made...

Paul

[bd139]

Option 1:  ext4 + MD.

Watch out for these gotchas (I run this shit in production):

1. ZFS on linux is buggy as fuck. Avoid it like the plague.
2. MD on spinning rust takes AN AGE to rebuild parity. On a 6TiB RAID5 SATA volume, expect 3-4 days at least. It's quicker to put one of those disks on the shelf and run a 3TiB RAID1 with a hot disk on standby plugged into the unit ready to roll.
3. Bringing that along, make sure you have a spare disk or two lying around NOW because they almost always disappear from the market instantly when you get a disk failure.
4. Make sure you know ALL recovery scenarios and know how to use mdadm before you even think about putting data on the volume. Also make sure you know how to deal with a hosed systemd and mdadm setup because it almost certainly will happen one day.
5. If you're running MD buy a UPS because a big hunk of times a power outage will cause a full fsck which takes forever on huge volumes which are populated.

Large volume stuff now I prefer to keep it off the node and on a black box NAS (synology ones are nice) on a UPS and keep the root disk a disposable NVMe jobby.

Really the best thing to do is get rid of as much data as possible. It really does help the cost and management headaches.

[magic]

I run XFS on MD raid at home, it's OK. I don't even remember the exact reasons for choosing it over ext4. I think ext4 may be easier to recover from screwups because it has a somewhat simpler structure.

Also make sure you know how to deal with a hosed systemd and mdadm setup because it almost certainly will happen one day.

To each his own, I guess. I would never trust that rubbish so I set up RAID autodetect for my root partition. Unfashionable but effective

The whole thing is less of a problem with some noncritical storage partition, though.

[RoGeorge]

1. ZFS on linux is buggy as fuck. Avoid it like the plague.

This is absolutely not true in my experience.

Quite contrary.  ZFS worked perfectly for me, with zero intervention for about two years now.  I used it mostly with Ubuntu, but also with Fedora for about half a year.  No problems so far, no maintenance required, not even after power failures (there were about 10-20 mains power failures during the last 2 years of ZFS).

Of course, in the last 2 years there were countless updates in the the Linux kernel or in the zfs tools, but never encountered any ZFS bug so far, and never lost a file.

Do you have some examples for why ZFS was so bad for you?  What bugs do you encountered?

[bd139]

Well there was the absolute shit show that was 0.7.7 in 2018

Create a new file, get ENOSPC and any subsequent files in the dirent were orphaned and disappeared after a remount and/or reboot. Pretty major data loss bug that which showed exactly what little care and regression testing went into it. I had to spend a good 3 days unfucking something that was victim of that.

Then there was the issue where you issue a rollback and it blew the kernel out in mid 2019.

Big nope after that.

On Solaris / FreeBSD yes that's fine. On Linux, no!

[bd139]

I run XFS on MD raid at home, it's OK. I don't even remember the exact reasons for choosing it over ext4. I think ext4 may be easier to recover from screwups because it has a somewhat simpler structure.

XFS is nasty. I've had all sorts of problems, particularly on filesystems backed by other volume management systems (3PAR / EBS). Sometimes, on some kernels, and depending on what phase the moon is in, if you xfs_growfs on a high inode usage filesystem it doesn't update the inode free list until you remount which you can't do when there are 1000 threads with FD's open on it. Live resize my arse 

[0db]

I run XFS on MD raid at home, it's OK. I don't even remember the exact reasons for choosing it over ext4. I think ext4 may be easier to recover from screwups because it has a somewhat simpler structure.

XFS is nasty. I've had all sorts of problems, particularly on filesystems backed by other volume management systems (3PAR / EBS). Sometimes, on some kernels, and depending on what phase the moon is in, if you xfs_growfs on a high inode usage filesystem it doesn't update the inode free list until you remount which you can't do when there are 1000 threads with FD's open on it. Live resize my arse 

That's bullshit. Which version? XFS v5? v4? v3? and some kernels, which version?

[bd139]

CentOS 7.2-7.6 kernel. I don’t have the version numbers handy as it’s 23:20 here and I’ll be fucked if I’m digging it out now.

I actually had a bugzilla ticket open with RH on it.

[Ampera]

Little can replace true hardware RAID, except a few hundred or so dollars that remain in your pocket. Though I do get the single point of failure thing.

ZFS is nice, but I have a personal belief that if a task calls for ZFS, that task calls for FreeBSD. If you can't do it on FreeBSD, evaluate why you need ZFS.

Wish to respectfully disagree with PKTKS, root on NVMe is perfectly fine, and I can't think of a single reason to use ext3 over ext4. NVMe drives are perfectly reliable, and if you're worried about swap degradation just don't put a swap partition on it.

I think the bottom line in general, though, is you'd be much better off centralizing your storage, and if speed truly requires it (I can't think of a single pedestrian reason why) 10GbE. Adding RAID or ZFS to a desktop environment just adds more places for it to go wrong, and on a person not used to dealing with Linux, it's likely prudent to eliminate as many of those as possible. Honestly I'm not sure what she's storing, but I'd swap out those drives for another 1TB NVMe SSD or even 1-2 2TB NVMe SSDs, as that would go beyond what most people could reasonably store before seriously considering some sort of network attached storage.

[BravoV]

1st, thank you for all replies made, really appreciate the inputs given. 

I will just do selective quoting and reply where I see necessary below.

How important is your data, and how much portability do you need?

Well, mostly are personal files, for big size like photos & videos, while important documents which are quite small compared to photos/videos, mostly are secured & backed up on external media .. "AND" ... compressed in cloud for portability reason (no, please stop discuss cloud thingy again). We just don't or never have the need to access our family photos & videos while roamed out side our house, and I guess never in the near future too.

So regarding the importance of redundancy, my & wifey's data are pretty secured, at least to our requirement, and it has been for decades, and yes, we experienced and survived hardware failures like dead HDs or dead mobos while using fake raid multiple times in decades, so far we've survived and pretty content with our current strategy.

One most important thing we've learned, keep diligently and routinely backup important stuffs to external media, thats all we need.

While the OS and Apps, we consider it is "not" important/crucial, they can be scratched and re-image/re-install if necessary, infact I've been doing this routinely a full OS re-installation or most of the times re-imaging the whole boot/OS/apps drive for decades.

Old farts must be familar with Norton Ghost, yep, I've been using imaging tools since DOS era up to now in Windows with the free Macrium Reflect, it just work flawlessly. 

Besides, ZFS calls for ECC.

Yep, decision made, ECC memory is coming as Ryzen supports it, even not officially.

Otherwise if you are as conservative as I am, go for EXT4.

I'm getting convinced, for the NVME boot drive, EXT4 is the way, especially for the ease of recovery.

I would consider using those 3 8TB drives as basis for a FreeNAS set up, but that would mean getting another computer suitable for that purpose.

Both me & wifey, had experienced NAS few years ago, and we both decided to prefer local storage.

ZFS can do many things that ext4 can't, and by design is more reliable, definitely ZFS.  I will go for Option 3, or maybe an Option 4:  Ubuntu 20.04 LTS + 1TB (ZFS OS) + 8TB ZFS + 2x8TB ZFS RAID1 powered only during backups.

There is an adventurous side of me that other option that I didn't reveal, it is for the RAID-Z1 to use a cheap 250/512GB SSD as cache. Not final yet, still keep learning, reading and comparing options. 

IMHO - ?  use a plain good SATA HD for your boot device
with regular  ext3 - SAFE , FAST, BUG proof and very reliable.

Sorry Paul, probably I'm the one of those crowd that believe and experienced ... "once you're NVMEd, you will never look back" .. type of people. 

Btw, both our desktops are covered & protected by UPS, and yes, they work flawlessly too when experienced power outages.

Keep going please ...

[bd139]

A point actually on RAID. We found over 5 years that a single decent quality SSD has a better reliability outcome than two mechanical disks in RAID1.

Decent quality means Samsung Pro / HPE here, not "crucial instead of WD green"

[Wuerstchenhund]

In the past I used Reiserfs because that was the default with SuSE Linux.
After mr. Reiser murdered his wife, they switched to ext3 and then to ext4.
Now they use btrfs as default but I don't trust it and continue to use ext4.
The reason is, ext4 has never caused me any dataloss ever

We run a lot of servers and workstations on SUSE Enterprise Linux which uses BTRFS as default file system, and it worked absolutely fine so far. Which is expected, after all BTRFS is fully supported by SUSE.

At home I run a small server using openSUSE Leap, with all my data on a large BTRFS volume. No problems.

BTRFS has some issues with its built-in RAID levels (RAID 5/6 is still not considered stable) but you can setup BTRFS on top of lvm RAID anyways.

and we have quiet some power interruptions here 
I have been thinking about to buy an ups, but my impression so far is that ext4 is very robust.

Back in the days when WindowsNT 3.51 and 4.0 (both which didn't support ACPI and didn't switch off the computer after shutdown) was still a thing I tended to didn't bother with shutting the OS down but just hit the power button, causing a hard power off. I was well aware that this might well corrupt my NTFS file system (and I had backups of my data anyways) but for some reason I never had any problems.

However, this doesn't mean it's a good idea to power off the computer without shutting down the OS, and there's a good chance that eventually that bad practice would have catched up with me (even though in my case the actual damage would be negligible).

The same is true for ext4.

So yes, if you have regular power interruptions then some UPS would be a good idea. You might want to consider avoiding the usual cheap standby models and go for an active one, because if your local power network is so unreliable then it's likely you'll see a lot of brown-outs and power fluctuations, too.

[Wuerstchenhund]

Well there was the absolute shit show that was 0.7.7 in 2018

Create a new file, get ENOSPC and any subsequent files in the dirent were orphaned and disappeared after a remount and/or reboot. Pretty major data loss bug that which showed exactly what little care and regression testing went into it. I had to spend a good 3 days unfucking something that was victim of that.

Then there was the issue where you issue a rollback and it blew the kernel out in mid 2019.

Big nope after that.

At work we looked at ZFS on Linux for a specific project but while I don't know the details (as this was done by our Linux guys) our experience was quite similar to above. And that was shortly before Coronavirus (August 2019).

On Solaris / FreeBSD yes that's fine. On Linux, no!

Well, let's say it's stable. There are still some inherent issues in ZFS which can make it problematic.

I run XFS on MD raid at home, it's OK. I don't even remember the exact reasons for choosing it over ext4. I think ext4 may be easier to recover from screwups because it has a somewhat simpler structure.

XFS is nasty. I've had all sorts of problems, particularly on filesystems backed by other volume management systems (3PAR / EBS). Sometimes, on some kernels, and depending on what phase the moon is in, if you xfs_growfs on a high inode usage filesystem it doesn't update the inode free list until you remount which you can't do when there are 1000 threads with FD's open on it. Live resize my arse 

If I remember right that bug has been fixed I think 4 or 5 years ago, in any ase it's no longer an issue.

XFS in general has an excellent track record, which isn't surprising considering that it was developed for enterprise use by SiliconGraphics back in the days, and after SGI made it open source since then has been maintained with a lot of support from Red Hat.

XFS is probably the most reliable file system for Linux that exists.

[Wuerstchenhund]

But recent hardware is your biggest concern - not the file system.

I will not setup a  BOOT drive based on NVM sticks nowadays - to be safe
I would even consider problems of SWAP on SSD devices as a big issue.

Why? Because, frankly, even consumer grade NVMe sticks are much more reliable than hard drives - and a lot faster.

There's no reason to put swap on slow spinning rust if you don't have to.

So I would setup using regular HD (cheap TB available on SATA 6G)
That is unlike to have a single issue.

So your advice is to put the OS on a single TB sized SATA hard drive? Seriously?

IMHO - ?  use a plain good SATA HD for your boot device
with regular  ext3 - SAFE , FAST, BUG proof and very reliable.

ext3 is reliable, but so are modern file systems as XFS and BTRFS.

One thing that ext3 is however is obsolete. If you really have to use something which originates from the old UFS of back then then at least use the current implementation ext4. Which works as well as ext3.

The rest options has too many issues to discuss here..
But as a DISTRIBUTION ?  I would radically consider to be

** FAR AWAY AS POSSIBLE OF ANY systemd DISTRO **
systemd is a total crap security hazard

Try  any ARCH based feature rich for newbies like
https://endeavouros.com/

IF POSSIBLE a DEVUAN  (DEBIAN without systemd)
order of magnitude better than any *BUNTU
which mostly is kinda goonie made...

I'm sorry but that reads a lot as if your last contact with Linux or hardware has been in 2003. And your aversion against anything flash is not rooted in reality.

I know that there is a general (mostly ideological) hate for systemd but I have no problems with it (I actually prefer it over the old init system, because it's more efficient and cleaner than the System V Init mess), and it's now the de-facto standard so you might as well getting used to it. The idea that systemd is a security hazard is nonsense, too.

I also would recommend to go with a common distro unless the pirmary goal is to make the OS a purpose in itself, which it will become due to the time and effort it takes to essentially copy what the various distros have already done so the user doesn't have to.

Having siad that, one thing I can agree with your advice of staying away from Ubuntu, simply because of the various bugs and annoyances it comes with (there's a reason why Ubuntu is regarded as 'the Windows under the Linux variants').

Personally, I keep with what's good enough for mission-critical businesses, i.e. the free variants of enterprise-grade Linux from Red Hat and SUSE (which are CentOS and openSUSE Leap). Especially openSUSE makes it easy because many of the things which require command line use in other distros can be done through YAST, SUSE's unique administration tool. It's by far the most user friendly distribution that there is.

A point actually on RAID. We found over 5 years that a single decent quality SSD has a better reliability outcome than two mechanical disks in RAID1.

Decent quality means Samsung Pro / HPE here, not "crucial instead of WD green"

I can confirm that. We use HPE and Dell, where most of the drives we get are either intel DC Series or Micron enterprise-class SSDs. We also have a lot of older HGST and intel Pro Series drives, down to very early HGST SSD400S.B SAS drives.

We have seen a few of the drives reaching the end of their specified life (i.e. the spare capacity has been eaten up) so they are replaced before they fail, but the number of actual defects has been minuscule compared to what we see for spinning rust (of which we also have a lot).

On a personal level, I had a Crucial 32GB SATA something (one of the early consumer SATA SSDs) and two Kingston consumer SATA SSDs which died prematurely (stopped working completely, the Crucial caused data corruption), and a number of older intel SATA SSDs (which I bought 2nd hand) and several Samsung 840 EVO/850EVO reach end of their life (well, they are consumer drives, and thus have very limited write cycles).

Right now I only have some spinning rust in my homeserver, where it's connected to a proper HPE Smart Array hardware RAID controller. Everything else in my home which uses mass storage (1 PC, 2 workstations, one Mac, a Macbook Pro and three PC laptops) uses flash.

[SiliconWizard]

I've personally used ext4 exclusively for a long time now. Never had any problem.
Disclaimer: I'm not claiming ext4 is 100% flawless. Just saying it has worked well for me.

As for ZFS, I think there was another specific discussion about it, so not going to comment any further. I would personally not consider using it.

[SiliconWizard]

A point actually on RAID. We found over 5 years that a single decent quality SSD has a better reliability outcome than two mechanical disks in RAID1.

Decent quality means Samsung Pro / HPE here, not "crucial instead of WD green"

I have no such comparative experience here. Never used SSDs in RAID configurations so far.
I can just remember a pretty bad experience with WD green HDDs (they were low power but rubbish) in RAID 5. They started failing one after another after ~3 years of run time (and that was not heavy use!) Replaced them with WD red HDDs at the time, which are still going strong.

[radar_macgyver]

My experience administering a small workgroup's servers and workstations:

ZFS: I've supported several ZFS NAS installations, all on Linux (all in the 20-40 TB range). I did follow the usual recommendations (lots of RAM, use ECC). The oldest of these was first set up in 2014 and has been working fine since. The ability to create a snapshot, and back up a delta from the last snapshot makes automated data replication very easy. Of course, this is for a centralized data store. Not sure how all these features help in a workstation context.

BTRFS: Really like the features, but have been bitten by the incomplete implementation too many times. I suppose it's ok for a single-disk setup. The lack of robust RAID5/6 is well-known. Recently, a server with BTRFS RAID1 lost a disk. I found out that apparently mounting a BTRFS RAID1 with a missing disk will work *once*, after which it will refuse to mount. Also, unlike md, btrfs can lose a RAID1 device silently (no provision for sending notifications). Red Hat/CentOS had good reasons, then, for deprecating btrfs.

ext3/4: goto for root fs on workstations and servers. No particular reason other than "it's the default" and that it's worked fine in the past. I usually use SATA SSDs or NVMe, and set up a cron job to rsync the filesystem over to a backup server.