OT: Backing up video files long term

[SirNick]

Definitely.  I have no doubt it can be improved dramatically.  I tried "dd if=/dev/zero of=/mnt/md" after I built it, and got significantly higher throughput than what I see from across the LAN.  I don't remember what it came out to... pretty sure it didn't even come close to a couple hundred MB/s, but it was obvious that performance over the network -- including and especially CIFS -- is a constraint.

That said, throughput is so rarely a concern that I haven't bothered to pursue any of the bottlenecks.  Can stream 1080p?  Check.  Decent chance of saturating a single-spindle source or destination?  Check.  OK, good enough then.

And of course there's an excellent chance that both ends use Realtek chipsets, as both the NAS mobo and the PC are using onboard LAN.  That's hardly the worst sin though.  It's also stacked up on top of a cabinet in the laundry-slash-boiler room, keeping those 1U PSU CapXon electrolytics cooled by 25-30C ambient temps.   Wouldn't be caught dead implementing a system like that for a business, but at my own home?  Eh.  "Good enough for Australia." 

[mariush]

Ethernet speed isn't necessarily the bottleneck.  I typically get ~30MB/s over CIFS from a Win 7 desktop to my NAS.  All Gig-E.  But, the NAS is RAID 5 on JFS, all in software on a Mobile Core 2 Duo mini ITX motherboard.

It depends. For example, the $1000 NAS I linked to a couple of pages or so before ( Synology DS1813+ ) has 4 gigabit ports that can be linked together for a 4 gbps throughput.  If you have a smart switch capable to do port trunking, you're in business.
That NAS  uses 2.13 Ghz dual core Atom...

ps. You can spend 400$ on a hardware raid card with 8 sata ports like this one : http://www.newegg.com/Product/Product.aspx?Item=N82E16816151137

Add ~150-200$ on cpu, mb, case and ram and optionally get a 2-4 port gigabit network card from eBay and do port trunking.  The software's going to be a problem though, there is freenas and other ready made solutions but it's still not going to be as polished as that Synology comes from the factory.

[Rigby]

Ethernet throughput matters a lot on what is copied.

Lots of small files will not saturate gig-e.  A few large files will saturate any link.  There is an overhead per file that impacts transfer speed in any per-file upload/download/copy.

To maximize throughput, zip up the files with a non-compressing archive format, and transfer them in that form.

[Legit-Design]

http://www.smallnetbuilder.com/lanwan/lanwan-features/30201-need-to-know-jumbo-frames-in-small-networks
One word, Jumbo frames

[mariush]

Way to go, posting an article last updated in 2007 and a graph showing cpu usage on a 300 Mhz Sun server.

Now a dual core i3 can probably run in circles and saturate 1gbps even without jumbo frames.

[SirNick]

ps. You can spend 400$ on a hardware raid card with 8 sata ports like this one : http://www.newegg.com/Product/Product.aspx?Item=N82E16816151137

You can, and (IMHO) that would be a waste of $400. 

I have a rack chassis with 4 bays.  I have a motherboard with four SATA ports (and a CF card slot).  Connectivity is covered.

"But what about hardware-optimized RAID then?"  Good point, Bob.  But where you see a purpose-built ASIC that increases IOPS, I see a data striping algorithm that requires the original hardware to recover data from my disks.  I once had a server with a proprietary controller card that made RAID really fast.  Then the card died, and it was NLA, and I no longer had access to the data on those disks.

"But what about backups?"  Yes, Bob, I had backups.  After a few hours of rebuilding the server to the point where the backup software client could be installed, it only took another hour or so to get the data back.  Or, you can pull a commodity PC off the shelf, plug in the NIC and power cables, stuff the hard drives in there, and have any Linux Live CD recognize the MD RAID volume and you're back online in 15 minutes.

On another note:  I once did something colosally stupid and offlined two of the four disks in my RAID-5 set.  An ordinary controller would have dumped the volume at that point.  But, MD RAID just complained furiously until I gave it the "dash F" option, which stands for "Shut the dash-F up and re-mount the volume regardless".  After replaying the journal (and running a background integrity check), it was 100% accessible.  No lost data.  I am a big, big fan of Linux MD RAID.

[ve7xen]

ps. You can spend 400$ on a hardware raid card with 8 sata ports like this one : http://www.newegg.com/Product/Product.aspx?Item=N82E16816151137

Add ~150-200$ on cpu, mb, case and ram and optionally get a 2-4 port gigabit network card from eBay and do port trunking.  The software's going to be a problem though, there is freenas and other ready made solutions but it's still not going to be as polished as that Synology comes from the factory.

For the task at hand, I would say hardware RAID is a big no-no. It's expensive, you're not targeting high IOPS, and now your data depends on having a working RAID card of correct model and firmware. You can get an 8-port SAS HBA for under $100, if you even need one, since the 6 or 8 onboard SATA ports common these days are likely to be sufficient, at least to start. This guy did it recently with a pretty nice build for around $400 sans-drives. I would personally throw in an Intel PCIe NIC just because onboard NICs suck, but hardly required.

Also keep in mind that port-trunking/link aggregation chooses a link based on a hash of some portion of the header (often just the source/destination MAC addresses, though sometimes layer 3 or even layer 4 on higher-end gear). In other words, you're primarily concerned with throughput to a single client, this won't help at all, since all the traffic will end up on the same link.

Nas4Free is pretty polished these days, and gets you ZFS, which is hands down worth any slight increase in effort required over a commercial product. Automatic scrubbing and block checksums are exactly what you need for long-term archival. Snapshots, and ones which can be incrementally copied to a remote system by sneaker- or Inter-nets would also be pretty handy. Most of the commercial boxes are based on standard Linux technology, so you can plop the disks into any machine to recover your data if the NAS box goes tits up, but a nice benefit of building on Nas4Free/FreeNAS/Linux is that you know you can do this in a pinch and that the vendor isn't doing something weird (like Drobo and presumably others).

Edit: Ninja'd but I'll leave the post anyway

[Legit-Design]

Way to go, posting an article last updated in 2007 and a graph showing cpu usage on a 300 Mhz Sun server.

Now a dual core i3 can probably run in circles and saturate 1gbps even without jumbo frames.

nice video with real life test.
That NAS box probably doesn't run processor faster than 300 Mhz something? It all depends on situation.
In my opinion it's a fair comparison of almost decade old server vs modern NAS box.
Just found this quote:

I have a small ioMega NAS box that actually gets almost double the throughput using jumbo frames, but that's because the large frames allow more data transfer with lower CPU utilization (it has a low-end ARM-compatible cpu)

[rdl]

Any time the discussion of backing up data arises I am reminded of this video.