OT: Backing up video files long term

[EEVblog]

It would also be wise to back up individual episodes (or a couple, depending on how much space it takes) on BD-R.  Keep these in sleeves in a binder or storage box so you can go right to the material of interest if/when you decide to revisit old footage.  If the disc fails, go to your HDD backup.  This will keep the power-cycle count low on the spindle motor.  The optical disc won't care how much it's used.

Yes, I think I'll do this in addition to the two hard drives.

[EEVblog]

I have used every kind of media over the years for backup but I find now I tend to use two hard drives, fill, unpowered, and store one off site. I use exactly the same drive and I prefer WD drives (personal preference).

As someone else noted, I wouldn't use the exact same drive for both backups, but two different brands.
I think I'll go buy a bunch of 1TB WD and 1TB Seagate drives.

[EEVblog]

Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)  I have had some develop media errors over time (so have two backups if the data is important), but the usual fate is throwing away those that were just too small to be of any practical use anymore.

Any archive or backup system, regardless of how complex, is a simple (or maybe not so simple) matter of probability theory.

[ovnr]

Well, if you want an online solution, Amazon's S3 Glacier storage is pretty nice. $10/month per terabyte stored, pretty reliable ("99.999999999% durability of objects in storage", survives the complete loss of two datacenters). Obviously more expensive than disks over time, especially for massive heaps of data - but I consider it manageable. I don't use it myself (yet) - I currently run two 6x2TB RAIDZ-2 arrays and generally swap disks when they get five years old; this costs me around $400/year compared to $2000/year, but it's offsite and as reliable as you can hope for.

I'm going to be replicating all of my irreplaceable data onto S3 eventually; it's just a couple TB or so. The rest can be rebuilt, but at considerable cost.

[pickle9000]

Moving to a more modern (backup) system allowed me to reduce the physical size of the backups, and throw out some very old (large and cluncky / slow) hardware. I'm not saying it was unreliable just old. It was also good to see that the data was still viable.

The main motivation was to reduce the size from that of a couch sized space to that which could fit in a shoebox. Remember not just the media and or drives but a couple old computer systems to go with it.   

[EEVblog]

Why not just buy the most reliable one?

Because the point is that if you use two identical drives then you have the added (albeit small) percentage that a design or manufacturing defect in that model of drive will render both of your backups useless in X years time.
Especially so if you buy two identical models from the same manufacturing batch. Likely if you go to the same store to buy them, as they will likely get them in boxes from the same batch.

"The most reliable" model is data that is essentially only available in hindsight.

[Monkeh]

Currently backing up 61 episodes worth of footage (284GB) to the networked NAS, and that's going to take 8 hours at 10MB/sec 

And that's why they invented gigabit ethernet. 15 years ago.

[pickle9000]

Why not just buy the most reliable one?

Because the point is that if you use two identical drives then you have the added (albeit small) percentage that a design or manufacturing defect in that model of drive will render both of your backups useless in X years time.
Especially so if you buy two identical models from the same manufacturing batch. Likely if you go to the same store to buy them, as they will likely get them in boxes from the same batch.

"The most reliable" model is data that is essentially only available in hindsight.

I agree that using 2 different brands is a better system than what I have, no question. I do run the WD drives I use for a few hundred hours to help ensure they are up to snuff. That's not the same as using completely different hardware. My personal preference for WD relates to problems with the other brand Dave mentioned. Even so I will probably throw one into the mix next time I do a backup. 

[Richard Crowley]

Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)

I have seen SEVERAL (six or eight) hard drives in long-term archival service simply be DOA when you try to power them back up.  This is why a single hard drive is NOT reliable backup or archival media IME. I have also seen raid arrays fail because they used several drives from a single defective batch.

Hard drives are OK for backup and archival storage, as long as you take appropriate precautions.

[David Hess]

They are basically magneto-optical, which as we know is extremely robust and long lived. You still need to store them in a fairly cool, dark place so that they don't warp over time. Archival grade discs are being advertised as ideal for keeping your precious family videos for generations and testing does suggest that the claim is not exaggerated. Of course at 50GB/disc you will need a lot of discs, and ideally two copies for security. Unlike tape you shouldn't have any trouble reading the discs in 20 or 30 years time, considering that a modern BluRay player will still play an 80s CD.

BluRay is not even close to magneto-optical except maybe in reliability and I have doubts about that.  I still use my 3.5" magneto-optical drives which hold up to 640MB.

I would consider archival BluRay a good choice except for price.  Even multiply redundant hard drive storage is less expensive but I wonder what their unpowered lifetime is.

[David Hess]

I have had good results backing up onto good single layer DVD+R disks but those will be too small for your use.  I generate and save PAR2 recovery files along with the data and write the disks out at the slowest burner speed with verification and both UDF and ISO file system data so in the event of a damaged disk, recovery is easier.  The only disks failures I have had in 10 years were either do to cheap media or actual physical damage and almost all of them were completely recoverable.  I suspect BluRay would be similarly reliable under these conditions.

For larger data sets I would go with two identical file servers running Linux or BSD using 4 disk RAID5 or perhaps one of the newer file systems which support data integrity like ZFS or Btrfs.

[nctnico]

Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)

I have seen SEVERAL (six or eight) hard drives in long-term archival service simply be DOA when you try to power them back up.  This is why a single hard drive is NOT reliable backup or archival media IME.

I disagree. That particular failure mode is (often) caused by hard drives running too hot. If you keep hard drives cool you won't have this problem.

[mariush]

I thought about this as well and came to the conclusion that just plain hard drives in a RAID is the most future proof in long term.

Bluray is only ok for temporary storage, a few months to a year. After that, just like with DVDs, you can't really trust the media.  It's cheap enough (2-3$ for 25 GB) to have a 3rd backup on blurays but not as a primary backup.

Tapes are cool, lots of storage relatively cheap and in small space but like others said, you have the problem of continuously having to upgrade your tapes to whatever comes up next year or to keep buying tape reader/writers (and have some as backups) just in case they're discontinued at some point and your main unit breaks down.

Even though I respect free_electron and his experience, I don't agree with his suggestion of turning on hard drive, storing the data and then turning off the drive. Maybe it's silly, but I don't trust a hard drive NOT failing at startup all of the sudden, I'd be more comfortable with having the hard drive running 24/7 and potentially catching some error in close to real time and have the hard drive replaced.
I would go with one of those units that allow you to connect 4 or more hard drives and set up a raid5 or better and check from time to time the state of the disks and take corrective measures when needed.
In your case, you could have an identical unit at home (but maybe with other drives or another series just in case there's a bad batch) and you could just configure the home unit to slowly copy throughout the day what you have stored at the office (doesn't have to fast, it can be just 1mbps upload for example, you have it running 24/7 anyway)

Maxxarcade (of youtube "fame") recently upgraded his raid storage from a server with lots of ATA drives to a Synology DS1813+ ( http://www.newegg.com/Product/Product.aspx?Item=N82E16822108138 ) that can hold 8 hard drives, has 4 network ports that can be linked together for higher throughput  etc etc. It's a bit expensive, but there's the cheaper DS1513+ also which can only hold 5 drives.
I think it's a good investment if you're serious about backup.



PS. For extra security, you could use a tool like PAR2 to create recovery volumes for the video files. With about 20% of the video size in recovery volumes, you can recover pretty much any video file if it's just a bunch of bits or bytes messed up randomly (it won't help if there's a continuous 10-20 MB of totally damaged data)
md5sum and sha1sum will just make a checksum and that tells you if the file got corrupted or not, but par also is capable of repairing the files.

[rob77]

ok i didn't read the whole thread, but my 2 cents :

1. blue-ray - do you guys have experience with 5+ years of storage of those ? i guess not. if blue-ray (it's cheap) then refresh every 2-3 years to fresh media.

2. tape drives - most expensive but it's a industry standard and very reliable. the best for archiving are the WORM LTO tapes  (Write Once Read Many) archived in controlled environment (humidity, temperature). but as mentioned it's freaking expensive solution to set up, with a good price per TB once  it's there, but still i wouldn't recommend this for archiving videos just because of the price tag (not just the hardware and tapes, but the archiving of the tapes in a controlled environment gets expensive). and let's not forget - you would need to refresh the tapes as well - it's usually 10+ years guaranteed for properly stored WORM media.

3. hard drives - cheap and reliable to some extent. but still i would recommend to keep 3 independent copies offline and regular check/refresh

i think at the end of the day it doesn't matter whether blue-ray or HDD - the cost is very similar for those methods and both would require regular refresh.

[madires]

avoid nas boxes as the plague for this.. nas boxes often use a custom or weird  file system. if the nas hardware dies the drive is unreadable unless you have an identical nas. never mind the nas may use ext3 or anther linux filesystem. if you have it formatted in a linksys nas it will not be readable by a wd nas... even though they both use linux internally. there is metadata and journalling and other proprietary stuff and that may not be compatible between nas boxes from different brands !

The problem is based on the version of the linux kernel and the RAID tools used. I've managed to rescue NAS disks (RAID 1) by attaching them directly to a recent linux system. But I agree that USB disks are the best choice for backup at the moment, unless you have to backup a datacenter.

[David Hess]

Forget hard drives, they are totally unsuitable for long term offline storage. Modern drives need to periodically re-write data to the platters. It slowly degrades over time, and the surface of the disc itself degrades due to dust and various natural processes that affect metals. The magnetic fields are extremely weak and even the earth's shifting magnetism slowly each bit down.

The coercivity of the media in hard drives is incredibly high to support their high data density in the face of thermally activated relaxation so external magnetic fields, even strong ones, have an insignificant effect.  Magneto-optical media is tough for the same reason.

They come with errors from the factory and rely on the controller to map them out on the fly - they certainly don't get a 100% test. Don't get me wrong, they are miracles of technology, but they do rely on being powered up and able to re-write data in the background to avoid slow degradation and correct errors before blocks become unreadable.

I agree about using them for long term offline storage although I suspect they will fair better than many alternatives.  For that reason, I would use them in a small self contained server and run them through a complete background scrub while updating the contents.

Amazon S3 Glacier is very interesting. Hopefully competition will continue to push prices down, but 1TB is enough for me to back up a couple of machines. Sure, I have more data than that, but I can always re-rip/re-download a lot of it. It really depends if there is any good backup software that supports it. I will look.

At least where I am in the US, internet based storage is totally useless because of both bandwidth and ISP transfer caps.

[IanJ]

Hi,

My own backup solution for my Vegas Movie Studio video files is a 2nd drive on my PC where the files sit, and a QNAP NAS running RAID 5 (4 drives).
I also have a 2nd QNAP NAS running RAID 1 (2 drives) which is where I backup all my other data etc including my 1&1 dedicated server (web & email) backups.
All drives on both QNAPs are WD Enterprise edition RE3 drives.

Works for me, dead easy to administer. Have only ever had 1 drive failure (a 13 month old WD RE3) on the RAID 1 NAS.

Tips: Keep the server cupboard temperature stable, keep it dust free, and install a wee UPS.

Ian.

[XOIIO]

Hi,

My own backup solution for my Vegas Movie Studio video files is a 2nd drive on my PC where the files sit, and a QNAP NAS running RAID 5 (4 drives).
I also have a 2nd QNAP NAS running RAID 1 (2 drives) which is where I backup all my other data etc including my 1&1 dedicated server (web & email) backups.
All drives on both QNAPs are WD Enterprise edition RE3 drives.

Works for me, dead easy to administer. Have only ever had 1 drive failure (a 13 month old WD RE3) on the RAID 1 NAS.

Tips: Keep the server cupboard temperature stable, keep it dust free, and install a wee UPS.

Ian.

I'll guessing you use one that sends a shutdown signal when the batteries get below a certain percentage in case of long term power failure?

[Rigby]

The Library of Congress, when they designate something worthy of the time and effort required, save copies on hard disks and verify the data periodically, then when new hard disk tech comes out, they copy to new media while drives for old media still exist and are serviceable.

Last I heard they use a mix of LTO tape and hard drives.

LTO isn't going anywhere, and the new drives that come out are backwards compatible, IIRC.  LTO is pretty safe.  Hard disk is pretty safe.  I wouldn't bother with optical anything.

[SL4P]

As mentioned previously, you are archiving.
You need to look a the access requirements vs the cost / time to retrieve a specific title etc.

Read about HSM strategy (Hierarchical Storage Management), and consider the various combinations of online, near-line and offline repositories.  The use of stub files can also provide near-instant online access, while loading the large part of for complete playout while the stub is playing.

[IanJ]

I'll guessing you use one that sends a shutdown signal when the batteries get below a certain percentage in case of long term power failure?

That would be a bonus but actually my own UPS I fitted to help with a few 15 minute outages I have had in the past couple of years. The batteries have always lasted.
I also have the UPS keep my modem/routers alive.

One other issue with NAS boxes is I think it is important to power cycle every 6 months. I had a raid drive fail on one of my qnap NAS boxes, it was fine right up to the power up when the drive just failed to start. Hidden problems! Can't imagine maybe more than 1 drive being affected and then suddenly after a power cycle no data!

Seriously, for what Dave needs I would just stick with a good NAS box with quality drives.......and use it for other data also.

Ian.

[free_electron]

Forget hard drives, they are totally unsuitable for long term offline storage. Modern drives need to periodically re-write data to the platters. It slowly degrades over time, and the surface of the disc itself degrades due to dust and various natural processes that affect metals. The magnetic fields are extremely weak and even the earth's shifting magnetism slowly each bit down. They come with errors from the factory and rely on the controller to map them out on the fly - they certainly don't get a 100% test. Don't get me wrong, they are miracles of technology, but they do rely on being powered up and able to re-write data in the background to avoid slow degradation and correct errors before blocks become unreadable.

I'm gonna be very blunt here : Stay off of whatever you have been smoking because you are talking out your ass. What you are saying is pure nonsense.

-The surface of the disk is not the recording layer ! That layer is buried. The surface layer is a lubricant with underneath a layer of cristalline carbon (basically grown diamond)
Then there are several other layers before you even hit the actual recording layer.

-earth magnetic field have no impact. You can put a permanent magnet on a harddisk case while it is running. No harm will happen to the data.

-all platters have bad zones. These are NOT mapped on the fly. The mapping happens during manufacturing and is stored in large lookup tables on the drive. If, during operation, the drive encounters problems with a sector it can add this to the existing table. The surface is 100% tested during manufacturing when the drive formatter writes the servo signals and sector marker wedges. This is information is doublechecked and any problems are marked in the lookup table. All harddisks go through a 24 to 48 hour burn-in cycle. It takes about 3 hours for the formatter to do its work ( do not confuse the formatting process in the factory with formatting a filesystem. It has nothing to do with each other. Once a drive is completely assembled and sealed you cannot alter the physical formatting as you need an additional head that is now no longer there. If you look at a harddisk you will find silver metallic small labels , either on the side , or between the cricuit board and case. They cover a hole and or slit. That is where the formatters head enters the drive to write the synchro track on the platters.

-as for the rewriting of data. That is absolute nonsense. We try to minimise the amount of writing.

I design those things for a living. None of what you are saying is true.

[free_electron]

avoid nas boxes as the plague for this.. nas boxes often use a custom or weird  file system. if the nas hardware dies the drive is unreadable unless you have an identical nas. never mind the nas may use ext3 or anther linux filesystem. if you have it formatted in a linksys nas it will not be readable by a wd nas... even though they both use linux internally. there is metadata and journalling and other proprietary stuff and that may not be compatible between nas boxes from different brands !

The problem is based on the version of the linux kernel and the RAID tools used. I've managed to rescue NAS disks (RAID 1) by attaching them directly to a recent linux system. But I agree that USB disks are the best choice for backup at the moment, unless you have to backup a datacenter.

The problem is each nas seems to use its own thing. There are rarely updates to the firmware , some have bugs, some use reiser, so e use ext 2 , so e ext3 , some have journaling on, others off.
Nothing but trouble. Some even store the runtime image on the drive.

Usb box, format drive in ntfs and off you go. All computers can read ntfs these days. No need to screw around with obscure things .

[Richard Crowley]

-earth magnetic field have no impact. You can put a permanent magnet on a harddisk case while it is running. No harm will happen to the data.

You don't even have to put a magnet on your hard drive. If you disassemble one, you will end up with one of the strongest permanent magnets you have in the house. So strong, you could seriously hurt yourself with one if you got pinched between it and a piece of ferrous metal.

You can find all sorts of complete twaddle on the interweb. caveat lector

[Richard Crowley]

Forget hard drives, they are totally unsuitable for long term offline storage. Modern drives need to periodically re-write data to the platters. It slowly degrades over time, and the surface of the disc itself degrades due to dust and various natural processes that affect metals. The magnetic fields are extremely weak and even the earth's shifting magnetism slowly each bit down. They come with errors from the factory and rely on the controller to map them out on the fly - they certainly don't get a 100% test. Don't get me wrong, they are miracles of technology, but they do rely on being powered up and able to re-write data in the background to avoid slow degradation and correct errors before blocks become unreadable.

Some of what you say is true of "non-volatile" solid-state memory (i.e. "flash memory"). Which is why nobody I know who understands the physics considers NVram to be suitable for "archival" storage.

But NONE of what you wrote has anything to do with conventional rotating-disk hard drives. Complete nonsense.