OT: Backing up video files long term

[mariush]

Cameras record in full HD at about 17-28 mbps (varies from camera to camera, mine's best preset is 28mbps for 1080p 60fps).  At about 24mbps average and including sound, you're looking at around 3 MB/s or about 11 GB per hour of recorded video.

A 1 hour Youtube video can easily have up to 3-5 hours of content, takes, separate audio recording etc so it adds up.

I recently did a 3 part series about temperature measuring with an AD595 and a microcontroller, about 2 hours of actual camera shot video got on youtube from about 3.5 hours of actual footage ... the folder with raw recordings 46 GB.

[Richard Crowley]

Shooting ratios of 10:1 were not uncommon even back in the days of film. And with video, producers have become even more profligate. Shooting ratios for some types of productions (like documentaries) may approach 100:1.  Several multi-camera productions that I work on use ("iso recording") where we are recording each camera, in addition to recording the "live-switched" program.

[madires]

Do you think a company would give you 10TB for free? Wait until you know what you'll have to pay for that free storage.

They are the same as any number of other companies that offer free storage. They hope you will use their other services, either ad-supported or paid. In Tencent's case their biggest product is the extremely popular QQ messenger. It's a bit like MSN or ICQ. I use it to chat to my girlfriend a lot - it's very popular in China. They also do email, web search, games and all the usual random crap. You don't have to use any of it though, you can just use the free drive space.

I just looked up the end user price for a WD40EFRX (4TB, 24x7, SOHO NAS/AV usage), it's about EUR 150. That would be one of the cheapest 24x7 disks on the market (but you wouldn't find such disks in a SAN). We assume that we need 4 disks for 10TB and some redundancy. So the disks are EUR 600. They will run about 3 years, i.e. EUR 200 per year. We also have to add the SAN, datacenter, maintenance, spares, internet traffic, new Ferrari for the boss and what have you. Let's say that would be EUR 2300 a year. That means the cloud storage provider pays EUR 2500 a year to offer you 10TB for free. That would be a nice DSO, and you would get another one next year. All for free, just by watching some ads?

[mariush]

They probably resort to the same mentality as shared hosting... they know few will use more than a few GB of backup and if someone uses more than a certain limit, they can start looking into the account and just say they close the account because of storing copyrighted content or something like that.

Other companies like backblaze use their own custom cases and use software that works similar to rsync, storing only differences between files.. see blog post about their storage pods: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

[pickle9000]

Cameras record in full HD at about 17-28 mbps (varies from camera to camera, mine's best preset is 28mbps for 1080p 60fps).  At about 24mbps average and including sound, you're looking at around 3 MB/s or about 11 GB per hour of recorded video.

A 1 hour Youtube video can easily have up to 3-5 hours of content, takes, separate audio recording etc so it adds up.

I recently did a 3 part series about temperature measuring with an AD595 and a microcontroller, about 2 hours of actual camera shot video got on youtube from about 3.5 hours of actual footage ... the folder with raw recordings 46 GB.

It's not surprising (the size), the amount of effort that goes into a 2 hour production is.

I'm an always amazed at people complaining about backups and the cost of hardware. My reply was always you invest years gathering that information, it is your business. How much would it cost to replace it if it was lost? Would that be the end of your business? I wasn't a computer guy, production testing and setup, but still it's a part of being in business. 

[pickle9000]

Shooting ratios of 10:1 were not uncommon even back in the days of film. And with video, producers have become even more profligate. Shooting ratios for some types of productions (like documentaries) may approach 100:1.  Several multi-camera productions that I work on use ("iso recording") where we are recording each camera, in addition to recording the "live-switched" program.

I had a small amount of contact with local production cameraman 10 years ago. The shooting ratio is crazy. It is fun to watch productions being shot (not uncommon). Stargate often came out to a nearby lake and provincial park to film/shoot. It's very weird to the Stargate set up in a farmers field and a golf course just 100 feet away. 

[SeanB]

If you use pro photographers you might use around 5 photos from a shoot, but the photographer would have taken possibly 1000 photos and then developed the negatives only. That got rather expensive on 70mm film.

[Richard Crowley]

If you use pro photographers you might use around 5 photos from a shoot, but the photographer would have taken possibly 1000 photos and then developed the negatives only. That got rather expensive on 70mm film.

The difference between a professional photographer and an amateur is that the amateur shows you ALL their photos.

[SeanB]

The Pro only shows you negatives, then you select those that have promise and make a positive transparency of them. Then the final ones are blown up and edited to remove the inevitable marks.

[Shock]

I've now got two new 2TB drives, one WD, one Seagate and currently archiving all my raw video onto those. Doing about 65-70MB/sec still takes 7-8 hours per drive.
Will then store in separate locations at lab and home.
I've also got a new Bluray burner coming and will also burn one copy onto those just to be sure. They will be handy for regular access if I need it. Will try and get some quality TDK or some other brand discs, but they don't seem to be cheap.

Dave don't bother with Blu-ray.  It's as useful as tape drives which are totally a waste of time and expense.  Instead buy a 3rd disk in a usb or eSata caddy if you have to.

If you refer to my last post you also need to perform a proper verifies. Because backing up data is fine but you need to confirm it can be read byte exact to the original.  This needs to be repeatable over the life of the drive.

By using eSata you get the fastest backup and verify (read) speed you can.
Once you have sha256 hashes of the original data you no longer need to read it again.  You just read the backups.

You can use a utility like the PowerShell File Checksum Integrity Verifier (PsFCIV).
So no need for a backup program, all they do is add unnecessary complication and things to go wrong (unless you're backing up online databases).

[EEVblog]

Cameras record in full HD at about 17-28 mbps (varies from camera to camera, mine's best preset is 28mbps for 1080p 60fps).

I record at 17Mbps as a compromise. Used to be 12Mbps.

[EEVblog]

A 1 hour Youtube video can easily have up to 3-5 hours of content, takes, separate audio recording etc so it adds up.

I usually use about 80-90% of my raw material. A good lot of that is simply dead time at the start of the clip.
You record audio separately?

[David Hess]

Other companies like backblaze use their own custom cases and use software that works similar to rsync, storing only differences between files.. see blog post about their storage pods: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

I remember studying what Backblaze published carefully when I looked into doing this for myself on a much smaller scale.  This article discussing various storage controllers is informative as well:

http://blog.zorinaq.com/?e=10

[pickle9000]

The difference between a professional photographer and an amateur is that the amateur shows you ALL their photos.

I thought that was called "being a grandmother".

[apelly]

Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)

I have seen SEVERAL (six or eight) hard drives in long-term archival service simply be DOA when you try to power them back up.

Me too. Three anyway. One just yesterday.

[pickle9000]

Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)

I have seen SEVERAL (six or eight) hard drives in long-term archival service simply be DOA when you try to power them back up.

Me too. Three anyway. One just yesterday.

And thus the need for multiple backups. Hardware, media, fire, storage location, the list goes on.

[mariush]

A 1 hour Youtube video can easily have up to 3-5 hours of content, takes, separate audio recording etc so it adds up.

I usually use about 80-90% of my raw material. A good lot of that is simply dead time at the start of the clip.
You record audio separately?

No. I either let the camera record with its own internal microphones (5.1 dolby something) or I plug a lavalier microphone in it and then it switches to stereo recording automatically.

For one video, i recorded some segments without lavalier mic so the audio was 5.1.  I wanted to have that same 5.1 sound and that same audio processing the camera does when showing the screen capture, so I put the camera on tripod by the monitor and recorded with the camera in sync with the screen capture software. Then I copied video from camera to pc and extracted the audio and threw away the video, so now my screen capture segments had the original stereo sound produced by the desktop capture software (using lavalier mic in the pc soundcard jack) and the 5.1 sound from the camera.

I've had segments messed up for lots of reasons .. ambulance driving with sirens behind my apartment building, phone ringing.. not a big deal but if I stop I sometimes decide I could say that "paragraph" in a better way and reshoot the last 5 minutes of an explanation or something like that. It adds up.

[EEVblog]

Out of curiosity roughly how many gigs of files for say 30 minutes of YouTube video (raw and otherwise to be backed up)?

FYI, my latest 29min edited LVDC video is 4.86GB of raw video, with a total length of 39:46 over 53 clips.
There were a few clips in that were nothing and I didn't use. I don't bother deleting those in-camera at the time because it's just too much hassle to go into playback mode, select delete mode, switch back, and then reset my exposure setting (annoying bug in the Canon HF G10). Nor do I delete when nothing clips on the PC, it's just not worth the effort.

[pickle9000]

Out of curiosity roughly how many gigs of files for say 30 minutes of YouTube video (raw and otherwise to be backed up)?

FYI, my latest 29min edited LVDC video is 4.86GB of raw video, with a total length of 39:46 over 53 clips.
There were a few clips in that were nothing and I didn't use. I don't bother deleting those in-camera at the time because it's just too much hassle to go into playback mode, select delete mode, switch back, and then reset my exposure setting (annoying bug in the Canon HF G10). Nor do I delete when nothing clips on the PC, it's just not worth the effort.

To be honest that seems ridiculously good, both in raw video and shooting ratio.

[EEVblog]

To be honest that seems ridiculously good, both in raw video and shooting ratio.

Yeah, I think it's pretty good. Most of what comes out of my mouth ends up on camera.
If people are really keen to see that in action, I've posted some complete video of me shooting an entire mailbag video.
I don't fuss that much over the details, and I also rarely replay clips before I go edit. And also rarely go back to reshoot something after I watch it in the edit.

[EEVblog]

Dave, were you floating the notion of doing a video (or text blog) on your methods of videoblogging? The behind the scenes stuff about getting your edited video onto Youtube, monetising, dealing with comments and all the problems and traps a budding VBer might expect. Things that you have dealt with and hopefully solved.
Perhaps I imagined it because I would like to know. IIRC it was for your second channel.

Yes, I've mentioned it a few times it's something I want to do.
Just the methods of shooting/editing/tools etc would take several videos.
Not easy to shoot either because I'd have to setup multiple camera shots etc. e.g. to record how I record. Showing the effects of various settings etc. Not easy stuff to show that well.
The monetisation/comments thing would be another video again.

[SirNick]

Blu-ray is fine.  Will it last 20 years?  Probably not.  Some of the CD-Rs I burned on my first 4x writer are still readable, but things change.  Regardless, if you're not putting all your eggs in that basket, it's still a convenient way to store and catalog data for easy retrieval.  I envision something where each disc is labeled "raw footage from episode #327" or whatever.  If the disc fails, fine, go to the HDDs.  It's inexpensive to add an internal (or USB) reader to any computer, assuming it doesn't already have one, so accessibility is good.

As to HDD reliability, I have a whole stack of 40 and 80GB drives.  One developed bad sectors last time I tried to read it (about a year or two ago.)  Others still work and hold, for e.g., Hackintosh HDD images (because it's a laborious and painfully slow process to reinstall).  If I lose them, ... eh.

I have 4 drives (RAID-5) in a NAS.  When I replace those for bigger capacity drives, the old 4 become backup drives for the new ones.  NO RAID on the backups -- you don't want the hassle of trying to arrange for an appropriate read environment later on.  A USB-to-SATA cable or dock will be available for at least as long as the drive is viable.  This process of upgrade and trickle down keeps each generation of drives "fresh" within 2-5 years.  I plug them in every six months or so to backup new data.  (It really should be more often, but that's a risk I take.)  IMO, this is a decent compromise between leaving them at rest, and periodically checking for faults.

Since video archival won't be updated, doing a checksum validation of backed-up data once a year would suffice to catch drive faults or media read errors before both the BD-R and HDD have had a chance to fail.  Could a simultaneous failure still occur?  Yes, of course.  That's life.  Dave might be out one episode of raw video then.  I suspect he'll survive. 

[ve7xen]

If I were doing this:

- Replace commodity NAS with FreeNAS or NAS4Free running ZFS in a redundant mode as primary archive. Schedule regular scrubs. Comes with free replication to a remote server, if you decide that's feasible, plus block-level checksums and other useful features. Since this is fairly cheap, I would probably build a duplicate box at home and use USB media to carry the incremental snapshot home with you after you finish an episode. Don't remove your snapshots and you're safe from pretty much everything but simultaneous and catastrophic hardware or filesystem failure, and you have live access to all your data at any time.

- Secondary archive to BD with par2 or dvddisaster, keep in fire safe / safe deposit box / with data archival company

- Upgrade your network to GigE FFS!

[SirNick]

Ethernet speed isn't necessarily the bottleneck.  I typically get ~30MB/s over CIFS from a Win 7 desktop to my NAS.  All Gig-E.  But, the NAS is RAID 5 on JFS, all in software on a Mobile Core 2 Duo mini ITX motherboard.

Some of those little toaster NASeses are running at a few hundred MHz, and may or may not have any hardware acceleration.  Gig-E is merely a bullet point for marketing for all the good it's going to do you.

[ve7xen]

Ethernet speed isn't necessarily the bottleneck.  I typically get ~30MB/s over CIFS from a Win 7 desktop to my NAS.  All Gig-E.  But, the NAS is RAID 5 on JFS, all in software on a Mobile Core 2 Duo mini ITX motherboard.

You should be able to do much better. Any remotely modern hardware (which this is) should be able to saturate GigE if set up properly. I do software RAID6 and can get ~300MB/s read and 150MB/s write out of the array (which is poorly tuned for block size and stride). On NFS it saturates the GigE in either direction, but CIFS required a lot more messing around to work well and still doesn't match it for performance. Many times people complain about less than ideal performance it's due to using crappy Realtek NICs on one or both ends.

Some of those little toaster NASeses are running at a few hundred MHz, and may or may not have any hardware acceleration.  Gig-E is merely a bullet point for marketing for all the good it's going to do you.

Oh for sure, 10MB was just a "suspicious" number.