Write once, read forever, backup media advice

[Infraviolet]

Before I start, yes I know the best ways to manage backups involve constant attention and routines involving checking the checksums at a defined interval and constant migration of data on to new media but...

And before format obsolesence concerns come up, the size of the backups in question is as big as it is because for any files in a specialised format I've also included fully offline installer files necessary so as to install the programs that can read the files on to any reasonably normal Linux distro circa 2023.

I'd be really interested in an last resort emergency backup which wouldn't need as much attention.

My parameters are:

..20GB (smallest case) to 100GB (largest case) size, 20 gigabyes of the most crucial stuff, but if there's a cheap way I'd consider this method for a further 80 gigabytes of relatively important stuff

..Write once to the whole medium, if there's free space left over I'm ok with it being "wasted" and never being able to write to it again in a future session

..Read forever, but I would be ok with something which one could only read perhaps 10 or 20 times so long as those 10 or 20 times could be separated across many decades of future time (so something you might read in 50 years time, and then again a further 20 years later, then after that second read use the copied data to make a new copy)

..Stays unpowered, entirely, and offline except during the initial writing and during subsequent readings

..No reliance on a cloud service, or other continually paid service, to host it. I want something I can rely on, rather than having to rely on a company that may go bust any day.

Hard-drives and flash memory are subject to bit rot.

Normal optical disks are considered 50 years life at the very best, with laser burnt dyes decaying by then. Stamped commercially mass-produced discs have much longer life expectancies, but there's no apparent way to make stamped discs yourself.

M-disc looked a good idea, but doesn't seem to be made any more in its most resilient DVD format, and again it's hard to find a burning method one can be sure of as Linux compatible. I hear that since M-disc blu-rays came out, the only type of M-disc still being manufactured, they've been barely more resilient than regular laser-written blu-rays, and again I'm not sure if good Linux compatiible blu-ray burners exist.
 
People often joke about paper or engraved stone as long term means of storage, but for multiple gigabyes this is a non-starter.

I do note that EEPROM type memory often is rated for 100 years retention, as is the type of memory used to store program data in microcontrollers (AVR and such datasheets imply as much). But the idea of building a multi-eeprom chip circuit for this job collapses when you find the biggest eeprom chips on sale are in the few megabytes (or few megabits where the filters by size on seller websites aren't so good) range, so assembling 10s of GB of these would be huge and very pricey. If there were eeproms of sevral gigabytes each these would seem suitable.

What is likely to be my best option then?
Thanks

[ebastler]

If you are targeting "many decades", you will also have to think about availability of suitable reader devices and interfaces to connect those to. (Beyond the media lifetime which you discussed.) 

I'm afraid it is not a realistic goal. Be prepared to check the media health every few years, and to copy to new media (and potentially a new format) at least once per decade.

EDIT: Software which can still interpret the old data formats will be yet another challenge...

Having said that, my 44-year-old Apple II is still working, and 95% of my old floppy disks are still readable too. Transferring the data to a modern PC is still supported via a serial transfer cable. The PC itself can't make any sense of most of the old data formats though -- but at least it runs Apple II emulators which can. So if you bet on the right horse (i.e. host system and media) you might be alright. But it's a bit of a gamble...

[IanB]

Your question is unrealistic.

For 20-30 years time, use a standard external USB hard drive. It is highly likely there will be USB support continued in the coming decades. USB devices from 20 years ago still work. Magnetic storage is quite stable, as long as you store it carefully away from magnetic fields and mechanical shock.

For 70-100 years time, you are not talking backup, you are talking archival storage.

One way to do this is to transfer data to microfilm using some kind of bar code or QR encoding, and store it with a detailed description of the encoding so that future archivists can construct a reader and retrieve the data.

But this is not at all dense as a form of storage, and not something you can just buy and use. So it is not an answer to your presumed question.

The only form of storage that has been proven to last centuries and still be recoverable is books, because it uses a stable medium and requires no technology to read. Technology changes too fast today for any form of technology to be proven durable. Hard disks and floppy disks from the 1980's can still (just) be read, but it's not clear for how much longer. This is not due to "bit rot", but due to mechanical degradation and obsolete interfaces.

In summary, if you seriously want data to be accessible in 70 or more years time, you need to worry about being able to read the medium much more than preserving the medium itself.

[soldar]

Punched paper tape

https://en.wikipedia.org/wiki/Punched_tape

or printed code similar to qr code and optically readable. 

[rhodges]

One way to do this is to transfer data to microfilm

Maybe movie film?

using some kind of bar code or QR encoding, and store it with a detailed description of the encoding so that future archivists can construct a reader and retrieve the data.

Or provide the source code in COBOL? 

[daqq]

There's this:

https://en.wikipedia.org/wiki/5D_optical_data_storage

But not sure if it's commercially available. On a budget, something inspired by it could possibly be done? Far lower data density "engraved" into glass?

[Infraviolet]

Ok, so "archival" might well be a better word than "backup" but I think the integrity of the data on the medium still outweighs the difficulty of reading the medium for these circumstances. Yes if one is leaving data to distant future generations then readability becomes a problem, but for leaving it for as long as one's own lifetime plus the lifetime of at most one more generation... well nowadays it is still very possible to buy USb connectable floppy disc drives and if it wasn't there'd still be a decent hope for someone who needed to read a floppy disk on a one-off basis to pay some money to the owner of a floppy drive reader then come round for a visit and copy the floppy on to more current media. The reason that is harder for things older than floppy drives is because at that point you're going back in to an age where relatively few readers were ever made anyway, only when going back to an age with relatively few readers made do you get scenarios where there really aren't any reader devices left. The data on the medium is what needs to be considered irreplacable, for anything within a century and a bit I think it is feasible to say that in the event you haven't maintained the reading system for yourself, someone somewhere will still have one you can hire. So it comes back to the question of what has the best integrity for the data ( 20GB to 100GB range, hence punched tape is a non-starter) and the medium on these sort of timescales.

"Software which can still interpret the old data formats will be yet another challenge..."
This would usually be a major point, but given the data already includes the installer files necessary to set all this software up on a typical Linux distro, one is looking at a scenario where to recover in many decades all one needs is the medium itself, access to a PC which can have a circa 2023 Linux OS installed upon it, and the reading device. I think that the existence somewhere of the right reader device and of PCs from the 2010 to 2030 era, or simply emulation and/or virtual machines which could act like a circa 2023 PC as a layer running atop futuristic hardware, is something one can largely rely upon. Unlike individual pieces of data themselves, I think one can rely on the "crowd backup" scenario, that someone somewhere who can be contacted and offered cash to help, in regards to reading equipment and things like common Linux* distro iso files.

*hence no future threat of a licence checking server somewhere meeting its demise and after that there being no way to activate software without paying absolute fortunes for the services of a very skilled reverse engineer if they could do it at all, software that depends on licence servers to activate or checks them periodically while running is a whole other can of worms which thankfully need not be considered here

[nctnico]

I've come to the conclusion that the best way to actually keep data is to have it spinning on a couple of hard drives in parallel spread over several locations. If one drive goes bad or gets upgraded, copy the data over. This beats hardware and filesystem obsolescence AND it gets you active monitoring of drive health.

[Someone]

So it comes back to the question of what has the best integrity for the data ( 20GB to 100GB range, hence punched tape is a non-starter) and the medium on these sort of timescales.

There is no absolute best. More realistically what is the budget you envision for the project?

I've seen these sorts of project many times before, and they usually fall over when x parameter must be >>> existing solution but cost must be ==. It is hard to beat mass production volume when value comes into it. Where is the world's data being stored? Spinning disks.

[SiliconWizard]

I hear that since M-disc blu-rays came out, the only type of M-disc still being manufactured, they've been barely more resilient than regular laser-written blu-rays,

Do you have evidence for that? That's not what I've seen.

and again I'm not sure if good Linux compatiible blu-ray burners exist.

LG ones seem to work fine on Linux, at least the internal, SATA.

That's something I've been meaning to test for a while - will probably do at some point.

[David Hess]

I think M-Disc is still your best option.

[Gribo]

Encrypt it, and store it as stenographic pictures on Facebook/Instagram/whatever..
 

[Rerouter]

Its not at all cheap for that kinda data amount, but its sounding like NOR flash with some redundancy, and the write pin switched to lock after the initial write,

Most other physical media will suffer from standard standard shift over that kinda time frame so you need something that can be read really easily and is so simple that its less likely to be replaced. Its not exactly easy to find a new 5" floppy drive still in working order.

USB so far is the more likely standard to survive some backwards compatibility, but memory card interfaces are another promising one, after all people are still using film cameras,

[tooki]

What's wrong with just using multiple hard drives and periodically copying to new drives? The important thing is to have multiple copies that are totally independent of each other. Ideally, you would use different drive models, replaced at different times, to mitigate the risk of a bad production batch that fail prematurely at the same time.

This method is also how you ensure ongoing compatibility with modern hardware. Realistically, there's just no way to expect to save to a medium today and guarantee you'll be able to read it in 70 years. The media needs to survive, as does the drive, as does the system the drive connects to.

Depending on your goals, you might want to copy not the files themselves, but rather entire disk images containing whole file systems. This can be useful if you need to e.g. restore an entire setup back to a physical drive (or to use the disk image for a virtual machine).


This reminds me, I have some old stuff archived on CD-R that I should probably just copy as disc images...

[nctnico]

and again I'm not sure if good Linux compatiible blu-ray burners exist.

LG ones seem to work fine on Linux, at least the internal, SATA.

Does blu-ray still exist? 

[BrianHG]

and again I'm not sure if good Linux compatiible blu-ray burners exist.

LG ones seem to work fine on Linux, at least the internal, SATA.

Does blu-ray still exist? 

Best way to enjoy 4K UHD movies.  Bursts up to 200mbit for playback, over 200gb storage per disc. Each 200gb disc costs less than 5$ to press.

[squadchannel]

Flash memory are very susceptible to quality. I have seen some that volatilize data in a few years.
Unfortunately, BD is out of date.

What about LTO tapes? You can get them cheaply if they are a few generations old.

Panasonic is discontinuing BD media.
Sony seems to be phasing out production.

[Phil1977]

A fail safe passive storage medium for 100 years definitely is difficult.

But if you actively care for your data then the combination of HDD + cloud should be the safest. Alternatively to the public cloud you can of course use any remote storage that´s available to you.

If you check accessibility of the data e.g. based on a yearly schedule, then the risk of failure at both locations should really be small. Much smaller than the risk of any non-redundant technology can be.

[PlainName]

I've come to the conclusion that the best way to actually keep data is to have it spinning on a couple of hard drives in parallel spread over several locations. If one drive goes bad or gets upgraded, copy the data over. This beats hardware and filesystem obsolescence AND it gets you active monitoring of drive health.

This. If it ain't live it's dead and forgotten.

Suffered several instances of storage media failing, readers becoming broken and unavailable, drivers no longer existing, etc. So anything needing to be kept is on a live system and thus gets into the normal backup regime. Reading isn't a problem, filesystems not a problem, obsolete hardware not a problem, having to think about how it's surviving not a problem.

[SiliconWizard]

and again I'm not sure if good Linux compatiible blu-ray burners exist.

LG ones seem to work fine on Linux, at least the internal, SATA.

Does blu-ray still exist? 

Uh, of course.

[Alex Eisenhut]

Does blu-ray still exist? 

Long live HD-DVD!

[Infraviolet]

Has anyone experience with M-Disc burning then? With a USB connectable burner of some sort? M-disc's own site hasn't been updated in years, and the only M-disc DVD media still for sale are old one's being sold at massively marked up prices (relative to what they were when in production), so it looks like M-disc blu-ray is the only M-disc left. I had seen rumours that M-disc blu-ray is less resilient than M-disc DVD, because it has to somehow make the "features" involved smaller and more vulnerable so as to cram more data in to the same disc area, but these were indeed rumours. If anyone has hard data, on the still-available blu-ray M-disc types, or any tips about Linux compatible external USB burners, I'd be very interested.

As for the redundancy NOR flash suggestion, are there long retention time NOR flash storage chips available at the gigabyte per chip sort of level?

As for the encrypt and steganograph in to shared images, the idea is quite good for smaller data volumes. But I'm not sure if many sites would let you upload enough images to hold 10s of GB within them, and uploaded images sometimes get subjected to compression which would destroy any extra data inserted in them. Plus, a lot of those sites delete everything uploaded by any account that has been inactive for more than a particular number of years.

Thanks

[SiliconWizard]

No experience yet with M-DISC as I said, although that was on my to-do list.
It would appear that the rumours you talked about originated on Reddit. Just found this video which deals with it:

while inconclusive, it seems none of the arguments that were given hold. Still doesn't mean that current discs sold as M-DISC are "genuine", but apparently Verbatim denied that the recent ones would be fake/rebranded regular discs.
Would be interesting to see a recent test of recent Bluray M-DISCs, unfortunately I didn't find much. Mostly old tests dating back several years.

I also mainly use Linux now and wanted to give this a shot. I was leaning towards an internal writer (LG WH16NS40 typically) which was tested compatible with Linux. But I also would now prefer an external writer if possible. These of course tend to be much more expensive and possibly less mechanically reliable.

I wouldn't trust Flash memory for long-term archival, especially if it just sits unpowered for long periods of time.

[rsjsouza]

TBH, for just 100GB of absolutely critical data I would buy several cheap(er) HDDs with at least CMR and create such redundancy that I would not worry about losing one or another disk. If needed, I would have around a few SATA to USB enclosures to have this data accessible in case the SATA itself goes away from newer machines. USB seems to be more resilient to interface changes.

As others have said, it is unrealistic to believe this will be still operational decades from now if absolutely zero maintenance is provided to this dataset.

[BrianHG]

TBH, for just 100GB of absolutely critical data I would buy several cheap(er) HDDs with at least CMR

Make sure you use several different brands.
You never know, stick with one brand, and for some odd reason the type of lubricant they use on the bearings seizes up after 25 years, then all the drives would be compromised.

I would also add SLC flash drives, if you cannot find them, then at least try to find MLC and refresh them every decade or so.  Stay away from TLC and up.

Maybe store you data on a few flash EEPROMS which still use SLC cells.
In 100 years, you will still find those who either kept or made their own eeprom programmers for historical restoration of antique devices like arcade video game consoles and ancient PCs.