The end of the Hard Disks

[David Hess]

I would have gotten a modern DVD or Blu-ray drive and used M-Disc media.

I consider write speed to be a red herring since the lowest write speed yields the best write quality and drive life.  Blu-ray should be faster anyway even at its slowest write speed.

[DiTBho]

I would have gotten a modern DVD or Blu-ray drive and used M-Disc media.

I don't know about Blu-ray, but in my experience DVD is worse than DVD_RAM ​​when it comes to data reliability.

Specifically, if on the one hand I have never lost any data saved on a DVD_RAM ​​in over ~1000 cartridges (~50 cartridges per year) in the last 20 years, I have lost almost all the CDRWs that I have burned from 2008 onwards, and I have had at least a dozen cases out of 200, of DVDs with unreadable sectors.

DVD better than CDROM? Definitively Yes, but not as good as DVD_RAM

I consider write speed to be a red herring since the lowest write speed yields the best write quality and drive life.
Blu-ray should be faster anyway even at its slowest write speed.

As I indicated in the previous post, I have always burned CD ROMs at 1x with a (SCSI) Plextor CDRW unit, and I have always purchased good quality media, SONY and TDK, and yet I have lost practically everything!
(when I say lost ... I lost the data on CDROM, but I had copies both on hard disks and on tape, both DDS4 which absurdly performed better, and on LTO2)


DVD-RAM
1997, 1x
1999, 2x
2002, 3x
2004, 5x <------------  my default speed
2005, 6x
2005, 8x <------------  my other two units
2005, 12x (I know it exists, but I've never seen it marketed where I come from)
2006, 16x <----------- my new Japanese unit

I just bought the fastest DVD_RAM ​​in the world, but I don't use it to its fullest, different writing technology from DVDs and CDROMs, however I only ever use it at 5X, like with my other two units

[Nominal Animal]

Verbatim M-DISCs are supposed to have an extremely long lifetime (compared to writable CD and DVD discs using organic dyes), but apparently the 25G capacity M-DISCs are just ordinary BD-Rs using inorganic write layers since 2022.  This indicates Verbatim is pretty darn sure their BD-R's longevity greatly exceeds that of CD- and DVD-based media.

To me, the problem is the high cost of the drives and media: 100€-150€ for a drive supported in Linux, plus about 5€ for each Verbatim M-DISC BD-R (25 Gbytes, 4x write speed).  I just don't have anything worth that kind of cost to me, especially when I can get three 240Gbyte WD Green SSDs for under 100€, and use them for backups in a round-robin manner at a regular basis.

[Kjelt]

I thought the M-discs were made of some kind of stone material lasting at least a  thousand years, now you are telling me they changed the layers? 

[DiTBho]

To me, the problem is the high cost of the drives and media: 100€-150€ for a drive supported in Linux

If we talk about costs... it's a thorny issue.

I really don't like Tray-Loading optical units, because I think that leaving an optical medium bare (like CD-ROMs and DVDs) is not smart due to the way the cases are made and the ease with which they allow the medium to become contaminated with dust and humidity

My DVD_RAM cartridges are sealed on 4 sides, and have a slide that opens only when the cartridge is inside the optical drive. While it's not perfect, this helps a lot in my opinion. In addition to the fact that the cartridges can integrate an RFID-EEPROM tube, which I find both bloody convenient and bloody ingenious!

-

I've spent at least ***7 years*** looking specifically for ***that*** DVD_RAM drive unit, and as much as I've looked for it I've never seen it on eBay and the like.
Many many years ago I had the opportunity to try it in a laboratory in Japan (as well as Laser-memory units ... with 5xCDROMs all in one single cartrige. Love this!) and it was love at first sight, I immediately liked it, in addition to the excellent review given to me by the owner, a friend who I respect very much.

So, when I finally found one in Osaka I was full of different emotions: Incredibly happy to finally be able to use one for my personal stuff but also disappointed at the amount of money and time I was going to invest in importing.

~600 euros, for both the 120 and 80mm units, plus qty=40 120mm-carriages (double side, 2x4.6GB) with an RFID and qty=12 80mm-carriages (single side, 1.45GB), including S/H and customs. All NOS!

Both types of cartridge are really very rare (especially the 80mm) and the 120mm with RFID costs about the 2x after import and customs, so much so that I would almost be tempted to try to buy common 120mm cartridges (even single side, 1x4.6 GB) on Amazon and modify them. In the end you just need to find a way - without breaking anything - to install a small RFID tube, which you can still buy on Aliexpress or on Amazon itself.

We will see 

Anyway, I have to thank a colleague who I met there when I visited Tokyo years ago and he contacted me via email to tell me that he had found some interesting things in the warehouse, news published by a local app that he was consulting.

Like for my Japanese PDA that you practically only find in Japan, or the education writing units (which we, "Europeans", call it "distraction free, writing devices") that they only use in Japanese schools, it seems that this kind of thing is only used over there, where it costs much much less compared to their salaries. Blessed are the Japanese!

---

On the Kernel Linux side... it's a SCSI/50pin(SE) drive that doesn't create any problems with LUNs and has less "quirks" than most of CDROMs and DVD units. It falls into two categories:

• removable disk block device for the storage part
• char-device (there is a physical lvTTY 3.3v serial on the back) for the RFID part.

It's almost like it's a hard drive, but it's seen as /dev/sg with a large buffer (here, I hacked the kernel to have it), there's no problem with the partition scheme (I only use mac-parts), and there is no big problems with the filesystem.

However UDF/fs (mkudffs) is more suitable (I mean ... the media lives longer with UDF) for DVD-RAM because the number of times you can rewrite a DVD-RAM is limited to ~100K cycles, so if you delete and rewrite a file UDF tries to spread the writes equally over the media, while a filesystem like{  ext2, ext3, ext4, xfs, ...) does not.

So, it's also very kernel friendly.

[Nominal Animal]

If we talk about costs... it's a thorny issue.

Cost relative to perceived need and perceived benefits, rather.  There is nothing wrong in plonking down the $$ for a tool you actually use, even if it is more about comfort and satisfaction than strict needs.  Being happy with the tools you have is pretty important in keeping your motivation and productivity high.

[bson]

Cost relative to perceived need and perceived benefits, rather.  There is nothing wrong in plonking down the $$ for a tool you actually use, even if it is more about comfort and satisfaction than strict needs.  Being happy with the tools you have is pretty important in keeping your motivation and productivity high.

Indeed.  I like to say the most expensive tool is the one I never use, no matter how little I paid for it.  Whereas an expensive quality tool I use every day I will keep benefitting from long after I forget how much it cost me!

[tooki]

In addition to the fact that the cartridges can integrate an RFID-EEPROM tube, which I find both bloody convenient and bloody ingenious!

What do you mean by “tube” here? There aren’t any vacuum tubes in them, and I don’t think anything is tubular in shape, is it?

[DiTBho]

rfid eeprom in tube-shape.
they drown them in plastic.

[tooki]

rfid eeprom in tube-shape.
they drown them in plastic.

Oh, like the little capsules that are also used to “chip” pets?

I’ve never heard them called “tubes”, and the lack of search results under that name makes me think it’s not what they’re normally called. What isn’t obvious (even after trying to find out) though is what their generic name actually is! Rods is maaaaybe the closest I found.

[radiolistener]

If it ends, it will be bad. Where is alternative for long term data storage?
SSD and USB-flash and SD cards are not designed to store data for a long time.

[Messtechniker]

For SSD long term data storage we will need something equivalent to watch movers.
I.e. a device which switches the SSDs for long term data storage on and off in certain intervals.
Maybe an idea for a patent ....

Oops. Mentioning such here might be considered as prior art.

[DiTBho]

Oh, like the little capsules that are also used to “chip” pets?

Yup

[Kjelt]

Where is alternative for long term data storage?

98% of all the data is worthless in 10 years anyways. The 2% that is worth long time storage and has to be transferred from generation to generation is written in books and will be spread over the whole world in so many places that it can never get lost.
Humans tend to over exaggerate the importance of their tiny little time of the entire timeline of the world.
And we generate dayly too much useless data.

[Messtechniker]

98% of all the data is worthless in 10 years anyways.

Maybe. But some 12 years after I had translated a manual for a one-off piece of equipment, I was asked for a copy of the translated manual, which the company had lost somehow. Fine with me. Saved the company several xxxx Euros and quite a few tech hours. And there was also an incident of a similar patent related kind with some judicial implications. These things do happen.

[BradC]

98% of all the data is worthless in 10 years anyways.

I have a box here of photographs, slides and super-8, some of which are > 100 years old and still in "relatively" good condition. They show things as esoteric as the Queen Marys first visit to the port of Fremantle, through to my grandparents and parents history. Based on the rate of deterioration, I'd be surprised if they weren't still readable in another 100 years.

I have a (several) hard drive full of photos and video of my dead son. They document and detail a significant part of his life and are the only remaining visual artifacts we have of him. We can print them out, but the prints deteriorate rapidly when compared to a proper photographic process. Therefore we're left to curate and manage this digital archive. It won't be worthless in 10 years, but unless it's replicated onto new media periodically it also won't be readable. I don't trust the LTO will still be readable in 50 years (as specified), and even if it is maintaining a working drive to retrieve it will be a challenge. So we're left to ensure the storage media is kept up to date. Currently magnetic trumps solid state for longevity, so I don't see hard disks going away any time soon.

To add to that already mentioned by Messtechniker, we've had numerous opportunities over the years to leverage data that is well and truly > 10 years old finding it valuable not only to our immediate family.

[CatalinaWOW]

Bradc, you have identified just one of the reasons why long term data storage is useful.  For more than 2% of data.  If widely useful data is so widely distributed that it "will not be lost" why are so many on this forum searching for old manuals and old software?  I will agree that not all of the children's homework and snapshots of parties littering our memory devices is worthy of long term storage.  But a sizeable fraction is widely useful for human lifetimes, and an even larger fraction is useful for ten years or more.  Things to consider when defining storage life requirements are legal issues.  In the US contracts and taxes require terms of a few years for almost everything, with patent and copyright law stretching those lengths to decades and centuries respectively.

As an aside there are photo printing processes that should have life and quality comparable to traditional photographs.  None in consumer level printers currently, but there are those who print for museums and others, although at prices that will limit the quantity.

[Picuino]

Have you read about this news?
https://www.muckrock.com/news/archives/2024/jul/10/grace-hopper-lost-lecture-found-nsa/

We are living in the digital dark age.
https://en.wikipedia.org/wiki/Digital_dark_age

[Kjelt]

I don't know about US or Australias government funded agencies for the preservation of historic audio and video recordings?
In Holland we do have such an agency who often gets material from legacies or heritage. They make sure the material is stored safely and if known it will deteriorate or is vulnerable for example the famous movies on celluloid it will preserve them in a more long lasting format.
If you have such material that is historical important I would contact such an agency for their opinion, often or they return the original or when really important copies.
The material that is important for your family that is for you to preserve. We have pictures from 1880s from my grand-grand parents and many of which we do not even know their names anymore. Is it important? For us maybe for the rest of the world not really.
In any case media becomes pretty cheap, you can create and refresh and maintain many backups, where 30 years ago the fotographs would only reside in a single book where a fire could destroy it permanently.
It is a question for each and anyone how much money they think it is worth to preserve and keep for the future, remember the more data you store the more time it takes to label select and keep.

[Connecteur]

I'm a little concerned about data longevity on flash RAM. I don't know much about the technology, but as die sizes get smaller, towards 1 nm, I'm not sure the cells will be as robust as before.

I have a couple of 1.5 TB micro SD cards, but I never use them for long term storage.  I have a few of the older memory cards from about 20 years ago, and their data is still intact.  I'm hoping that the longevity of data stored on the new high capacity cards is just as reliable.  I suppose it's too early for there to be any real world data about this.

[Connecteur]

I suppose I have my own data preservation regime, going back almost 40 years.  Every time I got a newer computer with larger storage capacity, I copied my whole archive forward on the new machine.  Looking back through my archives, I can find files I created back in the 80s.

[Connecteur]

It's never seemed worthwhile to comb through it all, deleting the obsolete stuff.
Much easier to just save it all.

[coppice]

I'm a little concerned about data longevity on flash RAM. I don't know much about the technology, but as die sizes get smaller, towards 1 nm, I'm not sure the cells will be as robust as before.

I have a couple of 1.5 TB micro SD cards, but I never use them for long term storage.  I have a few of the older memory cards from about 20 years ago, and their data is still intact.  I'm hoping that the longevity of data stored on the new high capacity cards is just as reliable.  I suppose it's too early for there to be any real world data about this.

High density flash has a pretty short retention period. When powered up the cells are regularly refreshed, but stored in a draw somewhere, a drive may lose some of its contents in as little as a year. Its less about the whether they are 1nm or 28nm than the techniques which differentiate the 100 year cells of the type of flash in an MCU, from the cells in the type of flash in a high density flash drive.

[Circlotron]

If I had a 1TB SD card or similar, how long would I need to have it powered up for it to refresh itself? A similar time it would take to write data to all locations like in normal usage?

[Halcyon]

98% of all the data is worthless in 10 years anyways.

I have a box here of photographs, slides and super-8, some of which are > 100 years old and still in "relatively" good condition. They show things as esoteric as the Queen Marys first visit to the port of Fremantle, through to my grandparents and parents history. Based on the rate of deterioration, I'd be surprised if they weren't still readable in another 100 years.

I have a (several) hard drive full of photos and video of my dead son. They document and detail a significant part of his life and are the only remaining visual artifacts we have of him. We can print them out, but the prints deteriorate rapidly when compared to a proper photographic process. Therefore we're left to curate and manage this digital archive. It won't be worthless in 10 years, but unless it's replicated onto new media periodically it also won't be readable. I don't trust the LTO will still be readable in 50 years (as specified), and even if it is maintaining a working drive to retrieve it will be a challenge. So we're left to ensure the storage media is kept up to date. Currently magnetic trumps solid state for longevity, so I don't see hard disks going away any time soon.

To add to that already mentioned by Messtechniker, we've had numerous opportunities over the years to leverage data that is well and truly > 10 years old finding it valuable not only to our immediate family.

Cloud storage is a good and viable option. In the background, you have people working in data centres to ensure the data remains available. There's a bunch of options out there for low to medium data volumes which is quite economically priced.