Another SSD bites the dust

[magic]

I woke up today and saw a bunch of disk write failures in the kernel log on one machine. Linux ended up resetting the drive and it came back, but everything reads as zeros.

Not even some junk brand, but a pretty pricey Intel from 8 years ago. SMART no longer works too (of course, why not ) but the last run that I have saved showed some 4.5TB total writes at 600GB capacity and everything at 100/100.

I think I'm starting to appreciate the failure modes of spinning rust. I have had quite a bunch of problems with those, but none has ever gone out completely. Will have to see if secure erase brings this crap back to operation.

Maybe the proper way is to buy two SSDs and run them in RAID1

[magic]

Secure erase seems to have reset it to a working state. Half of SMART data are at zero, though.
Maybe I should put it up on an auction site as "almost unused"

I'm really losing respect for those things a lot today.

[Wuerstchenhund]

That mirrors my experience with consumer grade SSDs (and that includes the widely hyped Samsung EVO drives).

For my own use I moved to enterprise class SSDs since then, which have a longer endurance and in general seem to be a lot more reliable. Haven't regretted it since.

Having said that, have you checked if your drive has the latest firmware?

[bd139]

Nothing wrong with consumer SSDs. Just make sure you have backups!

Incidentally out of a sample size of about 450, we haven't had a single Samsung Evo or Pro failure.

[BradC]

Incidentally out of a sample size of about 450, we haven't had a single Samsung Evo or Pro failure.

You must have skipped the 840 Evo "amnesiac edition". Don't get me wrong, I love Samsung SSDs, but that was an absolute clanger.

[bd139]

Ah yeah. These are all 850's or later. The 830s were problematic.

My main one is a 970 Evo Plus. My laptop has a random SKhynix one in it (Lenovo's default now apparently  )

[BradC]

Ah yeah. These are all 850's or later. The 830s were problematic.

I have 3 830s in service. They've lost a yard or two of pace over the years, but they're still cranking along.

I bought the 3 Samsung 830 and 3 Intel 330 together. The Intels are still in production. The 830's have been moved to non-critical machines.

Code: [Select]

/dev/sdd - INTEL SSDSC2CT240A3 - 6 years 217 days 3 hours
/dev/sdq - INTEL SSDSC2CT240A3 - 6 years 218 days 12 hours
/dev/sdc - INTEL SSDSC2CT240A3 - 6 years 244 days 1 hours

This is my favourite spinner though :

Code: [Select]

/dev/sda - Hitachi HTS542580K9SA00 - 11 years 197 days 18 hours

Model Family:     Hitachi Travelstar 5K250
  9 Power_On_Hours          0x0012   001   001   000    Old_age   Always       -       101106


Just keeps trucking.

[magic]

Yeah, latest (i.e. the last ) firmware revision. Earlier firmwares were known to lose data on unclean shutdowns so it was the first thing I checked after receiving the drive.

It showed the same "BAD_CTX 00000115" serial number as the units after unclean shutdown and I assume it just means corruption of internal data structures (flash translation layer, whatever). What disappointed me is that it apparently happened due to uncorrectable errors during normal operation. And that I was either unlucky enough to lose the root of the FTL tree, or that the drive simply detected some problem somewhere and stopped working altogether instead of allowing me to recover whatever was still recoverable. I suspect the latter and I'm not happy about it in the slightest.

I have copies of most stuff, but the OS will probably need to be installed from scratch

[george.b]

A single point of data doesn't mean much, but my OCZ Agility 4 has been happily chugging along for over 5 and a half years of power-on time and 49.24TB lifetime writes. SMART says it is at 86% remaining drive life, whatever that means - reallocated sectors count is zero.

[Ranayna]

Careful with "enterprise class" SSDs
At least HP and DELL both had cases where some internal counter of the ssd would run over at something like 30.000 power on hours. Poof, that SSD is just a brick now, good luck recovering your data. I do not recall the OEM of the drives, at least one was made by LiteOn.

As with any storage medium: backup is king.

[Halcyon]

I do remember a firmware update for Intel SSDs a while ago. There was an issue with some of the consumer drives bricking.

Careful with "enterprise class" SSDs
At least HP and DELL both had cases where some internal counter of the ssd would run over at something like 30.000 power on hours. Poof, that SSD is just a brick now, good luck recovering your data. I do not recall the OEM of the drives, at least one was made by LiteOn.

Probably Intel. They released a firmware update for these drives as well: https://downloadcenter.intel.com/download/28673/SSD-S4510-S4610-2-5-non-searchable-firmware-links/

[Mechatrommer]

Never put important data in ssd. Ssd only for operating system and aplications. When corrupted or damaged, easily reformatted. Data in 2nd drive TB HDD. Keep backup every 1 or 2 years... 2cnts.

[Monkeh]

It's so cute that you think an HDD is a better place to keep your data.

[BravoV]

At my main desktop, the primary (soon to be secondary  ) boot drive NVME SSD on W10, and 2nd boot drive ordinary SATA SSD loaded Linux Mint, and local spinning rust 2 x 8TB on RAID 1.

Important data is never stored in SSD, only at the local RAID 1 volume, and this is also get duplicated incrementally to my NAS (RAID 6). While for backup, another set of HDs (duplicate set) as offlined/air gapped backup to cover the NAS and also for archival.

The only exception of data that is considered not super important, like VMs is stored in 3rd SATA SSD for speed, as spinning disc is too slow, and this is also gets incrementally backup to local RAID 1 volume.

Primary NVME OS gets imaged routinely to the RAID 1 volume, and should the NVME toasted, all I need is just buy another new one, and then restore the saved drive image that took just 10 minutes, and restored gracefully, and this is annually tested.

[james_s]

I had someone bring me a PC that just stopped booting one day. I discovered I could access the SSD and read it however it became read-only. Format, delete partition, anything I tried, it would appear to succeed but then the contents remained unchanged. I guess as far as failures go that's not a bad way to go.

[Mechatrommer]

It's so cute that you think an HDD is a better place to keep your data.

the key is proven and established statistics for many decades... but still as said, need backup/change drive every few years (statistics says 5-10 years if you dont knock it for no reason) my intel SSD past 7 years i think, i havent check but kids is not complaining so i guess its still working. so SSD is gaining trust, if i can have SSD or NVME 1TB at $100+ i will buy anytime and store my data dont worry, i'll buy another next year for backup. another more proven way for thousands of years storage is animal skin, suit yourself. another unproven way but claimed can store beyond human extinction is that archival place in north pole storing bad jokes programming codes. ymmv and have a nice days.

[Monkeh]

It's so cute that you think an HDD is a better place to keep your data.

the key is proven and established statistics for many decades... but still as said, need backup/change drive every few years (statistics says 5-10 years if you dont knock it for no reason) another more proven way for thousands of years storage is animal skin, suit yourself. another unproven way but claimed can store beyond human extinction is that archival place in north pole storing bad jokes programming codes. ymmv and have a nice days.

I've clearly run a few more drives than you - one drive can never be trusted, no matter what type it is. HDD failures occur far more than you think, you just don't notice them.

[BravoV]

It's so cute that you think an HDD is a better place to keep your data.

the key is proven and established statistics for many decades... but still as said, need backup/change drive every few years (statistics says 5-10 years if you dont knock it for no reason) another more proven way for thousands of years storage is animal skin, suit yourself. another unproven way but claimed can store beyond human extinction is that archival place in north pole storing bad jokes programming codes. ymmv and have a nice days.

I've clearly run a few more drives than you - one drive can never be trusted, no matter what type it is. HDD failures occur far more than you think, you just don't notice them.

+1 , I've experienced this my self, wife's HD drive experienced bit rot without any error message nor warning. Several of her "highly sentimental JPG files"  got hits last time, hence she is now have raid 10 at her desktop as she has tons of photos & videos from her cellphone. 

At my archival (not backup) drives, which are mirrored too, I do another layer by PAR-ring the data at 50%.

[Halcyon]

+1 , I've experienced this my self, wife's HD drive experienced bit rot without any error message nor warning. Several of her "highly sentimental JPG files"  got hits last time, hence she is now have raid 10 at her desktop as she has tons of photos & videos from her cellphone. 

At my archival (not backup) drives, which are mirrored too, I do another layer by PAR-ring the data at 50%.

Most RAID set ups won't protect you from bit rot. If a file becomes corrupt, that corrupt file will just be replicated across the array.

To prevent that, you'll want to use a more robust file system like ZFS that detects and corrects that kind of corruption.

[BravoV]

+1 , I've experienced this my self, wife's HD drive experienced bit rot without any error message nor warning. Several of her "highly sentimental JPG files"  got hits last time, hence she is now have raid 10 at her desktop as she has tons of photos & videos from her cellphone. 

At my archival (not backup) drives, which are mirrored too, I do another layer by PAR-ring the data at 50%.

Most RAID set ups won't protect you from bit rot. If a file becomes corrupt, that corrupt file will just be replicated across the array.

To prevent that, you'll want to use a more robust file system like ZFS that detects and corrects that kind of corruption.

Yeah, I'm aware of that, was backing Monkeh's post, that single HD as backup (even offlined), is just too risky.

[Mechatrommer]

I've clearly run a few more drives than you..

good for you.. practically speaking, it works for me with my only few (2 or 3 maybe) HDD generations. if it doesnt for you, i'm sorry you can use RAID or mirror drive whatever fancy they are there are always options for better.. or maybe just a simple SeaGate drive (my trusted brand), hunglow brand is highly unrecommended some of them got paper weight inside, big no no.

btw i can make program for data match or maybe even magnetic strength detector HW to see field difference or collapse in few years, or automated checksum maker but... i dont feel like to because its just practically working for me. i got corrupted pictures or videos from time to time, sometime even the one i downloaded few weeks ago (bad H264 encoding i guess) but not that i cant see them entirely, and i can always redownload the same or better entertainment. but so far, i havent seen my text code comes out in jumbled ascii/hex garbage, or corruped hobby project files, those are #1 priority data...  so thats good enough for me. ymmv and be safe... your data

+1 , I've experienced this my self, wife's HD drive experienced bit rot without any error message nor warning. Several of her "highly sentimental JPG files" 

typical, top on the list storage problem, the SWMBO. when asked what did you do? i didnt do anything.. did you knock it? yeah but its just a little bit it should be no problem! now she got the biggest affordable SP storage in town, not much complain so far except the super slow Win10 i formatted few weeks ago.

[Halcyon]

+1 , I've experienced this my self, wife's HD drive experienced bit rot without any error message nor warning. Several of her "highly sentimental JPG files"  got hits last time, hence she is now have raid 10 at her desktop as she has tons of photos & videos from her cellphone. 

At my archival (not backup) drives, which are mirrored too, I do another layer by PAR-ring the data at 50%.

Most RAID set ups won't protect you from bit rot. If a file becomes corrupt, that corrupt file will just be replicated across the array.

To prevent that, you'll want to use a more robust file system like ZFS that detects and corrects that kind of corruption.

Yeah, I'm aware of that, was backing Monkeh's post, that single HD as backup (even offlined), is just too risky.

Oh yes, absolutely.

A friend of mine once asked me if I could have a look at a drive for him and attempt to recover data. He returned home one day and a metal-on-metal scraping noise was coming out of his (still spinning) disk. I popped the lid off and the head had made contact with the platter and cut some nice grooves into it. The entire inside of the drive was covered in what used to be the magnetic surface of the disk platters. Needless to say, I threw it in the bin and gave him the "backup talk".

[bd139]

+1 , I've experienced this my self, wife's HD drive experienced bit rot without any error message nor warning. Several of her "highly sentimental JPG files"  got hits last time, hence she is now have raid 10 at her desktop as she has tons of photos & videos from her cellphone. 

At my archival (not backup) drives, which are mirrored too, I do another layer by PAR-ring the data at 50%.

Most RAID set ups won't protect you from bit rot. If a file becomes corrupt, that corrupt file will just be replicated across the array.

To prevent that, you'll want to use a more robust file system like ZFS that detects and corrects that kind of corruption.

Just a heads up on that. ZFS won’t protect you from corruption. It’ll try to and tell you when it can’t. It’s far from perfect. 3PAR got the job in the end after that evaluation cycle.

But as mentioned planning for corruption is what you really need to do.

The best strategy I’ve found for personal data is offline validation. I’m using a semi manual tool for that “Beyond Compare” and I will periodically before performing a backup allow it to checksum both source and destination directories. I keep two completely segregated offline snapshots of everything I have, one encrypted and on my person and one unencrypted and stored securely at home. The rest floats around between my desktop and laptop on onedrive which has file history. That covers point in time recovery, complete disaster and carnage too.

[Wuerstchenhund]

Careful with "enterprise class" SSDs
At least HP and DELL both had cases where some internal counter of the ssd would run over at something like 30.000 power on hours. Poof, that SSD is just a brick now, good luck recovering your data.

Yes, this one:

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00092491en_us

I agree that this was a bummer but it made the headlines because it's also exceptionally rare (intel had a similar issue once, but as far as I remember that was in their lower tier drives). In addition, these are server drives, and usually operate in some kind of redundancy setup (RAID) so unless they all reach 32768 hrs at exactly the same time this is unlikely to bring down your storage.

What it does highlight, though, how important firmware updates are for SSDs (much more so than for spinning rust), and that it's important to keep your SSD firmware at the latest level.

Which, of course, requires that there, actually, are firmware updates.

I do not recall the OEM of the drives, at least one was made by LiteOn.

LiteOn doesn't make enterprise drives (thank god!). The drives were all made by SanDisk, in fact these were all SanDisk Ligntning Ascend SAS drives. Which even without the bug weren't great, but when you buy OEM drives for your server it doesn't really matter who makes them as the drives are covered by the server's support contract anyways.

For individual purchase (i.e. to use at home) OEM drives, especially those from Dell, are a pretty bad choice as the firmware can only be updated with Dell tools on a suitable Dell system (server). It's better to stick with the original brands (Seagate, WDC, Micron, Samsung) which provide firmware updates and tools which work on every system.

As with any storage medium: backup is king.

Indeed.

[bd139]

I find it’s best to just buy everything from HPE. Then you only have one vendor to blame who generally can’t wriggle out of it by blaming another one. Supermicro were good at that when we had storage problems.  They blamed it on the drives. So we bought new ones and still had the issue. That was an old 1U SATA RAID box so it put me off their higher end offerings.