Ideas for storing frequently updated data in a FLASH chip

[peter-h]

I am using the FLASH chip with 512 byte sectors. It is a specific variant. It may be that the 528-byte one can "just work" but a) I have a fair stock of them (obtained at considerable hassle - all these chips are "totally sold out", especially the more common 528-byte ones) and b) I have no appetite for the regression testing after substituting it.

It is possible that it is the same chip inside, but I think it is not the same because you can just keep clocking the SPI and the data bits just keep dropping out (up to 64k clocks IIRC) so you could use either type if you only ever used that mode and never addressed by page #.

Interesting point about indeed using 0x00/0xFF and escaping these. That may actually be the simplest overall, and with zero chance of a collision on the pattern and binary data.

And yes RLL then becomes doable and can dramatically reduce data storage (in the right situation; can make it worse in the wrong one ).

[Nominal Animal]

I am using the FLASH chip with 512 byte sectors. It is a specific variant.

Right.  To be clear, I was surprised to find that there are chips that do support 528 byte sectors!  I am only a hobbyist in the electronics stuff, after all; I'm much more a software person.  Consider the 528-byte sector suggestions just an overflow from my internal musings that considered all the ways such a Flash chip could be useful in my own designs .

The circular buffer using a fixed-position counter, containing a markup/escaped stream of log data, seems the best option to me, comparing features vs. code complexity.

The only suggestion of mine that I would not immediately throw out, is to implement the API and logic in normal C/C++ (whatever is portable to your MCU code), and test it thoroughly.  If you can have your victims colleagues test the API, do not provide the sources for the implementation of the API; instead, provide it in library form, and ask their opinions about the usability of the API.

Things to look for is whether you implement synchronous and/or asynchronous operations (i.e., whether a read or write blocks until the operation completes, or whether it only initiates it with a subsequent wait/completion checking if (and optionally blocking until) it has completed).

[voltsandjolts]

Not all SD cards did wear leveling, but modern SD usually do, ex. WD Purple cards.

I like this answer.
WD Purple indeed has wear levelling according to reddit, so it must be true.
I could quite believe that since its aimed at video storage but it would be worth checking directly with WD.

Downside is I can't find any stock to purchase!

[peter-h]

Yeah... reddit... must be true

One challenge is making sure you are getting a non-counterfeit one. Most things which can be counterfeited are counterfeited. SD cards, camera batteries...

Another issue I have is that someone would write an RTOS task to write to this thing, and another RTOS task to read from it. I am already using the FreeRTOS mutexes to lock stuff like SPI3 (which is used for a variety of devices, so maybe that's the best way to do that.

[SiliconWizard]

Since I'm currently working on something related...

Note that if you're using SD cards with exFAT for data logging (which has an essentially predictable storage usage), this filesystem allows to avoid having to update the FAT most of the time, as you can allocate contiguous space for files and in this case, only the allocation bitmap is used, the FAT is ignored. You can also pre-allocate space for files, which makes it possible to update the allocation bitmap itself only very infrequently. The only thing you'd have to update frequently would be the directory entry for the file you're currently writing to (for updating the current size of the file). But thinking about it, for wear leveling, it would be possible to write the directory entry at different locations on a regular basis to avoid having to write at the same spot all  the time. Not very hard to implement once you have a working exFAT implementation.

[amyk]

Not all SD cards did wear leveling, but modern SD usually do, ex. WD Purple cards.

I like this answer.
WD Purple indeed has wear levelling according to reddit, so it must be true.
I could quite believe that since its aimed at video storage but it would be worth checking directly with WD.

Downside is I can't find any stock to purchase!

All SD cards do wear leveling, except perhaps the very earliest tiny ones using SLC small-block flash. Whether they do it well is a different story.

The same applies to USB drives and SSDs.

[voltsandjolts]

All SD cards do wear leveling

I'm sceptical about that statement, can you link some references to back that up?

[DC1MC]

Exhibit 1:

"The SD card spec has NO entry for wear leveling. That is completely dependent on the SD manufacturer to handle that if they so choose. We have seen that some likely do, while others very much do not (beware the super cheap knock-off SD cards)."

Exhibit 2 (WARNING: Reddit cancer):
https://www.reddit.com/r/raspberry_pi/comments/ex7dvo/quick_reminder_that_sd_cards_with_wearleveling/

Verdict:
Use a specialized hi-cycle EEPROM.

[peter-h]

Do SD cards contain CPUs? They need to if they do wear levelling.

But in any case I would not regard it as wise to use one (which does) for an industrial product which has to run for years. Using one for just storage is fine (SD cards will "always" be around, especially if you allow for micro-SD) but to rely on specific features is asking for trouble. I've seen this in avionics where various SD and CF cards have been used and all have become a nightmare. One company used a Sandisk 48MB CF card (in a product launched c. 1999) which had some special security feature (no idea what it was but it could prevent reading it in a normal CF reader, even at block level) and you can guess what happened. 10 years after 1999 the cards were unobtainable. But the product is still widely used. Now those cards are going for $ hundreds.

EEPROMs are not a solution; a device with 1M write endurance merely postpones the day or reckoning by 10x. In my project I have 512k, spread across 1024 512-byte pages, so wear levelling across that lot would enable the storage of 50GB of total data before it all wears out. A non wear levelled implementation with a single device, writing the 512 bytes over and over, would need a write endurance of 100M writes.

[DC1MC]

Do SD cards contain CPUs? They need to if they do wear levelling.

But in any case I would not regard it as wise to use one (which does) for an industrial product which has to run for years. Using one for just storage is fine (SD cards will "always" be around, especially if you allow for micro-SD) but to rely on specific features is asking for trouble. I've seen this in avionics where various SD and CF cards have been used and all have become a nightmare. One company used a Sandisk 48MB CF card (in a product launched c. 1999) which had some special security feature (no idea what it was but it could prevent reading it in a normal CF reader, even at block level) and you can guess what happened. Now those cards are going for $ hundreds.

EEPROMs are not a solution; a device with 1M write endurance merely postpones the day or reckoning by 10x. In my project I have 512k, spread across 1024 512-byte pages, so wear levelling across that lot would enable the storage of 50GB of total data before it all wears out. A non wear levelled implementation with a single device, writing the 512 bytes over and over, would need a write endurance of 100M writes.

Yes, the SD cards do contains specialized MCUs, for the people in the know they may even get firmware SDKs to implement their own version of wear leveling, the evil ones are using these SDKs to change the size of the attached NAND flash and this how one can get a (fake) 128GB SD card for 4.99.

Normally the cards/EEPROMs for industrial use have internal wear leveling features and they cost a lot not because they're using some magic algorithms but because to have the specified reliability the NAND has to be 1) of good quality (SLC) and 2) grossly oversized to allow for transparent spreading of writes over a lot of cells.

For large amount of data and long time use, there are like a gadzillion of papers, doctoral thesis and patents, basically covering all the desperate attempts to make reliable a storage media that is inherently unreliable. One can choose all forms of error corrections, RAID configurations optimized for NAND, filesystems optimized for NAND and so on.
But no matter what you do, the flash memory is basically a consumable, and whatever industrial machine is there, if it has flash storage, it has a finite operating life and has to be replaced periodically, before it dies catastrophically, there is no silver bullet to solve this currently, but only to try to push the operating time (a bit) outside the warranty period.

 Cheers,
 DC1MC

[NorthGuy]

would enable the storage of 50GB of total data before it all wears out.

How much do you need for your product? Do you write at regular intervals or randomly? How often? How big are single writes? Maximum size? Average size? What is the maximum size of the data that must be preseved before the old data are deleted?

I don't think it is possible to develop a sensible solution without knowing answers to these questions. There are many ways to do write leveling (or not to do if it's not needed). A single algorithm won't be the best, or easiest to implement, for all the situations.

[DavidAlfa]

The fake cards are just some values tweaked in the partition table. Nothing really awesome, there're tools to do that. Enter the desired size: 512GB. Done.

I remember once in the 2000s, a friend had a Windows Me computer, had to be formated every month or even less because...oh boy I wish I knew what the hell he was doing to screw up so often.
Once he did whatever he did and the 40GB disk showed as 2TB. That value was absolutely mindblowing back then, like 200TB today.
Of course that was another fu** up he did. Whole data was damaged and I had to format it for 10000th time.
It took more than 10 years to forget the installation key!
But I still remember part of the W98 key...20 years later! MP7TK-KPT9F... hard BSOD times  .

Sorry for the offtopic, it brought some memories!

[peter-h]

I don't know the future usage of this thing, so I am just trying to do the best reasonable approach.

The API could be just a crude one: write a page, read a page, erase a page, erase all, etc. Then the next coder could decide what wear levelling scheme to use.

I can't think of any API which could hide the wear levelling implementation. One can do that with a file system, of course, although such a file system would not be compatible with FATxx/Windows. One can also do it at sector level, by remapping sectors (AFAIK hard disks do that) but I think that requires either an assumption that a sector doesn't have an inherent max write # count or that you can reliably mark a sector as unusable after you detected a failure.

[peter-h]

I am still working on this but in the meantime I did one simple thing: only if the 512 byte page has changed, will it be written. That should offer some limited protection from runaway code.