The obligatory file system question...

[abyrvalg]

But does your embedded application really need all “desktop” FS features? Directories, human-readable file names, timestamps etc. Many quite big systems (i.e. cellular basebands) running FSs stripped down on this side (numeric ids for file names, no dirs/timestamps/access rights), but with improved reliability/power loss tolerance (things like “write new record then mark old one as deleted” etc). Can’t recommend any name unfortunately, the only one I remember is ancient Intel’s FDI.

[SiliconWizard]

Of course not, as I clearly mentioned in this thread. There are many applications for which you absolutely don't need the FS to be easily accessible from a "PC" by the end-user. And just because we may be using SD cards (for reasons I mentioned earlier, there may be others) doesn't mean we do that to let the end-user easily read and write data to it directly.

But in cases for which the SD card is at least physically accessible to the end-user - and thus you know they may be tempted to take it out and read it on a PC - having a small FAT partition on it is a good idea. It avoids them accidentally formatting the card if they ever do this, and yes, you can take advantage of it to put some Readme file, license, whatever... including a mention that the user can't do anything useful with the card themselves.

[peter-h]

I am working with FatFS and it is very good.

The only issues I found where where I needed file system access before the RTOS (FreeRTOS) got started. I don't remember what the problems were exactly but this was achievable only by compiling it in the non-reentrant mode. The reentrant mode could not be used pre-RTOS-start. I solved it by using the FreeRTOS mutexes around the file ops, once RTOS code was running and potentially multiple threads could be reading/writing files.

On this project the file system is also accessible via USB as a removable block device with 512-byte sectors, and this works great.

Date/time stamps are easy but obviously do need a running RTC to be meaningful.

[SiliconWizard]

Speaking of RTOS, I'm currently considering how to use FatFs (or another lib) in a non-blocking way. That precisely doesn't seem easy if at all possible unless you use an RTOS - in which case, FatFs calls are still blocking, but can be preempted...

[Berni]

Yeah non blocking use of FatFs is tricky.

The FatFs code is fast and won't really execute much on its own. Its mostly the storage device driver that will be waiting for data. So what you can do is make the driver call a function during its wait loop. This function could do some background processing or just do an iteration of a "Hot loop" (the cooperative way of multitasking where you infinitely loop over calls to various functions and expect them to return in a timely manner).

If you are mostly writing to file then another solution might be to introduce a cache in the storage driver. That way writes go into RAM instead and so return instantly while the data slowly gets flushed to disk in the background.

Tho i don't see an issue of FatFs not working until you start FreeRTOS. That OS pretty much "boots" in an instant since it just has to initialize some structures as part of the boot process. If the application should not start doing things straight away then you can just simply start up those threads later. If this is for logging use so that you could log errors during boot then this is the wrong way to do it in the first place. Since even once the system is running a crash may leave the system in any weird state (including a locked up disk driver) so the crash handler function has to reinitialize all hardware required to write the error report, so typicaly this would spit the error out of the UART and save it to flash so that it can be written to disk the next time the system comes up. If you want to write it straight to disk then you might have to stick that tiny PetitFatFs inside your crash handler, init it from scratch and do the log write.

[SiliconWizard]

Yeah non blocking use of FatFs is tricky.

The FatFs code is fast and won't really execute much on its own. Its mostly the storage device driver that will be waiting for data.

Yep. But that will be triggered by calls to FatFs functions, making them in turn blocking... A related point is that if you want to use non-blocking IO in your driver, you're a bit doomed.

So what you can do is make the driver call a function during its wait loop. This function could do some background processing or just do an iteration of a "Hot loop" (the cooperative way of multitasking where you infinitely loop over calls to various functions and expect them to return in a timely manner).

Yes. I've seen that kind of stuff done too, and I do not like this approach much at all. It will make your cooperative multitasking look like spaghetti.
But if you don't have a choice...

That raises an interesting issue though. Depending on the way you want to implement "tasking", some libraries are less than ideal. I'm still giving it some thought, but going for my own implementation is still not out of the question...

That said, I think this would be an excellent example of how using a preemptive RTOS can have clear benefits - since this is a commonly asked question.

[SiliconWizard]

So, I decided to implement my own library for supporting FAT32 and exFAT, with non-blocking functions. Those file systems are relatively simple.

One annoying thing, IMO, is the case-insensitive part. I'm curious what you guys' opinion is about that. I know there are a few points in favor of that from a UI's perspective, but that could have been sorted at the UI level, rather than at the file system level. (Oh yeah, I know this is one of those endless debates...)

From a software POV, up-casing ASCII only is trivial. But up-casing Unicode, as needed for FAT32 long file names, and for exFAT file names, is a nightmare. exFAT has at least added "up-case tables", that can be tailored for the supported language(s). But the downside is that such tables take a lot of memory. If you want to cache them in RAM to have fast up-casing, they take 128 KBytes of memory (the whole 16-bit unicode set). exFAT has added the provision to store the table as compressed (RLE basically), but most implementations I've seen read the table when mounting the volume and then uncompress it in memory - so you basically need 128 KBytes just to be able to up-case. I'm thinking of keeping the table compressed in memory and scanning it for up-casing, but that will be kinda inefficient.

All in all, that's a lot of mess for implementations, for something that should have been handled at the UI level IMO - if that.

Any thoughts on this up-casing mess?

[lucazader]

This suggestion comes with two caveats:
-> you are or can use cpp in the project
-> you are ok with adding locale to the project (as this can/will bloat the output binary but quite a bit)

Feel free to ignore this suggestion if either of these are a dealbreaker.

You should be able to use std::towupper to perform this conversion quite easily:
https://gcc.godbolt.org/z/3o757d6fa

Code: [Select]

std::wstring capitalize_string(std::wstring s)
{
    std::transform(s.begin(), s.end(), s.begin(),
                   [](wchar_t c){ return std::towupper(c); });
    return s;
}

[SiliconWizard]

Those would be deal-breakers indeed. But apart from the language/dependencies/and bloat considerations, this solution would still be a problem, at least with exFAT.

exFAT volumes embed their own up-case table. Implementations are supposed to use it for up-casing, and nothing else. The problem here, is that if you use another means of up-casing that happens to have a few differences, even if minor, that could lead to a wrong "up-casing" for a given exFAT volume, and make it unusable on other systems.

You can't use a "locale" per se, as the conversion table is stored in the volume, but the locale identification itself is not provided. Thus, when mounting a given exFAT volume, you have no clue what locale it uses. All you have is a conversion table. It's flexible, but it's pretty annoying to use if you have limited memory, due to the large size of the required table.

Of course, you could ignore all that, and even implement exFAT as case-sensitive, if you're exclusively using the volume on your own system. But if you want to share the volume with other systems (which is one of the points of using exFAT), that would make the volume non-conforming and would cause all sorts of issues, from a given OS not mounting it, to mounting it read-only, to mounting it read-write but not "seeing" some of the files in the volume, etc.

The exFAT spec defines a minimal up-case table (which is basically just the ASCII upcasing) that you are allowed to use for a given volume, but if you use it, you'll need to format the volume yourself. No OS I know of, that supports exFAT, will format say an SD card in exFAT with the minimal up-case table. And for the fun fact: if you try mounting an exFAT volume that has only the minimal up-case table, on Windows, it will mount it read-only.

[magic]

 

One could argue you deserve it for attempting interoperability with Microsoft

I wonder if it would be feasible to maintain a reasonably-sized cache of symbols that you have actually encountered on the given FS and fall back to scanning the data from disk on a miss. Or maybe create a map of ranges that are affected by upcasing or not - I suspect they are quite continuous and sparse.

[SiliconWizard]

One could argue you deserve it for attempting interoperability with Microsoft

exFAT may have been designed at MS, but it has become the de-facto standard for SD cards. So, it's not just a matter of interoperabiity with "Microsoft" here. As I mentioned, if I used exFAT in a purely case-sensitive way, you're right, it would NOT be usable on any Windows system, but it would likely cause problems on any other OS compliant with the exFAT spec anyway...

And, the question is more general too. This is the same for all file systems that are case-insensitive. Of course all FAT versions are, but AFAIK, all Apple file systems are too.
NTFS supports both. While it's used in the case-insensitive mode in Windows, it can be set for case-sensitivity on a per-directory basis. But I certainly don't intend on implementing NTFS.
And, of course, most Unix/Linux file systems are case-sensitive.

But just saying that, case-insensitivity has been considered much easier on the average user (so again it's purely an UI problem), but back when pretty much only ASCII was supported, this was not a problem. Once Unicode came into play, that suddenly became a nightmare.

I wonder if it would be feasible to maintain a reasonably-sized cache of symbols that you have actually encountered on the given FS and fall back to scanning the data from disk on a miss. Or maybe create a map of ranges that are affected by upcasing or not - I suspect they are quite continuous and sparse.

I've thought about some kind of cache, yes. I suppose this should be effective in practice.
As I mentioned, I've also considered storing the table in memory in a compressed form (RLE). Sure it would make upcasing inefficient, but that's typically not something that is heavily done either in a typical use case, so could be considered too. I'm thinking of a mix of the two: storing the table compressed in memory, and adding some cache as well for a few tens of characters or so.

[brucehoult]

And, the question is more general too. This is the same for all file systems that are case-insensitive. Of course all FAT versions are, but AFAIK, all Apple file systems are too.

Apple file systems are case-insensitive by default, but Apple's tools support formatting a partition with HFS+ or APFS in either case-sensitive or case-insensitive versions. I often format external/backup drives to be case-sensitive and you can do it for your boot drive to if you want. It is required for building certain *nix software, including for example gcc. The OS and Apple applications work fine on a case-sensitive drive, as of course do all command-line tools, things you port from Linux etc.

However there are some applications known not to run on case-sensitive file systems. Steam and Photoshop are two of them. Presumably they generate or hard-code the names for certain files internally in multiple places and didn't take care to make the names the same case in each place. It should be easily fixed, but as neither Mac nor Windows is case-sensitive by default they somehow haven't bothered to.

[DiTBho]

I don't like filesystems that allow filenames to have spaces or weird characters. They are a pain to be handled by scripts.

One year ago I made a filename-filter program to handle the filenames of internet downloads.

-> unicode for me is only a thing I don't want to have.

[brucehoult]

I don't like filesystems that allow filenames to have spaces or weird characters. They are a pain to be handled by scripts.

Only if you use a scripting language with variables based on text substitution and re-parsing. Even then, you're fine if you properly quote uses of variables.

In a scripting language such as Perl, Python, Ruby, or JavaScript there is no problem at all with "weird" file names.

[SiliconWizard]

Yep. use proper tools. Filenames with whitespace need to be properly either put inside quotes, or escaped (with a backslash, for instance). Not rocket science.

For the purpose of generality, while not making it too horribly complex to implement (which *always* makes the probability of having bugs much higher), case-sensitive and character-agnostic, except for the very few path separator characters, is IMO the way to go. So, basically the unix-like approach.

Now 'whitespace' can still be questioned. The problem again if you exclude whitespace, is that, if we want to support Unicode (and not doing so is absolutely impossible if you live outside of the USA or maybe UK too, or otherwise you get back to the infamous code pages, *vomits*), whitespace could come in other forms than just the 0x20 ASCII character... Now granted that most command-line tools for most OSs, and in particular shells, only see the ASCII whitespace characters as whitespace, and the rest as regular characters - well, that is, if they even support Unicode. Sometimes that's a bit iffy.

But as with the case-insensitivity matter, it's in the end all in the UI. The filenames in "clear" are ONLY for the purpose of UI. Otherwise, a filesystem could just use a GUID for identifying a file, and move on.

But case-insensitivity is the worst thing to handle IMO. There are just too many possibilities if you want to support Unicode as a whole (and if you don't, you're just ignoring a big part of the planet), and it's a big mess, while, as I said, if you mess up the upcasing, it would not just be a cosmetic issue: it would render accessing some files impossible. Awful.

So all in all, while all of this is trivial using pure ASCII, it becomes almost intractable if you want to do it 100% correctly in Unicode. It's horrible.

[DiTBho]

Only if you use a scripting language with variables based on text substitution and re-parsing. Even then, you're fine if you properly quote uses of variables.

Yes, you can properly quote, but
with Bash scripts -> it's easy to break stuff.
with Php code -> it's easy to break stuff
with Curl scripts -> it's easy to break stuff

[brucehoult]

Don't forget TCL.

(though I'd really prefer to -- for some unknown reason it's still heavily used in EDA workflows)

[SiliconWizard]

It's easy to break stuff using any language in general. And this is kinda off topic =)