I tried to think of examples illustrating why the cultural reasons that lead to CrowdStrike disaster are not specific to any software product, but are ubiquitous. (I do not know
why that is; I am only trying to point out the bigger picture here.)
There are some that come to my mind, all on the web server side, but they do apply to embedded/IoT development, integration, and deployment too.
The first is server configuration in Linux distributions. How many of those who maintain web servers (at the Apache/Nginx configuration level) use a different active server configuration directory from the default one? Very, very few. This means that whenever software updates are applied, any web server related configuration will be automatically applied and take effect, too. While most distributions are quite good at using "safe" default configuration files, installing the wrong package may enable a web-based service (specialized Apache/Nginx module) that compromises the security on the machine due to cross-service interactions.
(I always use a dedicated directory, with a separate configuration information. For about a decade, I even published one designed for group-based maintainer access (when you have may webmasters with different levels of access to a mixed tree) at my previous home page. This way, I can allow automatic security patches to the server, but monitor the distribution-provided configuration directory –– which does not affect the running service at all –– for important changes that may need to be ported to the server. I did this on a number of rather public servers, with very good results (both security and support to webmasters); and not because they were small and quiet: they were constantly bombarded with exploit access patterns based on the log files! Yet, to introduce the pattern to new sysadmins/sysmasters/web root admins, I always need to explain the whole story with examples before they realize the small inconvenience
they imagine this causes, actually improves their work product for everyone, and typically reduces the time taken for maintenance in the long run. This should be the obvious
default pattern, not something that has to be pushed through heavy resistance!)
The second is self-modifying web scripts –– "required" only so that one can update the web framework by clicking on a button on a web page ––, and the third is embedding credentials in the firmware.
Allowing web service executables to rewrite themselves or create new executables is the reason why script drops, trojans, and worms are still plagueing web frameworks. If they could not modify themselves or create new executables, almost all exploits would immediately stop working, with entire approaches becoming impossible to exploit.
Embedding credentials in the firmware, even if in salted password form, allows anyone excavating those credentials to pose as that device. It makes it possible to create infiltration and exploitation devices that pose as the original device, but produce attacker-controlled output. In IoT devices communicating with upstream servers, it allows the attackers to communicate with those upstream servers without the upstream servers being able to detect it. This applies to both custom protocols and common protocols like http and smtp. A lot of spam is injected to email servers via exploited trusted clients, for example.
Now, I'm not here to argue whether the above is correct or not (especially with zilp), because I've observed this in practice for a long time; decades. I described them only as examples, to illustrate the analogous human behaviour to that underlying the Crowdstrike disaster, so give some slack.
The point I am trying to make here is that doing things the above way is what most sysadmins, webmasters, systems integrators, developers, et cetera nowadays
learn to do by default, because it aligns with the cultural approaches we have. We don't build buildings to last, so why would one expect us to build software to last either? It is
not cost-effective to do so. When we teach software development, error checking is at best an afterthought, normally omitted to "save time".
You have to train people to think along better lines, in better patterns, or they will repeat the mistakes others have repeated before, learning nothing from history. (And, they will typically complain all through training how much a waste of time this is, because <the example exploits> are so rare in practice, they're not worth worrying about. Just like you see when you write code that say checks the return value of
close(); because "it cannot fail for normal files on typical desktop file systems".)
Of course, exactly what those better patterns are, is a separate discussion; it is more about "how to avoid Crowdstrike disaster situations" than "what to learn from the Crowdstrike disaster").