Is there something to learn for embedded/IOT from the Crowdstrike disaster?

[peter-h]

Obviously nobody should deploy a "critical" system which is subject to 3rd party updates which have not been tested by the implementer on another (non critical) system.

But obviously millions of IT guys didn't know that

And then we have the endless debates about "latest patches" on IOT products. Who wants to destroy their company? Micro$oft and Crowdstrike will survive this because they have contractually excluded all liability and anyway nobody is big enough to sue them.

[jzx]

Learn to do backups 

[mikerj]

Learn to do backups 

Restoring "Backups" is not a very practical solution when hundreds or thousands of important systems have gone down due to an update and you need to get them running ASAP.

[peter-h]

Someone will point out that Samsung, Apple, etc, regularly issue updates to their phones. So how do they manage the risk? Probably by

- having a lot more testing resources than Crowdstrike
- doing a rolling deployment, starting with some small country in Africa

Indeed; restoring backups is useless. Firstly, you will lose all new data, and secondly if the system has "done a BSOD" then it needs a site visit which is a disaster in most cases.

[hans]

Don't update if you cannot handle any prolonged downtime. But also don't postpone updates forever. Vulnerabilities will be exploited, so schedule maintenance and downtime regularly.

Having a "rolling release" update model has never worked for critical infrastructure. Just like "Continuous Delivery" is not a thing in the embedded world; if devices instantly get updated with the last Git commit you make to the production branch of your repository, as you can see, it has the potential to brick all your devices in the field. A webserver or computer OS can be rescued, but whatever embedded IoT gadget you in the field probably not without shipping it back and taking more costs than a 'throwaway' product would be worth.

[mark03]

Informed users are learning software updates are frequently not in their best interest.  Consider Windows for example.  Setting it to "critical updates only" certainly does not screen out the non-security-critical bloat.  There are similar examples in the embedded space, where a vendor decides to disable some popular feature against users' preferences.  Examples in smartphone apps are legion (the BBC news app is a prominent example).

So, this is slowly corroding users' trust in their SW suppliers, which---given the conflicting motives of user and vendor---is unavoidable, but obviously not good for security and reliability.

The Crowdstrike incident is different, mostly.  By definition, what they are trying to do requires frequent (daily?) updates, so the systemic blame (besides what has been mentioned above---which I agree with) needs to fall on the OS which requires mitigations like Crowdstrike to function safely.

Despite this, to me it's another example of how a computing culture of "continuous updates == security == good" can come back to bite you.  The benefits of software updates need to be balanced against the potential harms; this is rarely if ever discussed, probably because it tends to shine a light on user-hostile policies.  Personally, I would avoid software updates on most of my embedded devices if I could, and I make embedded software for a living.

[madires]

Some customers can blame themselves for not reading the terms & conditions. Crowdstrike explicitly warns to not use their products for critical systems.

[agehall]

Yeah, don’t let all your nodes update automatically. Test any updates on a few nodes that you can live without if something bad happens and only roll out the update once the test nodes have proven they can survive a reboot.

Everyone pushes a bad update once in a while. That isn’t the big f-up imho (but it was an f-up). Blindly trusting updates was the major f-up here.

[wek]

Despite this, to me it's another example of how a computing culture of "continuous updates == security == good" can come back to bite you.  The benefits of software updates need to be balanced against the potential harms; this is rarely if ever discussed, probably because it tends to shine a light on user-hostile policies. 

Maybe; but maybe it's just that people want things to be simple. "Discussing or estimating benefit/drawback balance == complicated == bad".

Ideally, the culture should be shifted so, that the burden of proof of benefit is on whoever provides the update.

JW

[tszaboo]

Seamless rollouts, or rolling upgrades, are critical strategies in software deployment that help minimize downtime, maintain stability and enhance user experience. They involve the gradual deployment of updates or new features across servers or users, ensuring that the system remains operational during the process.

[wraper]

Don't deploy to prod on Friday.

[wraper]

Learn to do backups 

Backup is not an issue here whatsoever, no data corruption happened. The issue is about how do you boot many thousands of computers to delete the file that causes BSOD. Basically needs a physical access and manually doing one by one unless it's running on a virtual machine.

[wraper]

Micro$oft and Crowdstrike will survive this because they have contractually excluded all liability and anyway nobody is big enough to sue them.

MS is not liable whatsoever. They have nothing to do with causing the problem. However MS stepped in and released a tool for simplifying the fix.

[langwadt]

Learn to do backups 

Backup is not an issue here whatsoever, no data corruption happened. The issue is about how do you boot many thousands of computers to delete the file that causes BSOD. Basically needs a physical access and manually doing one by one unless it's running on a virtual machine.

and if you are using bitlocker, which is default now I believe, you need the key you can't just boot in safemode 

[peter-h]

the culture should be shifted so, that the burden of proof of benefit is on whoever provides the update.

It is probably not possible to do that, in the contractual sense. Microsoft dominated that market (windows servers and the applications, basically) and you had to use that Crowdstrike stuff. The smart thing, which probably few did, was to test the updates first. The fact that some or many did a BSOD suggests that the simplest test setup would have shown the issue

In embedded/IOT the vendor may sometimes be dominant but often the customer will be dominant and thus be able to sink your company if you push out a damaging update. Customers in particular do that because of internal company politics: a corporate ladder climber needs to show ruthlessness, and of the three options on who to scalp (customer, employee, supplier) the last one is the only safe option. You can screw an employee but only if it is a heterosexual white male Especially if the failed update requires on-site action.

This business should be an eye-opener for IOT vendors. But it's not easy to explain to a customer that you are not doing updates because of liability

[langwadt]

Micro$oft and Crowdstrike will survive this because they have contractually excluded all liability and anyway nobody is big enough to sue them.

MS is not liable whatsoever. They have nothing to do with causing the problem. However MS stepped in and released a tool for simplifying the fix.

afaiu the way crowdstrike gets around having to get every update signed by MS (it's a kernel driver to run in ring0) is that the crowdstrike driver is basically an interpreter that can execute code in kernel mode delivered separately as update files.

MS allowing that to me sound like it basically makes the whole idea of drivers being wetted and signed, pointless

 

[SiliconWizard]

Micro$oft and Crowdstrike will survive this because they have contractually excluded all liability and anyway nobody is big enough to sue them.

MS is not liable whatsoever. They have nothing to do with causing the problem. However MS stepped in and released a tool for simplifying the fix.

afaiu the way crowdstrike gets around having to get every update signed by MS (it's a kernel driver to run in ring0) is that the crowdstrike driver is basically an interpreter that can execute code in kernel mode delivered separately as update files.

MS allowing that to me sound like it basically makes the whole idea of drivers being wetted and signed, pointless

Yes, this is what Dave Plummer explains in his video.
This is absolutely atrocious.
We can all see how it would make CrowdStrike's life infinitely easier, but that's completely insane in terms of security. Compromising security for more security being a schizophrenic approach.

Besides, no OS should require a third-party kernel driver for adding security to the OS - that's bogus.
Everything is wrong in that story. The fact that it blew up in everyone's face is, IMHO, actually a very good thing. If it could be a lesson. Which, unfortunately...

[peter-h]

Wonderful indeed