Is there something to learn for embedded/IOT from the Crowdstrike disaster?

[SteveThackery]

Let's do a car analogy (...) Your suggested solution: Have car manufacturers put a lock on the engine hood that only they have the key to....

And there we have it: your analogy falls at the first line because Windows is exactly like a car with the hood bolted down.

You keep going on about how wrong that is, and how anyone should be able to get under the hood, modify things, even deliberately or accidentally break things. You claim that this level of access does not affect the integrity of the car. You might be right, you might be wrong, I really don't care!  But the crucial point is that Windows is not like that: Windows is a car with the hood bolted down.

Microsoft has the keys to the hood and can do anything they want to it. They also lend the keys to carefully chosen third parties, but they always test what those third parties have done and only when everything seems OK do they let the third party install their new component to the engine.  I believe this is almost identical to the way Apple works with MacOS.

I am not trying to convince you that this is a good model! I am trying to explain that this is how Windows works. You or I don't have to agree with it. But it is what it is.

If only you can open your mind to the fact that Windows is just different (not necessarily better, not worse, but very different) from Linux, I am certain that you will see the validity of my arguments.

Yes, my arguments are completely wrong and largely irrelevant to a Linux owner/admin like yourself. You have pointed that out countless times and I am obliged to take your word for it because you are a Linux expert.  But we are talking about Windows: a car with the hood bolted down that only Microsoft can open, and anyone who wants access needs Microsoft's agreement.

Yes, I know how difficult this is for you to stomach - you've made that oh, so clear! But try to put aside how you think it should be, and please at least try to embrace how it actually is. Nobody gets the keys to the hood without Microsoft's agreement, and nobody gets to install anything under the hood without Microsoft (or Apple) approving it.

At risk of putting too much reliance on an imperfect analogy, Microsoft has kitted out the cockpit with a wide array of controls (APIs) which they have designed and built, and which are the only way the user* can interact with whatever is under the hood.

* (and, breaking out of the analogy, applications)
--------

I think any reasonable person would answer my questions in the following way:

1/ Do you agree that most Windows users have a limited understanding of what Windows is and what it does? And that most of them have never written a line of code? And most of them aren't interested in the workings of Windows, they just want to get their work done, their emails sent, their Facebook page updated?  YES

2/ Do you agree that a BSOD is a pretty serious inconvenience for these users? That when a BSOD happens, most of these users would have no idea how to fix it? Even for system admins, it's a major ballache?  YES

3/ And in addition, do you agree that BSODs, especially widespread ones like CrowdStrike, reflect very badly on Microsoft (whether fairly or not)?  YES

4/ Not a question, a statement: Windows is a car with the hood bolted down, and only Microsoft and its trusted partners can unlock it and make changes.

My arguments are based on this clear, four-part foundation, which describes what Windows is, and how it gets used. It may seem a profoundly wrong model - objectionable, even - to @zilp, but that is the world of Windows.

------

Having laid that foundation, later on today I am going to review my earlier arguments and demonstrate how they are not "irrelevant" and "non sequiteurs" in the world of Windows. We need to stop talking about Linux now, because we are not talking about how Windows should be, but how it is.

[zilp]

.... the relevant points to this discussion are that Free Software comes with all the source code and a license that allows you to read, modify, and redistribute it all, which obviously flies in the face of any idea of "the vendor" preventing you from running kernel mode drivers, because it obviously is impossible for "the vendor" to prevent you from doing so if you can just edit and recompile the kernel, and if anyone is allowed to redistribute modified versions, and the fact that, obviously, people running Linux, FreeBSD, or OpenBSD servers, do not consider their systems to be insecure, even though nothing prevents them from running any kernel-mode code whatsoever if they wanted to.

Right! And this is absolutely nothing like Windows works!!

Yeah. And this is just missing the point.

You made some point above about how, supposedly, the security of operating systems is universally viewed.

This is the demonstration that that is not the case.

The view that you presented is particular to your corner of the world, and not universal.

That this is absolutely nothing like Windows works is exactly the point that I was making. And that therefore, what you consider universal, is in fact not universal at all, but particular to Windows (and other proprietary systems), because there exists a whole world outside of that that views things very differently. Therefore, your appeal to your view as a universally accepted truth does not work, this supposedly universally accepted truth is nothing of the kind.

This, I am certain, is why you can't see my point of view - we are coming from such different places that we have almost no common ground to stand on for our discussion.

The only common ground that is needed for most of it is simply logic.

With Linux, every app and the entire operating system is wide open and transparent, and literally anyone can do literally anything they like to any part of it. And obviously these individuals take full responsibility for their own actions.

The latter is not particular to open systems. If you decide to install proprietary software, you still are fully responsible for that decision.

The people you describe can write and run any kernel mode software they like, in the full knowledge that their code can crash the OS if they make a mistake, or even deliberately crash it. So most of the time these owner/admins write the best quality code they can, and that means of course they don't consider their systems to be insecure! They trust their own code!

But I repeat - nothing whatsoever in the paragraph above applies to Windows.

None of that addresses why it would be in the interest of Microsoft's customers to be prevented from buying certain kernel drivers from other vendors.

You really should reflect on why it is so important to you that people who want to buy that driver from some other vendor should be prevented from doing so. You seem to be very focused on how many people might not want to bother with other vendors, how they are happy to buy everything from Microsoft, how they think that they get the best product if they buy everything from Microsoft ... but none of that is relevant for justifying why people who feel differently should be forced to also buy from Microsoft, which is what you are proposing. People who want to buy from Microsoft do not need to be forced, and for people who don't want to buy from Microsoft, those arguments don't apply.

[zilp]

Let's do a car analogy (...) Your suggested solution: Have car manufacturers put a lock on the engine hood that only they have the key to....

And there we have it: your analogy falls at the first line because Windows is exactly like a car with the hood bolted down.

You keep going on about how wrong that is, and how anyone should be able to get under the hood, modify things, even deliberately or accidentally break things. You claim that this level of access does not affect the integrity of the car. You might be right, you might be wrong, I really don't care!  But the crucial point is that Windows is not like that: Windows is a car with the hood bolted down.

But it isn't. The whole premise of your contribution to this thread is that is isn't.

The competition authorities in the EU have forced Microsoft to remove some of the locks/bolts/whatever.

Your position is that this is bad and Microsoft should be allowed to bolt it all shut.

And I am asking you to justify why they should be allowed to do that.

Microsoft has the keys to the hood and can do anything they want to it.

No, they can't. And the anti-trust laws are only one of many that limit what they can do with it. Or at least, what they can legally do with it.

They also lend the keys to carefully chosen third parties, but they always test what those third parties have done and only when everything seems OK do they let the third party install their new component to the engine.

All of this is also subject to anti-trust laws. They are not allowed to freely decide on the criteria.

I believe this is almost identical to the way Apple works with MacOS.

I am not trying to convince you that this is a good model! I am trying to explain that this is how Windows works. You or I don't have to agree with it. But it is what it is.

I mean, for one, I have already established that that is in fact not how it works.

But also, the whole point of you writing in this thread was to argue about whether this model should be allowed, so it's kinda dishonest to now claim that you only wanted to describe how it works.

4/ Not a question, a statement: Windows is a car with the hood bolted down, and only Microsoft and its trusted partners can unlock it and make changes.

It just happens to be a false statement, because, by law, Microsoft is required to give certain parties access, whether they trust them or not. And you are proposing that that should be changed. And failing to justify this proposal.

My arguments are based on this clear, four-part foundation, which describes what Windows is, and how it gets used. It may seem a profoundly wrong model - objectionable, even - to @zilp, but that is the world of Windows.

Except it evidently isn't. If it were as you claim, then noone would have ever had the Crowdstrike driver installed, presumably, because Microoft wouldn't have allowed it. The fact that they did shows that that is not how Windows in fact does get used, because people do in fact voluntarily install software that Microsoft would not approve of. And it even demonstrates that at least a significant proportion of the people using Windows don't want the model that you are proposing either, because noone forced them to install Crowdstrike, and they did it anyway. So it is obviously absurd to claim that those people who completely voluntarily installed Crowdstrike would have preferred to not install it. If they had preferred to not install Crowdstrike, they would not have installed Crowdstrike.

Having laid that foundation, later on today I am going to review my earlier arguments and demonstrate how they are not "irrelevant" and "non sequiteurs" in the world of Windows. We need to stop talking about Linux now, because we are not talking about how Windows should be, but how it is.

You haven't laid any foundations.

[SteveThackery]

Quote from: zilp on Today at 12:45:44 pm

You really should reflect on why it is so important to you that people who want to buy that driver from some other vendor should be prevented from doing so.

You're doing it again and I DETEST being misquoted, having words put in my mouth, being deliberately misrepresented in order to win an argument.

I NEVER SAID that it is important to me that people who want to buy a driver from someone else should be prevented from doing so!

I explained that this is the way it works for Windows: kernel mode drivers can only be installed if they are from Microsoft or one of Microsoft's trusted partners (and they are tested and approved by Microsoft).

These are FACTS. At no time did I ever say that policy is "important to me". It isn't!  I was trying to explain why it's important to Microsoft!!

Time and time again you have misrepresented my position in order to undermine my argument. Your repeated refusal to even try to engage with my arguments, and now another blatant lie about what I said, is just too much. I'm out of here, and @zilp: if you are going to keep lying about what I said you can piss off.

[Siwastaja]

zilp, why are you wasting any time with this guy who produces a lot of unnecessary, incoherent, illogical and uninteresting wall-of-text?

[SteveThackery]

zilp, why are you wasting any time with this guy who produces a lot of unnecessary, incoherent, illogical and uninteresting wall-of-text?

@zilp has wiped the floor with me regarding word count!

[Siwastaja]

zilp, why are you wasting any time with this guy who produces a lot of unnecessary, incoherent, illogical and uninteresting wall-of-text?

@zilp has wiped the floor with me regarding word count!

Hence my remark about wasting time.

[Marco]

I explained that this is the way it works for Windows: kernel mode drivers can only be installed if they are from Microsoft or one of Microsoft's trusted partners (and they are tested and approved by Microsoft).

No, you are wrong.

https://learn.microsoft.com/en-us/windows-hardware/drivers/install/kernel-mode-code-signing-requirements--windows-vista-and-later-

Drivers signed through attestation are not tested by Microsoft.

[SteveThackery]

Thanks, Marco, this is the first useful contribution in pages. Much appreciated.

[SteveThackery]

"Attestation signed drivers can't be published to Windows Update for retail audiences. To publish a driver to Windows Update for retail audiences, you must submit your driver through the Windows Hardware Compatibility Program (WHCP). Publishing attestation signed drivers to Windows Update for testing purposes is supported by selecting CoDev or Test Registry Key / Surface SSRK options.

"When a driver receives attestation signing, it's not Windows Certified. An attestation signature from Microsoft indicates that the driver is trusted by Windows. But because the driver hasn't been tested in HLK Studio, there are no assurances made around compatibility, functionality, and so on. A driver that receives attestation signing can't be published to retail audiences through Windows Update. If you wish to publish your driver to retail audiences, you must submit your driver through the Windows Hardware Compatibility Program (WHCP)."

[SteveThackery]

zilp, why are you wasting any time with this guy who produces a lot of unnecessary, incoherent, illogical and uninteresting wall-of-text?

Says the guy who hasn't made a single useful contribution and probably doesn't understand the issues anyway. Come back when you're a grownup.

[Marco]

Attestation signed drivers can't be published to Windows Update for retail audiences.

Guess how most GPU drivers get released ...

[SteveThackery]

Attestation signed drivers can't be published to Windows Update for retail audiences.

Guess how most GPU drivers get released ...

Take it up with Microsoft. That's who I'm quoting.

[IanB]

Take it up with Microsoft. That's who I'm quoting.

Take what up with Microsoft?

If Microsoft is going to deliver updates to millions and billions of consumer machines around the world through their automatic update mechanism, you can be damned sure they are going to apply some quality control mechanism to the updates they ship.

"Hey, Microsoft, here's a bit of random code I wrote. I want you to install it on everyone's computer around the world through Microsoft Update. It's perfectly safe, honestly. Trust me, OK?"

I mean, what exactly is there to take up with Microsoft about this?

[SteveThackery]

Take what up with Microsoft?

I was talking to Marco. I believe he was taking a pop at Microsoft over their process for releasing GPU drivers.

FWIW, I entirely agree with your sentiment about Microsoft's interest in ensuring the quality of third party drivers.

[Marco]

Take it up with Microsoft. That's who I'm quoting.

Most GPU drivers get released through the manufacturer and their own update mechanism. Untested by Microsoft.

[radiolistener]

Windows is a car with the hood bolted down.

But it isn't. The whole premise of your contribution to this thread is that is isn't.

The competition authorities in the EU have forced Microsoft to remove some of the locks/bolts/whatever.

I think that "a car with the hood bolted down" analogy is not clear.

More close analogy will be "Windows is a rented car". You can use it to move, but obligated to pay for that and take care about - don't modify and use it according to the rules developed by their owner. You don't have right to dictate how the company (that rented you the car) should modify their car and how they needs to do their business.

If you want to do something that you want, doing it on a rented car is a wrong way. Just because you signed an agreement with the company  that you will use their car in such way as they want. If you want to do with car what you want, you need to use another car, it can be your own car bought in a car shop, but not rented car...

A company that rented you the car can install tracking device on the car and can spy where is the car and what you're doing with it. This is required for security reason...

The same with OS. If you want to use OS as you want, you need to use Linux, but not Windows. The same MS can track and spy where is their Windows and what you're doing with it. This is required for security reason... 

They don't lie you that these updates for security reason, just hide that this is not for your security, but for their security... 

[SiliconWizard]

Take it up with Microsoft. That's who I'm quoting.

Most GPU drivers get released through the manufacturer and their own update mechanism. Untested by Microsoft.

There are drivers for main GPUs directly from MS, but they are rarely "up to date". So people usually update them directly from the vendor (either manually or automatically if they have installed accompanying software).
But that doesn't mean these drivers are "untested" by MS. Actually, for main vendors, GPU drivers are WHQL (so "tested" by MS), except beta versions (which nobody forces you to install).

Among main vendors (AMD, NVidia, Intel), I don't know any of them which releases drivers that are not WHQL (again unless beta or explicitely marked as such and it usually takes some explicit steps to install these).

[mfro]

... Actually, for main vendors, GPU drivers are WHQL (so "tested" by MS) ...

My understanding is that WHQL does not mean "tested by Microsoft" at all, but rather "tested by vendor according to MS' guidelines" and "vendor's test protocol checked by MS".

[zilp]

zilp, why are you wasting any time with this guy who produces a lot of unnecessary, incoherent, illogical and uninteresting wall-of-text?

Says the guy who hasn't made a single useful contribution and probably doesn't understand the issues anyway. Come back when you're a grownup.

That's an ad hominem, by the way.

[SteveThackery]

My understanding is that WHQL does not mean "tested by Microsoft" at all, but rather "tested by vendor according to MS' guidelines" and "vendor's test protocol checked by MS".

That sounds right. Happy to be corrected.

[SteveThackery]

That's an ad hominem, by the way.

You don't say.

[zilp]

Windows is a car with the hood bolted down.

But it isn't. The whole premise of your contribution to this thread is that is isn't.

The competition authorities in the EU have forced Microsoft to remove some of the locks/bolts/whatever.

I think that "a car with the hood bolted down" analogy is not clear.

More close analogy will be "Windows is a rented car". You can use it to move, but obligated to pay for that and take care about - don't modify and use it according to the rules developed by their owner. You don't have right to dictate how the company (that rented you the car) should modify their car and how they needs to do their business.

At least as far as the EU is concerned, that is not correct. That might well be what Microsoft whishes were the case, but it just isn't.

And even with a rental car, the rental company can not make completely arbitrary rules as to how to use the car. They have a legitimate interest that they get their car back in the same state (modulo wear) that they handed it to you, but that's obviously not a thing that applies to software installed on your own computer.

If you want to do something that you want, doing it on a rented car is a wrong way. Just because you signed an agreement with the company  that you will use their car in such way as they want. If you want to do with car what you want, you need to use another car, it can be your own car bought in a car shop, but not rented car...

A company that rented you the car can install tracking device on the car and can spy where is the car and what you're doing with it. This is required for security reason...

That would also be illegal in the EU (because of data protection laws).

The same with OS. If you want to use OS as you want, you need to use Linux, but not Windows. The same MS can track and spy where is their Windows and what you're doing with it. This is required for security reason... 

That is also illegal for the same reason. (Though enforcement is lacking a bit ...)

[radiolistener]

My understanding is that WHQL does not mean "tested by Microsoft" at all, but rather "tested by vendor according to MS' guidelines" and "vendor's test protocol checked by MS".

Initially WHQL means that driver manufacturer paid certification fee to MS, 250 USD per operating system family 

Now WHQL just shows that someone who signed the code paying at least 200 USD per year for code signing certificate. That sign means that only this person, government security services or very professional hackers can modify driver and apply signature again. 

[SteveThackery]

1/ BSODs damage Microsoft's reputation; Microsoft is wise to try to prevent them.

2/ To the best of my knowledge, the architecture of Windows prevents user-space software from causing BSODs.  Code running in kernel mode can cause BSODs, as was demonstrated by the kernel mode driver from CrowdStrike.

3/ For some security functions, code must run in kernel mode in order to do it's job.  This code - regardless of who wrote it - has the ability to to BSOD Windows, so it should be high quality and as free of bugs as possible.

I don't know how many vendors of security software need to use the same gateway to kernel space as CrowdStrike - 50? 100? 10? Let's say 10, just for the sake of argument.  I expect it is more.

4/ At the moment, all ten of those vendors need to write their own driver with privilege escalation from user space to kernel space. If Windows is to remain free of BSODs, all ten of those drivers must be free of bugs, as far as possible.  That's a lot of code, and a lot of debugging.

5/ But there is another way. If Microsoft wrote their own privilege escalation driver, and forced all the security vendors to use that instead, then only one driver needs to be debugged and polished to near perfection.

6/ So on one hand we have one driver that could BSOD Windows, on the other hand we have ten drivers that could BSOD Windows. Having ten BSOD-capable drivers in the ecosystem (all doing much the same thing, remember) is obviously more risky than having one BSOD-capable driver.

7/ Therefore it is reasonable and logical for Microsoft to write that driver, and force our ten security vendors to use it, rather than writing, debugging and using their own. It does not guarantee freedom from BSODs, but it greatly reduces the risk.

This is not remotely controversial. Windows is already full of APIs that handle privilege escalation, and which programmers must use.  Keeping third party software out of kernel space is the rule, rather than the exception.

8/ Microsoft has invested billions in Windows: it is their crown jewels. Nobody has as much interest in Windows as Microsoft. They cannot control what third party applications do, but they absolutely can control Windows itself. Making Windows as impregnable as possible is obviously in their interest. And one way to do that is to keep as much third party code out of kernel space as possible. Microsoft has limited control over the quality of third party software (basically just a testing regime) but it has (almost) complete control over Windows. It can put as much effort into driving up the quality, stability and security as it chooses.

That is why, from Microsoft's point of view, forcing as much third party code as possible to run solely in user mode is an obvious step to take.

------

I am not supporting this model personally. I am simply trying to explain what Microsoft's interests are, and to explain one logical approach they can take to protecting those interests.

------

Now, our friend @zilp, if he has read this far, is at this very moment red-faced and the veins on his forehead are bulging. Even as we speak he is furiously typing a very lengthy essay rubbishing every single sentence in my post. That's fine, we are all free to do that. I just wanted to take this opportunity to tell you what I think. @zilp's rubbishing won't change that. And if you want to be hostile, then please - go ahead. You will be ignored, though.

On the other hand, polite and respectful disagreements are welcome, because such things are how we all learn and gain new insights.