University of Minnesota Linux code security issues; banned and to be removed

[mikerj]

What I am learning from this is that we're in the hands of asshats.

Welcome to the modern world.

[magic]

Lol, wasn't Greg KH one of the supporters of the infamous C.O.C.K. aka the Kernel Code Of Conduct? Something about a welcoming environment for everyone, regardless of education and level of experience (haha), freedom from insults and harassment, blah blah? Well, well, well...

[bd139]

Yeah that’s the one... 

Children the lot of them. Then again on the commercial software market it’s no better.

[Ed.Kloonk]

Funny how it is sometimes implied that GKH could have, within his rights, rejected the K.C.o.C.

[ataradov]

Yeah, the top developers were bullied into accepting the CoC. It is always a sad day when a project is forced to implement CoC.

But the spirit of CoC is not to be dicks to each other, and knowingly submitting malicious code pretty much falls into being a dick.

[magic]

They wouldn't have fired everybody if everybody rejected it. Besides, I'm not just talking tolerating it, but actually walking around and telling everyone it's a good idea. Linus wasn't doing the latter if memory serves me, Greg was

But the spirit of CoC is not to be dicks to each other, and knowingly submitting malicious code pretty much falls into being a dick.

That's one student who did it.
And what can we say about all the developers who publicly accused other students of doing the same, instead of admitting that they are incompetent or lazy fools who failed to notice problems with patches for potential security vulnerabilities submitted by a bunch of kids?


Reminder: there is no obligation to be competent when you submit stuff. It's on the team to handle everything you throw at them. They cannot ban you for lack of experience, kernel development background or education, they cannot ban you for mental disability even.

[ataradov]

And what can we say about all the developers

Nothing. When acting on behalf of an organization, the whole organization is suspect. They did not accuse anyone, they said that they don't know what to think and how many of those patches are intentionally buggy. Bug given the overall low value of the patches, it is easier to remove all of them.

Reminder: there is no obligation to be competent when you submit stuff. It's on the team to handle everything you throw at them.

Yes, but there is moral obligation to submit patches in good faith.

Again, bugs get through regularly. This is a known thing. That is unavoidable part of the development process.

But submitting the bugs intentionally is a good reason to be banned from ever submitting anything ever again.

Some people seem to think that is it possible to detect all errors, manually or automatically.  It is not possible in practice. So everyone just has to do their best effort to catch whatever they can. And the reset would be found and debugged later on.

[magic]

It is obvious that we could continue this back and forth about "trust in contributors" vs "security of the process" indefinitely, so I will just stop at the following.

They did not accuse anyone, they said that they don't know what to think and how many of those patches are intentionally buggy.

I will now have to ban all future contributions from
your University and rip out your previous contributions, as they were
obviously submitted in bad-faith with the intent to cause problems.

Bug given the overall low value of the patches, it is easier to remove all of them.

Based on my quasi-random sampling of the mega-review thread, most reverted patches were found to be alright. Of course my random clicking on random message titles may have been biased by whatever factors and things may have changed since two days ago, ICBA to do a proper analysis.

[madires]

Bug given the overall low value of the patches, it is easier to remove all of them.

Based on my quasi-random sampling of the mega-review thread, most reverted patches were found to be alright. Of course my random clicking on random message titles may have been biased by whatever factors and things may have changed since two days ago, ICBA to do a proper analysis.

The removal of old patches is a security precaution because the re-review of them will take some time. With version control tools you can remove the patches quickly to return to an "untainted" state. My guess is that the good patches will be added again after kernel experts verified them.

From my understanding the bad patches didn't add any security issues and were just some nonsense, which is annoying enough to be angry. Additionally to breaking the trust in the university's contributions they also indicate a lack of due diligence of the kernel maintainers regarding security and QA. That process needs to be improved which won't be easy.

[ataradov]

Well, it looks like the article was withdrawn and will not be presented. And the actual (non-malicious) patches would remain in place after additional review.

Although I personally would place a ban on any future contributions, at least for a few years, provided that there is no other way to hold them accountable.

[Ed.Kloonk]

I suspected the K devs would back down. Oh well. Too bad.

[tunk]

Letter requesting withdrawal (April 26, 2021):
https://www-users.cs.umn.edu/~kjlu/papers/withdrawal-letter.pdf
There's more info on his page, including a letter to the Linux foundation from the department head:
https://drive.google.com/file/d/1z3Nm2bfR4tH1nOGBpuOmLyoJVEiO9cUq/view

Looks like the he has been removed from the program committee of IEEE S&P 2022:
https://www.ieee-security.org/TC/SP2022/cfpapers.html
But this has not been updated on his home page:
https://www-users.cs.umn.edu/~kjlu/

Code: [Select]

News
[03/03/2021] I will be on the program committee of IEEE S&P 2022.
[02/12/2021] Our work on detecting unsafe DMA accesses was accepted to USENIX Security'21. Unchecked and inconsistent DMA accesses are very common in drivers; we found about 300 such bugs in Linux drivers.
....


[ataradov]

There is nothing to back down from. Those guys are lucky they did not get into more trouble than this.

You don't go to the store and start testing their security by shoplifting stuff. The laws exist against what they did. It is all fun and games until someone goes to jail.

[bd139]

There's actually no legal consequence here because quite frankly it's not illegal to do this. 18USC1030(a)(5) and (b) as previously mentioned does not apply because the authorization of the code is implicit in the patch acceptance process and they haven't accessed any protected computers. If they introduced a law to cover this, we'd probably have to ban pull requests on github and it would open everyone up to being sued in the sector.

The whole point of my previous argument is that the kernel team's process is completely flawed and that's where the responsibility lies and they need to fess up to that rather than shitting on the researchers.

The researchers did the equivalent of driving a tank through a hole in the fence at Area 51, donning chicken suits and dancing around making clucking noises. Everyone is complaining at them rather than wondering why there is a massive hole in the fence.

The hole in the fence scares the shit out of me more than the finger pointing game.

Whether or not that is a reasonable position to be in from a software sourcing perspective i.e. "should we trust the Linux kernel" is a massive can of compliance and security worms this entire thing has opened which is now on my plate as well.

"perhaps we should use zOS instead of Linux" is looking like a good option on paper...

[ataradov]

The whole point of my previous argument is that the kernel team's process is completely flawed and that's where the responsibility lies and they need to fess up to that rather than shitting on the researchers.

Their model matches standard industry practices. You can do this with any other project, open, or closed.

Sure, it would be better if that was not possible. It would be better if bugs did not exist at all. But we live in a reality-based world.

The researchers did the equivalent of driving a tank through a hole in the fence at Area 51, donning chicken suits and dancing around making clucking noises. Everyone is complaining at them rather than wondering why there is a massive hole in the fence.

This is criminal trespass, at least in the US. This is very much illegal whether there is a hole in the fence or a simple "NO TRESPASSING" sign and no fence at all.

[bd139]

It's not. Our legal counsel in the US has actually checked.

Also it'd be Computer Trespass under CFAA, not Criminal Trespass as that applies to physical properly only.

Realistically it's more a parallel to the point of if someone gave you a car and you accepted it contractually with no warranty, then parked it in your garage and it burned your house down then it's the house owner's mess to deal with not the car's original owner mess (really it's the insurers problem - another side issue to consider!).  If they introduced a statutory law to prove causality then there would be no second hand goods because the entire ownership chain of the goods would be responsible for future events. Also every software bug would end in a lawsuit.

YMMV but "the law" doesn't cover this as it stands at least in the US and UK.

The only tenable issue from this is the chain of trust was proven to be chock full of holes.

[DrG]

Those guys are lucky they did not get into more trouble than this.

On the one quoted point, I'm not sure that you can evaluate the trouble they got into since it is not yet complete. Even at this point, how do you think that this "issue" looks to potential employers?

[bd139]

I'd hire them in a snap. They'd be good for an adversarial security programme.

[DrG]

Letter requesting withdrawal (April 26, 2021):
https://www-users.cs.umn.edu/~kjlu/papers/withdrawal-letter.pdf
There's more info on his page, including a letter to the Linux foundation from the department head:
https://drive.google.com/file/d/1z3Nm2bfR4tH1nOGBpuOmLyoJVEiO9cUq/view

Looks like the he has been removed from the program committee of IEEE S&P 2022:
https://www.ieee-security.org/TC/SP2022/cfpapers.html
But this has not been updated on his home page:
https://www-users.cs.umn.edu/~kjlu/

Code: [Select]

News
[03/03/2021] I will be on the program committee of IEEE S&P 2022.
[02/12/2021] Our work on detecting unsafe DMA accesses was accepted to USENIX Security'21. Unchecked and inconsistent DMA accesses are very common in drivers; we found about 300 such bugs in Linux drivers.
....


Thanks for these cites. The University letter is the most interesting to me. A couple of things that I see:

The Linux folks issued some requests (demands, whatever you want to call them) and that letter is a response.

The signatures are Dept head and Deputy Dept. Head - no Deans or higher ups are involved on paper. IOW, they are keeping it at as low a level as possible, which is reasonable in my view.

The University did not withdraw the paper. The authors withdrew the paper. This is not to say that some pointed discussions may have taken place. If the authors refused to withdraw the paper, my view is that the University would have done so.

The IRB is, in a sense, off the hook. This is what I thought would happen. These folks have roles and charters and in the US, these are pretty close to being standardized and there is CFR around them. Interestingly, this situation may changes some things, at least potentially. These are complicated issues (see https://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/index.html).

The issue of ethics in research, however is much larger.

In my opinion (and only my opinion), is that the Univ seems to be saying that they are going to modfy their procedures to include/expand training and discussion with regard to research ethics. Some will view this as simple CYA and cosmetic attention. Maybe yes, maybe no. Their response in 3 and 4 does not seem, to me, to be weasel words....although I might change my mind at some point.

Does this satisfy doing X Y and Z as I mentioned earlier? Yeah, probably.

[DrG]

I'd hire them in a snap. They'd be good for an adversarial security programme.

Make them an offer, they might be interested.

[magic]

Their model matches standard industry practices. You can do this with any other project, open, or closed.

You are an embedded developer, we get it. You guys are infamous for your level of security

Not everybody enjoys the same luxury, though. Some software needs to has less bugs than industry average and has to face severely adversarial environment, such as just about any network or a multi-user machine for example. I was under impression that OS kernels do belong to this group.

I disagree about every other project, open or closed. For starters, you wouldn't even be able to submit patches like that to Windows. And even if you did, say by including them with an email to security@microsoft.com warning about a 0-day you just found, something makes me feel that they would review your fix quite thoroughly before shipping it.

And remember, Linux is the guys who used to laugh at Microsoft 10 years ago.
It's simply pathetic, there is no excuse.

edit
It's doubly pathetic because I still have seen no evidence nor admission from the submitters that the patches which actually made it to stable kernels were malicious. Remember, the deliberately malicious patches have been "outed" by the submitters themselves as soon as they received approval on the mailing list.

[ataradov]

I was under impression that OS kernels do belong to this group.

That is up to kernel authors to decide how to run their project. If you think their code quality is not sufficient - don't use it.

For starters, you wouldn't even be able to submit patches like that to Windows.

Do you really think that out of 1000's of programmers at Microsoft nobody left a juicy hole for later use? Are you sure that some of the huge holes that we are observing weekly are not introduced on purpose?

And even if we say that there are no intentional bugs. Why is Microsoft so bad at developing that their stuff gets hacked all the time? May be we need UoM to write an article how Microsoft needs to do better code reviews before accepting new code? I bet improved CoC would go a long way.

Is there any indication that Linux has more bugs than Windows as a result of their "poor" development process?

[magic]

I think they understand that a threat exists and must have review in place, or things would be much worse like they used to be in the past. And Windows is a much more juicy target too.

And if you want to compare Windows vs Linux, don't forget to include all bugs in GNOME and systemd

[ataradov]

Linux also has a review in place. Reviews do not catch everything. As I said, this is pretty standard in any practical industry. You can bog everything down in reviews that you will never do anything. And yes, the most secure software is the one that does not exist.

And which is juicier is a question for server environment. 

[bd139]

I think they understand that a threat exists and must have review in place, or things would be much worse like they used to be in the past. And Windows is a much more juicy target too.

And if you want to compare Windows vs Linux, don't forget to include all bugs in GNOME and systemd

Windows being a juicer target is the big deal. If one day it isn’t then it’s game over. There are numerous really bad privsep problems in Linux. A fine example is asking yourself what a compromised Firefox process can access. The answer is anything in your home directory. People make a big deal about that not being a system level compromise but if your data is borked then there’s no point having the system.

 There is at least proper sandboxing on macOS and windows for that scenario…

Server side isn’t much better. I’ve never seen anyone set up nginx properly for example. LXC is slowly fixing that but it’s probably better to subcontract that concern out to amazon and use their product abstractions as your security layering now.