Another deadly 737 Max control bug just found!

[windsmurf]

You're not Boeing to believe this, but... Another deadly 737 Max control bug found
Sim uncovers code-triggered hardware failure that pitches jetliner nose down
https://www.theregister.co.uk/2019/06/27/boeing_737_max_control_bug_found/
https://www.bbc.com/news/business-48752932

[EEVblog]

You'd have to laugh if it wasn't so serious.
But that's what these tests are for.
And who did the sim tests? Was it the FAA or was it Boeing? If it was Boeing you'd think that this would have been kept internal and just quietly fixed as part of the software testing?

[SparkyFX]

This mentions government pilots.
Microprocessor... could be anything.

[floobydust]

If the flight control computer crashed, it might be an older CPU out of RAM... due to new features.  Fault tolerant, redundant processor systems take years to engineer and test and we all know the 737 max. is a rush job.

When Muilenburg announced the MCAS fix is using three timers i.e. triple-redundant S/W I had to cringe because this is so stupid.
Boeing couldn't even deal with two sensors, let alone three timers. It's a noob fix, those have psychological value but no real value adding safety.
I'm sure there's an AoA sensor failure that can cut in and out at a rate that new MCAS will screw up on.

I can't see the planes being up in the air this year.

[BBBbbb]

I do hope BA got a hell of a discount on the recent order of 200pcs of these things...

[GeorgeOfTheJungle]

In the Lion Air crash they had just replaced the AoA sensors, so there has to be something else.

[sokoloff]

In the Lion Air crash they had just replaced the AoA sensors, so there has to be something else.

Because new parts are never faulty out of the box and maintenance activities never introduce faults?

[GeorgeOfTheJungle]

I would think they don't just plug it in (the new thing) and move on. You mean they don't test/verify it works?

[Halcyon]

I'm not a programmer, so forgive me if this is a naive statement, but how is it possible in this day and age for a processor to simply lock up or get stuck in some kind of loop without triggering some kind of watchdog timer (which I'm sure exists) much sooner? In a real-time system such as those found in aircraft, why can't a pilot assume control much sooner?

Even with computer "over loads" and resets during Apollo 11, the spacecraft didn't dive rapidly and crash.

[EEVblog]

I'm not a programmer, so forgive me if this is a naive statement, but how is it possible in this day and age for a processor to simply lock up or get stuck in some kind of loop without triggering some kind of watchdog timer (which I'm sure exists) much sooner? In a real-time system such as those found in aircraft, why can't a pilot assume control much sooner?
Even with computer "over loads" and resets during Apollo 11, the spacecraft didn't dive rapidly and crash.

A (hardware) watchdog timer usually has to restart the whole system, and I'd imagine that's not a trivial thing in a plane (that likely takes a lot of time). So you likely have to rely on the RTOS to handle that sort of stuff.
IIRC the Apollo computer would effectively cold reset every few milliseconds due to those overload errors, and it still managed to land them on the moon.

[EEVblog]

I do hope BA got a hell of a discount on the recent order of 200pcs of these things...

I can't help but visualise a Digikey order cart...

[mac.6]

It's possible that the lockup fires the watchdog, but then you have to restart the system, then the system must recognize and correct the current situation, could take a dozen of seconds or more, enough to put the plane in the dangerous zone.
Even if the watchdog is quick enough to recover, it's an unacceptable situation, especially in this case.

[BBBbbb]

I've haven't followed the issue close enough for the past two months, are they still trying to avoid re-certification of the aircraft and new training of the pilots, by using SW to mimic the 737 dynamics?
If so, that would explain lack of watchdog trigger...

I do hope BA got a hell of a discount on the recent order of 200pcs of these things...

I can't help but visualise a Digikey order cart...

well ordering 200 of something with such a hefty price tag, that has a very uncertain future, seems a bit irresponsible, more appropriate for a personal order on DigiKey... Heck, even for a personal project I'd think twice about ordering an IC with a suspicious EoL date.  I'm certain they made sure they can opt out without consequences if something goes wrong with the "fix", but still has to be one hell of a bargain 

[GeorgeOfTheJungle]

The toyota unintended acceleration issue/bug didn't trigger any watchdog either.

[sokoloff]

It's possible that the lockup fires the watchdog, but then you have to restart the system, then the system must recognize and correct the current situation, could take a dozen of seconds or more, enough to put the plane in the dangerous zone.
Even if the watchdog is quick enough to recover, it's an unacceptable situation, especially in this case.

If it's a system that only auto-drives stab trim, I'd think that dozens of seconds would not be a flight safety risk. (It's a jackscrew, so removing power just leaves the trim as-is, which is almost surely safe.)

[GeorgeOfTheJungle]

Dozens of seconds is an eternity to boot an RTOS. Or is it a Linux?

[ptricks]

Too bad they can't add a toggle switch in the cockpit to cut power from the pin on the micro that controls the specific part the computer is trying to control , something like a auto/manual option.
I'm sure it is more complicated than that, but something along the lines of how a car cruise control disengages with the brake pedal.

[EEVblog]

Too bad they can't add a toggle switch in the cockpit to cut power from the pin on the micro that controls the specific part the computer is trying to control , something like a auto/manual option.

It did/does actually have a switch that disables MCAS, and it could have saved those flights if they had been trained to use it.

[GeorgeOfTheJungle]

Yes they can, just have to 1) have RTFM well and thoroughly and 2) put the flaps into position 1...

[ptricks]

Dozens of seconds is an eternity to boot an RTOS. Or is it a Linux?

It isn't just one OS, they use modular systems so you could have multiple different processor architectures all running their own firmware or OS, so you could restart systems without restarting the entire avionics system.

[SparkyFX]

A (hardware) watchdog timer usually has to restart the whole system, and I'd imagine that's not a trivial thing in a plane (that likely takes a lot of time). So you likely have to rely on the RTOS to handle that sort of stuff.

Those are the finer details of these state machines, if they are well designed, they need to be very specific about the validity of the data and which component can render which data invalid - and when. Now you add certain operating modes (like autopilot or partial autopilot-functions) to this and although switching between these modes the system as a whole still needs to work properly.

I´d consider this to be a solved problem in aeronautical engineering, with very specific requirements on startup values, validity thresholds and sanity checks. Usually such sanity checks would require a more or less sophisticated physical model to be calculated in the background and by comparison of the sensor data to this model it is able to spot a problem (hopefully), but if any component can fail, so can the calculation of the model.

[sokoloff]

Too bad they can't add a toggle switch in the cockpit to cut power from the pin on the micro that controls the specific part the computer is trying to control , something like a auto/manual option.

It did/does actually have a switch that disables MCAS, and it could have saved those flights if they had been trained to use it.

There's a pair of switches in all 737s that cutout power to the stab trim and all 737 pilots have been trained to use that as a memory item (must be recalled without reference to a printed checklist) in the event of stab trim runaway.

[rt]

I do hope BA got a hell of a discount on the recent order of 200pcs of these things...

I can't help but visualise a Digikey order cart...

Just for clarification what was agreed at the Paris Air Show was a 'Letter of Intent to Order' by IAG (International Airlines Group, BA's parent) rather than a firm order.  No hard contract to purchase.  The delivery dates for the 200 aircraft would be between 2023 and 2027 so IAG would expect any of the current AoA-related problems to be sorted out by then.

The list price is US$24B but big forward orders get good discounts and I would expect IAG pushed even harder on price since they were allowing Boeing to announce a first bit of 'good news' at Paris amid all return-to-flight questions and some big Airbus orders.

Also I would expect some clauses in the agreement around delivering a bug-free product with clawbacks/cancellations if not. 

rt

[Sal Ammoniac]

It's possible that the lockup fires the watchdog, but then you have to restart the system, then the system must recognize and correct the current situation, could take a dozen of seconds or more, enough to put the plane in the dangerous zone.
Even if the watchdog is quick enough to recover, it's an unacceptable situation, especially in this case.

It's very unlikely that any warm reset would take dozens of seconds on a control system like this one. No way. The core control functionality probably takes much less time to reset--I'd be surprised if it took more than a few hundred milliseconds.

Watchdogs aren't always effective in resetting hung systems if not used correctly. I saw one product in which the watchdog was kicked in a timer interrupt. The rest of the firmware could hang up tight and as long as that timer interrupt still fired the watchdog would be happy and not reset the system.

[Kleinstein]

Those special computer with possibly some extra HW to check for faults can be difficult to program. It's likely not a normal OS - if at all a more special RTOS variant.  So it would be difficult to get programmers not used to this likely rather old system. I would not be surprised to see something like Motorola's old 88K  (not 68 K, but not that much newer) or similar.

Todays programmers tend to not really care much about computer resources and this could be a problem to an old system. Running out of computer power sounds a little like out of memory, out of stack space, interrupt saturation, latency violations or similar. A watchdog could in same cases even cause a hung system, e.g. if constantly triggered. It may only take a little more interrupt load to slow down old code to trigger the watchdog from time to time. This is kind of a hard to find error.

Still odd that the problem was found in the more official tests and not with Boing internal ones.  Though it might even be a good idea to no have internal tests - so the programmers have to make sure the program works without actually testing it without the public noticing failures. However I don't think Boing is going this far, especially not if in a hurry.