Another deadly 737 Max control bug just found!

[Gyro]

Still odd that the problem was found in the more official tests and not with Boing internal ones.  Though it might even be a good idea to no have internal tests - so the programmers have to make sure the program works without actually testing it without the public noticing failures. However I don't think Boing is going this far, especially not if in a hurry.

Could be a case of Boing technical folks still in the mindset of business as usual with the FAA, while the FAA suddenly feel the need to look really good at their job.

Their next design is really going to be under the microscope, regardless.

[David Hess]

A (hardware) watchdog timer usually has to restart the whole system, and I'd imagine that's not a trivial thing in a plane (that likely takes a lot of time). So you likely have to rely on the RTOS to handle that sort of stuff.
IIRC the Apollo computer would effectively cold reset every few milliseconds due to those overload errors, and it still managed to land them on the moon.

I have written programs like that for real time systems.  What has to happen is retention of state between resets such that the system can continue executing tasks and bypass the task which caused the problem if necessary.

[David Hess]

Watchdogs aren't always effective in resetting hung systems if not used correctly. I saw one product in which the watchdog was kicked in a timer interrupt. The rest of the firmware could hang up tight and as long as that timer interrupt still fired the watchdog would be happy and not reset the system.

I have seen that multiple times now.

Another one I remember was discovered by UL when testing a garage door opener.  The Zilog Z8 microcontroller had an unnoticed design flaw where the watchdog timer ran off of the crystal clock so if the crystal failed, the processor stopped but so did the watchdog timer which would otherwise have issued a reset which would have set the output ports to a known and safe state. (1)

The UL test included crushing the crystal with pliers while the door was closing.  My guess is that they had seen this safety issue before in designs which did not include a watchdog timer.

(1) There are other ways to handle this like AC coupling the output control signals so if the processor stops, the controls return to a safe state.

[raptor1956]

Finding things in testing is why you test, but the fact it was not a Boeing pilot is a bit troubling. 

In a potentially related story three managers at the FAA that are responsible for monitoring Southwest Airways were reassigned as subordinates complained that they were punished for finding problems at Southwest. 

https://www.reuters.com/article/us-faa-southwest-safety/faa-reassigns-three-in-office-overseeing-southwest-airlines-source-idUSKCN1TR011

There is a mindset in certain political circles that regulation is bad and that regulations should be rolled back or eliminated to the greatest extent possible -- decades of this mindset and the consequent reduction or elimination of regulations are now rearing there head.  That Boeing was able to 'self certify' due to these weakened regulations is certainly a factor in the deaths of 346 people.  So, to Boeing and other companies looking to reduce or eliminate regulations I would like to say ... be careful what you ask for because you might get it and then live or die with the consequences. 


Brian

[MrMobodies]

"During the FAA’s review of the 737 Max software update and recent simulator sessions, the Federal Aviation Administration identified an additional requirement that it has asked the company to address through the software changes that the company has been developing for the past eight months.

Boeing agrees with the FAA's decision and request, and is working on the required software. Addressing this condition will reduce pilot workload by accounting for a potential source of uncommanded stabilizer motion.

In light of the problems and they want to reduce pilot workload.

I thought there were at least two pilots in there but is that that the issues with stabilizer motion was causing them a lot of problems.

[windsmurf]

Too bad they can't add a toggle switch in the cockpit to cut power from the pin on the micro that controls the specific part the computer is trying to control , something like a auto/manual option.

It did/does actually have a switch that disables MCAS, and it could have saved those flights if they had been trained to use it.

There's a pair of switches in all 737s that cutout power to the stab trim and all 737 pilots have been trained to use that as a memory item (must be recalled without reference to a printed checklist) in the event of stab trim runaway.

They modified the switch functionality with the 737 max.

https://www.seattletimes.com/business/boeing-aerospace/boeing-altered-key-switches-in-737-max-cockpit-limiting-ability-to-shut-off-mcas/

[MrMobodies]

But on the Ethiopian Airlines flight, the pilots appear to have recognized the errant MCAS problem and flipped the cutoff switches as descrbed in the checklist. But then it appears that the pilots were unable to move the manual wheel, likely because the forces on the tail made it physically challenging to turn.

After failing to manually control the stabilizer, the Ethiopian Airlines pilots appear to have flipped the cutoff switches back on, which awakened the MCAS system. It soon sent the plane diving to Earth.

Lemme said he’s surprised that Boeing made the change to take away the functionality that could have allowed the pilots to shut off MCAS without shutting off the electric switches at their thumbs.

So turn off the cutout switches and it cuts out the controls turn it back on and the MCAS overrides the pilots control and controls the plane based on some faulty sensor.


https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Augmentation_System

Boeing and the FAA decided that the AOA display and an AOA disagree light, which signals if the sensors give different readings, were not critical features for safe operation.[42] Boeing charged extra for the addition of the AoA indicator to the primary display.[43][44]. In November 2017, Boeing engineers discovered that the standard AoA disagree light cannot independently function without the optional AoA indicator software, affecting 80% of the global fleet which had not ordered the option.[45][46][47] The software remedy was scheduled to coincide with the roll out of the elongated 737 MAX 10 in 2020, only to be accelerated by the Lion Air accident. Furthermore, the problem had not been disclosed to the FAA until 13 months after the fact. Although it is unclear whether the indicator could have changed the outcome for the ill fated flights, American Airlines said the disagree indicator provided the assurance in continued operations of the airplane. "As it turned out, that wasn't true." [48]

Boeing CEO Dennis Muilenburg has stated that there was "no surprise, or gap, or unknown here or something that somehow slipped through a certification process."[49] On April 29, 2019 he stated the design of the aircraft was not flawed and reiterated that it was designed per Boeing's standards.[50] In a May 29 interview with CBS, Boeing admitted that it had botched the software implementation and lamented the poor communications. [51]

It looks like some things were also missing and the airlines wanted save money.

[Gyro]

I knew this would happen as soon as a second thread started - go and look at the original thread, there is a lot of relevant information already covered there, I suggest starting at the last page and working backwards!

 https://www.eevblog.com/forum/chat/lion-air-crash-jakarta-boeing-737-had-prior-instrument-error/1075/

It looks like some things were also missing and the airlines wanted save money.

Irrc, that AOA disagree indicator was a $80k per plane option and Boeing couldn't make it work.

[MrMobodies]

I knew this would happen as soon as a second thread started - go and look at the original thread, there is a lot of relevant information already covered there, I suggest starting at the last page and working backwards!

Irrc, that AOA disagree indicator was a $80k per plane option and Boeing couldn't make it work.

Sorry about that. I forgot all about the thread.

Lot of money for something they couldn't get working.

[SiliconWizard]

That doesn't bode well for Boeing.

[djacobow]

Too bad they can't add a toggle switch in the cockpit to cut power from the pin on the micro that controls the specific part the computer is trying to control , something like a auto/manual option.

It did/does actually have a switch that disables MCAS, and it could have saved those flights if they had been trained to use it.

There's a pair of switches in all 737s that cutout power to the stab trim and all 737 pilots have been trained to use that as a memory item (must be recalled without reference to a printed checklist) in the event of stab trim runaway.

They modified the switch functionality with the 737 max.

https://www.seattletimes.com/business/boeing-aerospace/boeing-altered-key-switches-in-737-max-cockpit-limiting-ability-to-shut-off-mcas/

No pilots were ever trained to flip just one of those switches. They weren't labeled with different functions, they weren't described with different functions, at least not in the QRH. It was a weird design change, to be sure, but I'm not sure it matters all that much.

I've followed this whole saga pretty closely and I still am not sure I understand what happened. I mean, I get that single AOA + MCAS caused the machine to trim down hard. But that should have been manageable with the standard training. Knowing that MCAS existed would have helped pilots to have a picture of what was happening, but I don't think it would have been necessary.

There was some evidence that once the plane was trimmed down and the pilots were pulling hard to recover, that the load on the stab made it very difficult to turn the trim wheel, at least without the pilots pushing the nose even further over to unload the stab.

If that is the case, I think that's the more serious problem than MCAS, or at least MCAS alone.

[sokoloff]

The trim is hard to run manually when the horizontal stab is biased aircraft nose down, the elevator is deflected aircraft nose up to compensate, and the aircraft flying at high indicated airspeed, that combination making for high loads on the stab jackscrew and following nut. The answer once you've let things evolve to that point is to either rollercoaster the airplane (to periodically unload the elevator and use those moments to manually trim), to take the thrust down and thereby reduce indicated airspeed and trim manually, or to power the electric trim and run the trim nose up electrically.

I don't think that it's a distinct or Max-specific flaw. It's a matter of aerodynamics. While I'm generally critical of the Ethiopian crew, I find it hard to fault the crew for not inventing the rollercoaster solution on the fly [so to speak]. (In particular their decision to leave the engines at a very high thrusts is odd, but most disconcerting is the decision to turn the stab trim back on, momentarily command aircraft nose up trim, and then [inexplicably] leave the stab trim powered but without commanding continued aircraft nose up trim by thumb switch. I think that decision is very hard to understand.)

[djacobow]

The trim is hard to run manually when the horizontal stab is biased aircraft nose down, the elevator is deflected aircraft nose up to compensate, and the aircraft flying at high indicated airspeed, that combination making for high loads on the stab jackscrew and following nut.

...

I agree with all of that. If the Ethopian crew had gotten the plane trimmed, it's hard to understand why they left the system on. Well, actually, I could see that happening ONCE. But it happened several times, right?

The media has made a huge deal out of the MCAS flaw, but I think most pilots just "get" (or should get) that every so often under the right (wrong) circumstances, the automation is going to try to murder you. That's just life managing a very complex system. That's not to say that such flaws are OK. They are obviously not OK. You probably shouldn't be sitting up front in an airliner if you don't think there's a chance it might actively try to kill you.

The reason this got so deadly is that under the aerodynamics of trying to recover against the pitch, the trim is hard to move. In the heat of the moment, it might be hard to quickly reason out  that you need to unload the stabilizer before you can move the wheel, especially if this regime of flight had not been part of your training.

It's the combination that was really deadly, not either in itself.

[BBBbbb]

well ordering 200 of something with such a hefty price tag, that has a very uncertain future, seems a bit irresponsible.

With such scrutiny, I would bet B737MAX will be the world's safest plane ever once things are ironed out.

If you put the same amount of scrutiny on any other aircraft, you will find equally hilarious flaws.

Yes, for sure there are other very flawed aircrafts out there flying, but this is more to do with the lack of disclosure and mismanagement of all the attempts of a blame game that followed the series of accidents.
Isn't it concluded that with a proper training and added redundancy this could be a flyable system? But is it still a competitive one after the fixes?


BA (intent to) order does look like a gimmick, but unless there is a discount on other models that are getting ordered, or a restructuring of an ongoing payments to Boing, this really seems just like bad marketing for BA.

[CiscERsang]

With such scrutiny, I would bet B737MAX will be the world's safest plane ever once things are ironed out.
If you put the same amount of scrutiny on any other aircraft, you will find equally hilarious flaws.

Too late. The comprehensive scrutiny must have been performed before first commercial flight. Not today.

[CiscERsang]

Too late. The comprehensive scrutiny must have been performed before first commercial flight. Not today.

A320 is to date, the only commercial jetliner crashed on its debut, yet it is still one of the most successful planes among B737 and DC10, which on its own is also riddled with fatalities.

There's always flipside of the coin. Such things like, competition, struggle for the markets, rush... that I mean.

[BravoV]

Still odd that the problem was found in the more official tests and not with Boing internal ones.

The fact is no different from Boeing bought and used a lot of these similar to those One Hung Low's stuffs ...

[Kleinstein]

Watch from 1:05:00 to 1:06:30,  pilot admits Boeing engineers contacted them to tell them that there are "negative dynamic stabilisation problems" with the 737Max

The next 5 minutes following are interesting: The pilots essentially say that they consider the MCAS system without an explanation and extra training more of a problem than a help.

In hind sight this makes absolute sense: even if working as intended the MACS intervention to a pilot who does not know about the system (the initial state) should see this as a system failure /  run away trim situation. So it is odd to add such a system to avoid extra training - it is more in the opposite: due to the MACS the pilots would need extra training / simulator hours.

Back to the topic of the new thread: It looks like the new bug found was in the software fix that Boing made in a hurry. So it does not effect the old planes - except that flown by FAA test pilots for real world testing. Odd they found the bug in the simulator now and not before or during flight testing.

[Kjelt]

It's very unlikely that any warm reset would take dozens of seconds on a control system like this one.

Not sure how many systems you need to fly this plane probably the fancy GUI computers are not needed?
Also not sure how accurate the sims are these days but from cold start to having visual GUI displayed takes over one and a half minute
https://youtu.be/vW6gMzDsKCg

[GeorgeOfTheJungle]

No no no, they just had to keep pushing the (electric) trim button on the yoke before flipping the stab cutout switch, because that button overrides MCAS. They didn't use it properly, not pressing for long enough. Could also have put the flaps lever to position 1, this disables MCAS and makes the jack screw nut turn faster when trimming with the yokes button.

[dzseki]

Could also have put the flaps lever to position 1, this disables MCAS and makes the jack screw nut turn faster when trimming with the yokes button.

At that time perhaps this was a lesser known fact, also not sure how the flaps would have behaved at that high speed they were at, flaps is not meant for cruising.

[sokoloff]

Could also have put the flaps lever to position 1, this disables MCAS and makes the jack screw nut turn faster when trimming with the yokes button.

At that time perhaps this was a lesser known fact, also not sure how the flaps would have behaved at that high speed they were at, flaps is not meant for cruising.

Well, we wouldn't want to damage the flaps (by extending them at speeds over Vfe), now would we... 

[donotdespisethesnake]

In the future, all planes will be flown remotely by armchair experts on the internet.   

[Yansi]

In the future, all planes will be flown remotely by armchair experts on the internet.   

Sorry, but that is just a dumb argument. 

I do not see anything wrong with anyone (even laymen) pointing to more or less obvious design flaw.

[coppercone2]

classic
https://tech.slashdot.org/story/19/06/29/1555243/boeing-falsified-records-of-a-new-787-that-leaked-fuel

indian contractors writing aerospace software for EIGHT dollars an hour