Another facepalm 737 software bug.

[splin]

https://www.theregister.co.uk/2020/01/08/boeing_737_ng_cockpit_screen_blank_bug/

Amid the various well-reported woes facing America's largest airframe maker, yet another one has emerged from the US Federal Aviation Administration; a bug that causes all pilots' display screens in the 737-NG airliner family to simply go blank.

That bug kicks in when airliner crews try to program the autopilot to follow what the FAA described as "a selected instrument approach to a specific runway".

Seven runways, of which five are in the US, and two in South America - in Colombia and Guyana respectively – trigger the bug. Instrument approach procedures guide pilots to safe landings in all weather conditions regardless of visibility.

I wouldn't want to be the one resposible for that - not likely to be career enhancing. Software bugs are hardly uncommon even in safety crtical industries but the very high profile of quality control within Boeing and the close attention of the world's media looking for any new fault, however minor (I'm not suggesting this particular bug is minor, or otherwise), must mean a very uncomfortable working environment for Boeing's engineering departments.

[MT]

It comes to a point the question all by it self surfaces, that is someone actually trying to take Boing out of business?

[Yansi]

It comes to a point the question all by it self surfaces, that is someone actually trying to take Boing out of business?

Airbus, probably?

[tom66]

It comes to a point the question all by it self surfaces, that is someone actually trying to take Boing out of business?

Boeing doesn't need any help doing that with the rate these bugs come up.

US government wouldn't let them go under, too much pride. But they could face being broken up.

[Kleinstein]

It looks like the bug is not that new. It just got new public attention as more people look for an interesting story. Last year such a story would have gone largely unnoticed.

[MT]

It comes to a point the question all by it self surfaces, that is someone actually trying to take Boing out of business?

Boeing doesn't need any help doing that with the rate these bugs come up.
US government wouldn't let them go under, too much pride. But they could face being broken up.

But being broken up i bet likely means the end of manufacturing big airliners.

Airbus, probably?

A380 retired = apparently not the great success the had hoped.

[splin]

It looks like the bug is not that new. It just got new public attention as more people look for an interesting story. Last year such a story would have gone largely unnoticed.

Exactly. Must be other airline maufacturers muttering "there but for the grace of God go I we". Rather like the diesel car manufacturers who seem to have got away with NOX emissions far exceeding official limits (in the spirit of the law if not in practice) because, luckily for them, VW took all the heat by going a step too far by deliiberate fraud in the US. The EU forced VW to apply 'fixes' to reduce NOX emissions to EU vehicles even though the likes of Nissan and Renault who had models that were (allegedly) far worse than VW's were left alone.

Once the media and especially social media get their teeth into a wounded victim, rational discussion, analysis and risk assements of the victim's problems compared to their competitors are very hard to get re-established.

[nigelwright7557]

Even simple programs can have bugs in them so something like a massive system to control a plane will have a few.
I have 40 years experience writing software.
I found the main goal is try and write it correct in the first place rather than write average code and then test it to death.
Testing doesn't always pick up every bug.
I have some software that has matured over  30 years and still find the odd bug now and then.

A lot depends on the programmer too, if he just out of Uni and a novice then should he really be writing safety critical software.
Ironically the good programmers no longer program as they have progressed into management.

I remember my first program which was 3 lines of code and according to the compiler it had 11 bugs in it !

[SilverSolder]

Even simple programs can have bugs in them so something like a massive system to control a plane will have a few.
I have 40 years experience writing software.
I found the main goal is try and write it correct in the first place rather than write average code and then test it to death.
Testing doesn't always pick up every bug.
I have some software that has matured over  30 years and still find the odd bug now and then.

A lot depends on the programmer too, if he just out of Uni and a novice then should he really be writing safety critical software.
Ironically the good programmers no longer program as they have progressed into management.

I remember my first program which was 3 lines of code and according to the compiler it had 11 bugs in it !

There are some really good programmers around -  but they are only recognized as such (and hired) by really good managers, working for really good executives that recognize the really good managers.  How often do the stars, the sun, and the moon actually align?

[shakalnokturn]

About once for every five safety critical bugs that go public in aviation?

[donotdespisethesnake]

There are some really good programmers around -  but they are only recognized as such (and hired) by really good managers, working for really good executives that recognize the really good managers.

That really is the problem, software is viewed as a cost which can be reduced as much as possible. When the majority of managers are career "business" managers and not former engineers, it leads to a spiral of decline. I have seen this in several companies I have worked for. The engineering dept gets crunched into oblivion, but the managers who were "successful" at cutting costs move upwards.

I think there is an inherent law that companies trend towards being run for the benefit of the managers. I don't see a way out of that.

[SilverSolder]

There are some really good programmers around -  but they are only recognized as such (and hired) by really good managers, working for really good executives that recognize the really good managers.

That really is the problem, software is viewed as a cost which can be reduced as much as possible. When the majority of managers are career "business" managers and not former engineers, it leads to a spiral of decline. I have seen this in several companies I have worked for. The engineering dept gets crunched into oblivion, but the managers who were "successful" at cutting costs move upwards.

I think there is an inherent law that companies trend towards being run for the benefit of the managers. I don't see a way out of that.

Let's not forget that the world needs bad managers (and bad engineers) as much as the good ones - there is plenty of work around to be getting on with!

We have a right to expect, though, that the people that build and run safety critical areas of aerospace, nuclear, etc., are among the best of us.  When that starts to slip, you know the rot started at the top.

[Sal Ammoniac]

Perhaps the airline industry needs to write flight software the same way the software for the space shuttle was written.

https://www.fastcompany.com/28121/they-write-right-stuff

[eugenenine]

I always thought the scifi/star trek type of shows would be far more realistic if, instead of 'realigning the warp coils', they had to fix bugs in the ships code.

[Stray Electron]

There are some really good programmers around -  but they are only recognized as such (and hired) by really good managers, working for really good executives that recognize the really good managers.

That really is the problem, software is viewed as a cost which can be reduced as much as possible. When the majority of managers are career "business" managers and not former engineers, it leads to a spiral of decline. I have seen this in several companies I have worked for. The engineering dept gets crunched into oblivion, but the managers who were "successful" at cutting costs move upwards.

I think there is an inherent law that companies trend towards being run for the benefit of the managers. I don't see a way out of that.

   Nails it!  And to make matters worse, SW is one of the easiest things to outsource, frequently to a company in a place like India. But the programmers there don't have any experience in the field that the SW is to be used in and also frequently don't have experience in developing and testing large SE programs so tons of bugs slip through.

   As long as the US and regulatory agencies like the FAA and companies like Boing are content to use managers that aren't engineers and that aren't SW people and to let them subcontract out to small, inexperienced SW companies, particularly in developing countries, these kind of mistakes are going to continue to happen.

[floobydust]

Software is intangible, you cannot see how much is done, how good it is, where you are at.

Dumb managers go nuts not knowing the progress, especially since in software development you sometimes have to jump back to then jump ahead, it's a non-linear path. So the manager can panic and push the team- when it's wrong to do so and you get low quality software. Or he thinks just outsource it, no big deal.  Or he puts the programmers in noisy, open office cubicles where they can't concentrate.

From last Thursday, Boeing's release 100 pages of internal emails and text messages to Congress:
"This airplane is designed by clowns, who are in turn supervised by monkeys,” one Boeing pilot wrote. 

https://www.seattletimes.com/business/boeing-aerospace/boeing-internal-documents-reveal-culture-of-deceit-to-keep-down-costs-of-737-max
https://www.nytimes.com/2020/01/09/business/boeing-737-messages.html

[aargee]

I think MBA courses have a lot to answer for.

[floobydust]

What would the MBA program advise in this scenario?

Dennis Muilenburg directs the company into making a killer aircraft- 346 lives gone and more to come, Boeing stock loses over $50B in market cap. He gets fired and gets an $80.7M golden parachute with pension, even without severance or 2019 bonus.
"By contrast, Boeing set aside $50 million to compensate the families of crash victims."

[TerraHertz]

https://www.infowars.com/designed-by-clowns-supervised-by-monkeys-boeing-employee-shreds-737-max-in-internal-messages/
‘Designed By Clowns & Supervised By Monkeys’: Boeing Employee Shreds 737 MAX in Internal Messages

[donotdespisethesnake]

We have a right to expect, though, that the people that build and run safety critical areas of aerospace, nuclear, etc., are among the best of us.

People often say that, I'm not sure what legal or moral reasoning it is based on? I'm not aware of any specific principle, apart from a general sense of "natural justice". In practice, you get what you pay for, and people always pay as little as possible.

Having worked in safety critical areas, I can say that they draw from the same pool of engineers as everyone else. Some are good, some are bad, most are average. There is no special training given, you are expected to "learn on the job". Most engineers I have met learn the minimum they need to do their job, and have no interest learning more outside of that. There is no pay premium given to people with safety critical experience. It is normally a question of "who is available", and how cheap. Usually, the "safety critical" process methodology generates a huge amount of red tape, which has little impact on the quality of the software. In fact, it gives engineers less time to actually write good code.

[blacksheeplogic]

Some perspectig of the 737 saftey:

[chris_leyson]

@blacksheeplogic Thanks for posting that, it puts things into perpective as far as safety in terms of number of passengers and distance travelled. Not sure how the automobile industry would stack up.

Anyway back to blank screens when programming ILS approach

Seven runways, of which five are in the US, and two in South America - in Colombia and Guyana respectively – trigger the bug.

I think the flight control computers have more important tasks to do on an ILS aproach and updating screens takes a much lower priority. Allegedly seven runways are affected but thousands of other runways are not. Maybe it's a fault within the ILS system itself that didn't predict the amount of RF pollution that we see nowadays, perhaps the instrumentation is at fault.

It might not be a bug but flight computers giving ILS priority over everything else.
Have there been any recorded incidents of NGs missing these seven runways ? No.

[floobydust]

I have also done safety-critical H/W and S/W design mostly under IEC 61508.
At the engineering level, software quality is only obtained by documenting it to death- every requirement, subroutine function, test cases- all documented and signed off. With everything ultra-defined, then the S/W can be coded by clowns and still be decent. IEC 61508 has entire sections devoted only to a corporation's quality system, project management, and even employee qualifications to do safety-critical work in the first place.

Of course this takes forever and companies skip over things, cheat the process- extra easy to do when outsourcing.
"fresher" jobs for Boeing from http://www.durgajobs.com
Anyone wonder how much aviation experience there is in a country that does not make aircraft?
I see most credentials are fake, especially when you ask a few simple questions.

[chris_leyson]

As for the email "Designed by clowns and managed buy monkeys" allegedly posted by some grumpy or concerned pilot, lets see the rest of the email to put it into context.
There is an old expression, there are those who talk the talk and those who walk the walk and that appies to all walks of life. Those who talk the talk maybe believe that everything they read on the internet must be true and truth becomes a simple cut and past operation. Those who walk the walk know differently. Just saying.

[floobydust]

These sources are not, unlike Boeing, entirely full of deceit:
https://www.seattletimes.com/business/boeing-aerospace/beyond-the-macho-pilot-trash-talk-737-max-documents-reveal-how-intensely-boeing-focused-on-cost/
https://www.nytimes.com/reuters/2020/01/11/business/11reuters-boeing-737max-factbox.html

The emails and text messages are just out, not available to the general public yet, that I know of. The House Committee, FAA and press still digesting them.

I don't think the 346 dead will be talking or walking much. But as you wish, give Boeing the benefit of the doubt. The messages are all about the little simulator in Miami having a little bug and Mark Forkner being totally innocent of “... jedi-mind tricking (foreign) regulators into accepting the training that I got accepted by FAA.”

Most of the messages are simply damning of Boeing's culture, ethics and the 737 Max development program.