Analog Electronics vs Microcontroller Reliability

[schmitt trigger]

Keyword: Therac 25;

https://en.m.wikipedia.org/wiki/Therac-25

Quote:

“These accidents highlighted the dangers of software control of safety-critical systems, “

[German_EE]

If you want a good example of a safety critical system that must deal correctly with failures then look at traffic lights. Years ago I spoke at length with one of their service engineers and he insisted that a double green at a crossroads CANNOT happen with the system as designed. Obviously there are more complex systems than crossroads but there are specific circuits built into traffic light systems that will turn a system off rather than display an unsafe output.

[Gyro]

Yes, traffic light systems include redundant (separate H/W system in addition to the normal watchdog protected S/W control and monitoring) Green Conflict protection to shut down the controller. Green conflict is the worst of all situations for traffic and pedestrian crossings.

They also include Red lamp current monitoring, to protect against too many Red signal lamps going out on any entrance to a junction - No Red lights can look like a failed junction to the driver, not quite as bad, but 'use caution' may not be sufficient if other vehicles have green signals.

[iMo]

It took ~40years to replace the Argon-16 computer in the Soyuz rocket:

In use since 1974, the triply redundant Argon-16 had 2 kilobytes of RAM and 16 kB of ROM and ran at 200 000 operations per second. Its replacement [in 2010] is a much lighter CPU called the TsVM-101 (for "central computing machine"). The TsVM-101 is a much smaller, faster unit capable of 6 million operations per second but with a very modest memory—2 megabytes of RAM, 2 MB of ROM—befitting its modest mission requirements.

https://spectrum.ieee.org/aerospace/space-flight/a-digital-soyuz

http://www.computer-museum.ru/english/argon16.htm

[nctnico]

I worry that the Agile/Continuous Integration/<insert buzzword here> trend of move fast, break things and fix often is spreading like cancer and leaking into things that should never be designed with that philosophy. Anyone who learned to develop consumer electronics or mobile/desktop/web software is going to face a huge culture shock developing anything safety critical.

Actually that is much less of a problem then you'd think. The safety critical stuff has to be well defined so you can add a single layer which checks the conditions and signals an error if there is a mismatch (I usually put in two of these layers written in different ways for redundancy). Whatever is on top can be any kind of software and give whatever command it likes. The safety layer will take care of any illegal situation.  Ofcourse this assumes competent engineers working on -at least the safety related part of- a project.

[SeanB]

there is one application where you only use plain old electrical controls, and that is inside reactors. No electronics, no digital electronics, just plain old magnetic amplifiers and AC synchronous motors, as those are easy enough to design ( as they have been around for over a century)  into a near fail never part. They will work till the materials they are made from literally transmute into something else, though your power bill to run them is not ever going to be small, and they will never be small assemblies either.

However aircraft still use them in the engines, control systems and avionics, despite the complexity of interfacing them with digital computers for input and output. Aircraft magnetic compass ( yes still a mechanical part) in the tail section is basically unchanged from the original Sperry designs, and there is still the old alcohol filled glass unit in the cockpit as the ultimate all power is gone backup, along with the artificial horizon gyro, standby altimeter and the standby air speed indicator, the rest may be glass but they are still there. Just motors, pressure capsules and geared mechanisms, capable of being dormant for decades, and still able to do that mission critical operation when needed.

[schmitt trigger]

If you want a good example of a safety critical system that must deal correctly with failures then look at traffic lights. Years ago I spoke at length with one of their service engineers and he insisted that a double green at a crossroads CANNOT happen with the system as designed. Obviously there are more complex systems than crossroads but there are specific circuits built into traffic light systems that will turn a system off rather than display an unsafe output.

A poster in another forum who was knowledgeable about traffic light controllers, mentioned that there is a redundant controller to detect invalid states, in which it overrides it and starts flashing the yellow light on all roads within the crossing.

[Gyro]

That's certainly possible in some countries, the requirement in the UK however is lights out (all lamp power off).

[SeanB]

Here the conflict monitor will default to all flashing red on fault, or, for some intersections, to a flashing red stop on the minor road and a flashing amber yield on the major road. With the lights all going to LED they had interesting problems with the controllers not recognising the lower load, and with moisture as well causing problems. As the LED lights use a very robust wide range SMPS ( it has to be robust, it can be flashing at 0.5Hz for years in some applications) to generate a 12V power rail, they use around 15W instead of the standard 75W of the original 15000hour rated Traffic lamps, went and looked at the print on an old bulb.

In some cases a temporary (probably till they replace the entire control cabinet and cabling next century) fix is to simply place a lamp in a socket inside the cabinet, to draw the leakage down to a low enough voltage not to trip the conflict monitor. If the lights are out most drivers just mutter "Eishkom again", and wait for a gap in the careening taxis to cross safely.

[Gyro]

... and that's why we shouldn't venture too far from our own comfortable environments... too many country specific hidden hazards just waiting to bite you. 

I spent 5 years designing traffic controllers back in the '90s (for my sins!). The assumption over here is that you can't know where the lamp drive fault is - S/W or logic, shorted triac, damage to buried signal cable cores etc. So the only safe thing to do is kill the power to the whole lighting circuit.

I'm curious about how other countries manage to safely (presumably independently?) bring up other warning light patterns on the same signal heads - different rules, or something we never thought of?

LED signal heads were only just coming in at the time but were already a monitoring headache.

[Zero999]

I imagine monitoring for a failed TRIAC is interesting, since they normally fail on. I suppose you'd monitor the current and if it doesn't switch off, when told so, cut the power by switching off the main contactor.

I would have thought it would be easy to monitor LEDs by including photodiodes inside the lamp module. Granted, this solution would be no good as a simple retrofit, without modifying/replacing the existing control system.

[schmitt trigger]

Speaking of Led traffic lights;
The ones I see over here each individual light is made o a cluster of perhaps 50 or more discrete  LEDs.

The most common (anecdotal) failures appears to be that many, but not all, LEDs go dark. Leaving a dark area on the individual light.

[Someone]

I'm curious about how other countries manage to safely (presumably independently?) bring up other warning light patterns on the same signal heads - different rules, or something we never thought of?

From the Australian standards.

Flashing yellow signal operation may be used to indicate an equipment failure. Red or green signals shall never be flashed

All dark has been the common failure state. It is allowed to be implemented as a routine in a single processor:

The control program shall implement the primary conflict monitor system by means of software algorithms executing in the microprocessor.

With some requirements of a watchdog timer to ensure this integrity check is completed frequently enough.

[NiHaoMike]

I imagine monitoring for a failed TRIAC is interesting, since they normally fail on. I suppose you'd monitor the current and if it doesn't switch off, when told so, cut the power by switching off the main contactor.

Activate a crowbar if the indicator is to be off. Then a stuck on control or line energized by a short to another live line will blow the supply fuse.

[SeanB]

Here the standard Siemens controller uses relays, in plug in modules so you can replace them in the field. Current sensing is done with a simple current transformer per relay, giving feedback so the controller can detect single lamp failures with incandescent, though they had to change the sensitivity for LED clusters. overcurrent is via a circuit breaker per relay, along with a main breaker, and a back up master fuse in the power delivery side, in case a vehicle takes the controller off it's concrete pedestal. feedback is via sense resistors and optocouplers for the controller, but the conflict monitor is separately implemented with it's voltage sensing and change over relays to disconnect all the incoming supplies and change to the flashing relay circuit. You can set the monitor to automatic, where it will disconnect the lights from the controller, and flash for a few seconds, while monitoring for conflict, and switch back if it is transient, or to require manual reset, either by cycling the power, or resetting it.

Can result in some odd displays in rain, with the lights cycling between the normal cycle and flashing every few seconds.

[Gyro]

Wow, relay switching hasn't been used here for decades!    It's all triac switched (apart from the main lamp supply contactor / dimming relay). I guess, yes, it must be down to how much rain we get here - relays wouldn't hack it long term here (although they used to I suppose).

The only things we have rack mounted over here are loop detectors, and monitoring stuff (again talking from my '90s experience). The CPU and logic is all on a main PCB on a panel stood off from the the rear of the case and separate modular (replaceable) Triac lamp switching modules also at the rear of the case. The rest is 'metalwork' a nice big chunky (up to 30A) autotransformer (for night time dimming) mounted on the support cage (the concreted in bit) and keeping everything warm, together with all the clamps to wrangle the heavy armoured street cable coming in from below into some sort of order.

I imagine monitoring for a failed TRIAC is interesting, since they normally fail on. I suppose you'd monitor the current and if it doesn't switch off, when told so, cut the power by switching off the main contactor.

Activate a crowbar if the indicator is to be off. Then a stuck on control or line energized by a short to another live line will blow the supply fuse.

A crowbar on a 30A 240V large junction controller could be something to behold! Dropping out the contactor is deemed sufficient here.

[james_s]

In the USA every control cabinet has a conflict monitor, this is a dedicated piece of hardware that monitors the outputs of the load switches and any kind of conflict will immediately trip the conflict monitor. When this happens a mechanical relay disconnects the signals from the load switches and transfers them to a separate flasher set up to either flash red in all directions, or red on the side street and amber on the main street. Any kind of fault detected in the controller will also cause it to fall back to 4-way flash mode until a worker goes out to reset things.

My uncle spent the last 30 years of his career as a traffic signal electrician, he was often on-call and it was not uncommon for him to have to go out during a wind storm to reset several signals that failed to recover properly after a power outage. When I was a kid he showed me inside one of the cabinets a couple times and he used to give me failed or obsolete hardware that had been replaced. I still have an old Honeywell controller, a few load switches, signal heads and pedestrian signs.