Problems in Modern Embedded Dev

[country2304]

I'm thinking about making an open-source project related to some problems I've experienced building commercial IoT products and I'm interested to hear other people's thoughts on the stumbling blocks they have encountered in embedded device development.

- I would like to be able to use higher-level languages with nicer syntactic sugar, (using Micropython in the past has sped up dev time, especially during rapid prototyping). However, most solutions like micropython don't seem viable in most constrained environments.

- When I've built CI pipelines in the past it has been not easy to integrate the device firmware into end-to-end tests of our entire system, backend included. I would like to be able to use as much of the actual code run on the device to be used in the tests and not have to write mock code for different hardware functions. (Of course, this kind of test is more useful in scenarios without critical timing components etc. that would require proper simulation of the hardware).

- More generally, I've found that in the embedded world there are fewer tools for managing projects with systems like package managers, maybe more equivalent to what exists in the js world.

[Slh]

I'm not sure if there's a question in all of that but there's no substitute for having real hardware for testing. You need a rig that can input all of the inputs (eg button presses/shaking/audio whatever your IOT thing measures) and take all of the outputs (audio/lights/motors). Unfortunately that costs money and takes time to set up and program. But no one ever said that hardware was cheap

You can get fancy HIL rigs that might make some of this easier. Probably not cheaper though

[nctnico]

If you have a good model of the hardware, you can do automated testing for sure. So the claim that you absolutely need hardware to do software verification is false. But you'll need to invest the time to create  a good model of the hardware in software (including timing). The advantage of having a model instead of real hardware is that you can test things like hardware failures and values going out of bounds without causing damage.

- I would like to be able to use higher-level languages with nicer syntactic sugar, (using Micropython in the past has sped up dev time, especially during rapid prototyping). However, most solutions like micropython don't seem viable in most constrained environments.

I would not really care about constrained environments. Microcontrollers continue to be equiped with larger and larger memories with dropping prices. So by the time you have a solution, it will be outdated (*). You could opt to use Lua instead of Micropython as Lua has a slightly smaller footprint. Just check how incredibly cheap an ESP32-S3 chip is for the amount of processing power, memory and storage space you get.

* Just look at how Nokia failed by clinging on to their rather crappy Symbian OS while Google killed them by creating Android using Java. There couldn't be a bigger contrast in approach but what Google did was enabled by advances & price reductions in technology.

[dobsonr741]

If you can allow the luxury of micropython, then you can also think about having more compute intensive features for the product, or lower the hardware requirements to lower cost. I bet you would not find micropython in shipping devices by the millions in qty.

As far as stumbling blocks: not enough test vectors/cases in testing. True for both software and hardware: Not enough variance of data, or not enough variation of environmental disturbance/noise/transients if the DUT is hardware.

[Smokey]

There is a reason the device categories of " microcontroller" and "microprocessor" are two separate things.  The problem isn't always resource constraints (it often is), unless you count time as a resource.

More and more stuff will accommodate higher levels of overhead as the clock frequencies and memory levels continue to go up, but there will always be applications that are better suited for real optimization.

Target raspberry pi and that stuff if you want an easy application for high level languages and integration.  Real embedded microcontroller type engineers probably aren't going to care what you are doing and are going to keep working as they always have for quite some time.

[nctnico]

Real embedded microcontroller type engineers probably aren't going to care what you are doing and are going to keep working as they always have for quite some time.

Real embedded engineers adapt. Even stuff like washing machines and coffee makers are connected nowadays. Using Micropython or Lua are good solutions to create web pages and/or connected to a server on internet without needing to run Linux. There is likely a huge amount of devices out there which use higher level (scripted) languages to implement high level functionality quickly. Security is just another reason. Python and Lua run in their own constrained sandboxes so the chances of interfering with low level processes is minimal to non-existent.

[Smokey]

Real embedded microcontroller type engineers probably aren't going to care what you are doing and are going to keep working as they always have for quite some time.

Real embedded engineers adapt. Even stuff like washing machines and coffee makers are connected nowadays. Using Micropython or Lua are good solutions to create web pages and/or connected to a server on internet without needing to run Linux. There is likely a huge amount of devices out there which use higher level (scripted) languages to implement high level functionality quickly. Security is just another reason. Python and Lua run in their own constrained sandboxes so the chances of interfering with low level processes is minimal to non-existent.

I'm with you.  Everyone has to adapt eventually.  But you would probably be fired from the coffee machine company if you tried to use a processor that ran micropython.  Hell, you would probably be fired if you didn't fully optimize out that last $0.10 of bypass caps.

[metebalci]

- I would like to be able to use higher-level languages with nicer syntactic sugar, (using Micropython in the past has sped up dev time, especially during rapid prototyping). However, most solutions like micropython don't seem viable in most constrained environments.

I do not know much what libraries/APIs are used in embedded world, but what I see is they are too much low level. I mean there is a need for low level but even high level ones are not high enough. So I think this point can be divided into two. One is independent of the language, it can be C, the level of abstraction can be increased. Second is like you said the language can be different, but I guess then there cannot be only one language, but (micro)python would have a strong support probably.

- When I've built CI pipelines in the past it has been not easy to integrate the device firmware into end-to-end tests of our entire system, backend included. I would like to be able to use as much of the actual code run on the device to be used in the tests and not have to write mock code for different hardware functions. (Of course, this kind of test is more useful in scenarios without critical timing components etc. that would require proper simulation of the hardware).

Do you mean some kind of simulator/emulator for embedded platforms (e.g. an emulator for STM32F0) or something else ?

[drago462]

Just chiming in on your IoT product development woes:

Using Cool Languages in Tight Spaces: You've got a point about MicroPython being handy for quick builds but a bit of a pain in resource-tight spots. Wondering if others have found ways to slim these languages down for the crunch.

Making Firmware Play Nice with Testing: Meshing device firmware with full system tests, including the backend, sounds tricky. Curious if anyone's found a slick way to do it without resorting to mock-ups.

Need Better Tools for Keeping Things Organized: Totally hear you on the lack of solid project management tools in embedded systems. Sharing some pro tips or brainstorming new tools could be a real lifesaver.

Your open-source project idea to dive into these issues seems like a solid move for anyone tangled up in IoT and embedded systems. Definitely worth digging into!

Cheers! 🚀

[Berni]

MCU firmware development =/= PC software development

The main reasons being that MCUs are less powerful so they need efficient optimized software, as well as MCUs are tightly integrated with the hardware. There is also no OS to help bring together drivers and applications to coexist.

So in order for a MCU simulator to be useful it also needs to simulate all the hardware around it, otherwise the program is just going to sit there constantly trying and failing to communicate with other chips on SPI or I2C while spitting data out of a UART that you can't see if it is not simulated. There is a simulator that can do it tho:
https://www.labcenter.com/simulation/
(but you have to be lucky for it to support the hardware you want to use)

So as a result you tend to get very purpose built software in C/C++ that is built for the particular platform. This is the same reason why PC games tend to be written in C++. Going low level like this allows the programmers to extract the most performance possible out of the hardware.

This modern day software reliance on abstraction layers upon abstraction layers is the reason why computers seam ever slower even tho the modern smartphone is more powerful than the supercomputers of a few decades ago. But the software is becoming so bloated that we still have to wait for things to load.

[tggzzz]

MCU firmware development =/= PC software development

The main reasons being that MCUs are less powerful so they need efficient optimized software, as well as MCUs are tightly integrated with the hardware. There is also no OS to help bring together drivers and applications to coexist.
...
So as a result you tend to get very purpose built software in C/C++ that is built for the particular platform. This is the same reason why PC games tend to be written in C++. Going low level like this allows the programmers to extract the most performance possible out of the hardware.

This modern day software reliance on abstraction layers upon abstraction layers is the reason why computers seam ever slower even tho the modern smartphone is more powerful than the supercomputers of a few decades ago. But the software is becoming so bloated that we still have to wait for things to load.

Yes, but also no.

Those constraints are real, and the abomination of software abstraction layered on software abstraction is real and ... inelegant. But - with the right architecture - useful development techniques are possible within those constraints.

Here's a set of techniques I used too many decades ago using C on a PDP-11 and Z80-based hardware...

One good set of architectural principles that enables unit testing and incremental development is:

• describe the overall algorithmic function in terms of one or more finite state machines (FSM), i.e. where the actions (i.e. computation/processing) occur where an event (i.e. input) arrives and the specific actions are dependent on the current state (i.e. what has happened previously). Yes Virginia, FSMs are useful for more than parsing text in compilers
• the events can be any input, e.g. a message arriving, an input toggling, a timeout, etc
• the actions can be any computation and output, but the outputs should be "fire and forget", not blocking
• clearly events and action are closely tied to hardware
• the states are anything convenient
• roughly speaking, events and actions are instantaneous whereas states are continuous

Then split each event and action into two parts:

• hardware independent meaning
• hardware dependent i/o bit twiddling

i.e. in effect a very thin and efficient hardware abstraction layer.

Implementation will be that

• the hardware inputs cause interrupts, and the interrupt routine only creates a hardware independent event that is put in a FIFO for processing by the FSM. That means interrupts are short.
• the main program is to wait until there is an event in the FIFO, pop the event, process the event to completion, and go back to waiting for the next event. That means event processing must be kept short, and the FSM must be designed so that can be the case

On the real hardware, use unit testing to verify that

• every input changes generate the correct hardware independent event
• each hardware independent action causes the correct output changes

On any convenient platform, e.g. a workstation, use unit testing to verify that sequences of mock input events placed into the FIFO cause the correct FSM algorithms and the correct mock actions.

[SiliconWizard]

tggzzz gives a few good starting points here.

As a more general thought, contrary to a common belief, programming itself is not the most difficult or even important part of software design, be it embedded or not.
Everytime one focuses on the programming part while neglecting the rest, the result is poor.

[Berni]

Yeah that would be test driven development where you write a test harness around a module of code that runs the module of code trough its paces as if it was running for real.

For some things this is indeed a very good approach.

Emulating simple peripherals like for example a temperature sensor is really easy by replacing the init function with nothing and just returning some global variable when asked about temperature. Similarly GPIO driven LEDs and such can just be stored as a variable and displayed somewhere or printed to console or something.

But if you start having very complex relationships with your hardware this can become increasingly more difficult. Like for example if your MCU was also acting as a sensorless BLDC motor controller then you would have one hell of a time writing a test harness for it. Not only would you need to emulate peripherals like ADCs, Comparators, Timers etc..  but even emulate them with a lot of timing precision, you might even need to replicate interrupt latency or delays in code execution in interrupts. Then on top of all that you would also need to make a live a physics simulation of a BLDC motor....etc At some point emulating all of this becomes so hard that it makes more sense to just test that on real hardware.

But for a lot of IOT things id agree that it shouldn't be too hard to write a test harness for the code. Since they don't usually have that kind of timing critical closed loop hardware. Most of the complexity tends to be in the application layer anyway. And you can likely figure out how to intercept raw MAC packets destined for the network and feed them into a ethernet card on your PC, so that the whole embedded TCP/IP stack thinks it is talking to the internet.

[nctnico]

You need to seperate a few things here. For example: physics simulation of a BLDC motor is something you can test seperately in a simulated environment (including failure modes and extremes). Bottom line: test in seperate blocks and then integrate.

[tggzzz]

Yeah that would be test driven development where you write a test harness around a module of code that runs the module of code trough its paces as if it was running for real.

"My" approach doesn't have to be TDD based, although it could be. The first time I triumphantly (re)invented it was 40 years ado, and TDD didn't become a named technique for another quarter of a century.

If you want to attach a methodology label, Tony Hoare's CSP would be more appropriate. That is sufficiently good, both practically and theoretically, that if formed the basis of the commercially available hard real-time XMOS xCORE processors and xC. Yes, the IDE tells you whether your code meets deadlines; none of this "measure and hope you blundered into the worst case" crap

Emulating simple peripherals like for example a temperature sensor is really easy by replacing the init function with nothing and just returning some global variable when asked about temperature. Similarly GPIO driven LEDs and such can just be stored as a variable and displayed somewhere or printed to console or something.

But if you start having very complex relationships with your hardware this can become increasingly more difficult. Like for example if your MCU was also acting as a sensorless BLDC motor controller then you would have one hell of a time writing a test harness for it. Not only would you need to emulate peripherals like ADCs, Comparators, Timers etc..  but even emulate them with a lot of timing precision, you might even need to replicate interrupt latency or delays in code execution in interrupts. Then on top of all that you would also need to make a live a physics simulation of a BLDC motor....etc At some point emulating all of this becomes so hard that it makes more sense to just test that on real hardware.

Well, I used "my" technique for a life support machine. I think that consitutes "complex relationships with hardware".

As with all techniques, you have to use it with judgement and good taste and only in the right circumstances. For too many modern zealots, sensible techniques are elevated to being The One True (religious) Technique; yes XP/Agile/TDD, I'm looking at you.

I would not employ "my" techniques to emulate the physics of a sensor or controller. For a hypothetical controller where the current is continually measured to ensure it isn't too high, the split I would make would be:

• split the processing into several FSMs, each with their own FIFO
• interrupt causes instantaneous current to be measured, and event containing the value is pushed into a FIFO
• small peripheral FSM continually sucks those events from the FIFO, and calculates whether the current is too high for too long
• when it is too high, that small FSM creates an "overcurrent" event and puts that in the main FSM's FIFO
• the main FSM picks up the overcurrent event and takes the appropriate "high level" action

You can then test the hardware and two FSMs independently to verify:

• hardware inputs cause the correct instantaneous current reading event
• sequence of instantaneous currents do create the appropriate overcurrent event
• overcurrent event does cause graceful shutdown as per specification etc

[nctnico]

The hard part of modern day software development is to design the skeleton of the software (data structures, object relations and flow) in a way that it allows the specifications to change afterwards. Top-down was great in the past but nowadays products are rolled out in phases where user input keeps refining/altering the specifications. I used to have a co-worker who had to rewrite whole lots of code due to a relatively simple specification change.

[tggzzz]

The hard part of modern day software development is to design the skeleton of the software (data structures, object relations and flow) in a way that it allows the specifications to change afterwards. Top-down was great in the past but nowadays products are rolled out in phases where user input keeps refining/altering the specifications. I used to have a co-worker who had to rewrite whole lots of code due to a relatively simple specification change.

That requires experience, good taste, and clairvoyance. All are in short supply

[YTusername]

I recently read the book of Jacob Beningo called " Embedded Software Design A Practical Approach to Architecture, Processes, and Coding Techniques" (TOC). It is an amazing book with plenty of advices and tips for modern firmware and embedded systems development. The book starts with architecting, then implementing, and going through the DevOps and CI/CD.

- More generally, I've found that in the embedded world there are fewer tools for managing projects with systems like package managers, maybe more equivalent to what exists in the js world.

It is impossible to have such package manager idea for many reasons especially that firmware is not standardized like operating systems.

[thermistor-guy]

I'm not sure if there's a question in all of that but there's no substitute for having real hardware for testing...

You can do a lot of algorithmic development, and coding, without the real h/w. Ultimately, yes, you need the real h/w.
I have no experience in IoT. A few examples from telecoms:

1. Two processors on a board communicated via a FIFO chip. Every now and again, the FIFO would misbehave, under high-load
conditions. Logic analyzer logs showed nothing wrong. Software review showed nothing wrong. The chip vendor's technical
support had no suggestions. With deadlines approaching, we decided to implement a light-overhead error detecting/correcting
protocol over the FIFO channel. It worked fine in extended full-load tests, so we went into production with it. We never solved
the FIFO problem, we just coped with it.

2. Code developed for a processor board worked fine. We used a slower "pin-compatible" processor on a
cost-reduced version of the board. On the cost-reduced version, we couldn't do remote firmware download and upgrade
reliably, a crucial feature. This feature had problems because the slower processor's on-chip Flash memory had much slower
write timing - something missed in the s/w reviews. Easily fixed, once discovered.

3. A product had to be powered by a telephone line. It had to use less than 75 uA. It had to fit in a small left-over space in
another vendor's housing. With minimal code running, the board was using twice that current limit, placing the
project in jeopardy. It turned out that unused I/O pins on the processor were floating, and that was causing excessive
current drain. The solution was to initialize all unused I/O to outputs. Easily fixed, once discovered.

The h/w doesn't always work the way you think. If you find out early, it's fixable. How late in development
can you afford to find out?

[nctnico]

Hardware which misbehaves is not a software development problem. It is a hardware problem which should be caught during hardware verification which can run in parallel with software development. After all, it is known how the hardware should behave.

[tggzzz]

Hardware which misbehaves is not a software development problem. It is a hardware problem which should be caught during hardware verification which can run in parallel with software development. After all, it is known how the hardware should behave.

Putting any two subsystems together (hardware, software, mechanical, wetware, etc) often brings integration problems to light. The causes can be bugs, misunderstandings, etc, etc. What subsequently changes depends on many factors. In the building trade this is called snagging, and on large projects it causes games of "pass the parcel" or "musical chairs", and it is traditional that the business left holding the parcel stops goes bust.

It is often easier to fix misbehaving hardware by fiddling with the software

[SiliconWizard]

Yes, in any team larger than 2 people (in worst cases, you can make that 1), there's bound to be misunderstandings and parcel passing as a consequence. And yes, at that point, the hardware usually "wins", as it's thought to be more inflexible and costlier to fix (which isn't always the case btw, as software projects being delayed can increase costs by an enormous amount, and it's unfortunately still way too often underestimated.)

A lot of this can be avoided or at least alleviated significantly by properly documenting interfaces, including *of course*, interfaces on the hardware level. But technical documentation is hard and annoying, and no one wants to do it unless forced to.

[tggzzz]

Yes, in any team larger than 2 people (in worst cases, you can make that 1), there's bound to be misunderstandings and parcel passing as a consequence. And yes, at that point, the hardware usually "wins", as it's thought to be more inflexible and costlier to fix (which isn't always the case btw, as software projects being delayed can increase costs by an enormous amount, and it's unfortunately still way too often underestimated.)

A lot of this can be avoided or at least alleviated significantly by properly documenting interfaces, including *of course*, interfaces on the hardware level. But technical documentation is hard and annoying, and no one wants to do it unless forced to.

Having a good and properly documented interface is absolutely necessary.

• it gives each party something to "work towards", and mock out the other side
• it is a good logging and instrumtentation point, allowing fingers to be pointed during problems in integration, operation, and maintenance

I've used the latter to avoid disputes involving lawyers: the other party simply admitted their "issues"