Question: Design descision, when to use an embedded OS or not.

[HackedFridgeMagnet]

Hi
This question is sort of directed at Ntnico, but anyone is welcome to answer it. I'm asking ntnico because you seem to have strong opinions and experience in embedded firmware design and I mostly seem to agree with you, with the exception that I really like being able to remotely debug firmware.

In a current project of mine I am wondering whether to go with RTOS or just use a main loop and run Real time code within the interrupts.

Currently using C and would like to stay under 32k firmware if possible. The only reason is I am using the free version of Atollic. Though I am comfortable with eclipse so I could go to a larger codebase if necessary, but I seem to have a real mental block when it comes to doing my own makefiles.
I wouldn't mind moving to C++ either so I could use classes and <STL>. I am fairly sure that if I stick with RTOS I will have to move to Eclipse.

It is a fairly basic system, Stm32F1 based
I would like:
at least 1000/ samples per second from an SPI a2d
writing to an SD card once a minute
usb read write very occassionally
serial comms to a remote display approximately once per second
Leds/GPIO
RTC
possibly a local multiline display.

I guess want I want most out of the OS is the multithreading and the file system.

Any thoughts?

[IanB]

Just dipping in here, let me throw out some reasons I can think of for using an operating system:

1. Providing supervisory functions: if your program dies for any reason there is a reset/recovery/clean-up option available
2. Providing system services: file system, network access, I/O sub-systems, display drivers, etc.
3. Speeding development: you need to spend less time on system level functions and can focus on the application code.

I'm sure these are not complete or exhaustive, but they are some of what first comes to mind. So what you probably have is a trade-off that varies from application to application. The more "system-ey" your application, the more likely you are to benefit from hosting it in an OS instead of running it on the bare metal. I'm tempted to say that only you can decide.

There is probably a gray area in the middle where you add all sorts of standard libraries to your project to provide pre-packaged services. For instance, is Arduino an OS, or an application development/hosting environment?

[c4757p]

For instance, is Arduino an OS, or an application development/hosting environment?

I'm sure somebody disagrees, but I think it solidly qualifies as a (very lightweight) OS. It has a proper bootloader, standard libraries, and it manages system resources like memory, timers, interrupts, etc. You can develop entire programs on it without knowing a thing about the underlying hardware - most Arduino software could be ported to almost any other micro without touching the software itself.

[nctnico]

Oh oh... what did I do to deserve this... 

When it comes to an OS I think about multi-threading and time slicing which can make life very hard. It may seem nice to divide tasks over seperate threads (processes) but exchanging data between them can cause big headaches with messaging systems and semaphores. Even when writing PC applications I try to avoid using seperate threads.

In my microcontroller projects I use a scheduler. Every module (you can call it a process) has an init function which is called once at startup and a run function which is called from main. Each run function must exit as soon as possible (so no while(wait for a pin)...). This scheme works excellent until you have to do a very long calculation which causes the system to stall. Waiting for I/O can be solved by using a statemachine which tests for a condition.

I recently did a similar project. I used a timer interrupt to send a byte over SPI 10000 times per second. The rest (serial I/O, display handling, user I/O, dealing with an SD card) is done from the scheduled 'processes'. The only thing to watch out for is to make sure an USB interrupt isn't taking too long; it may mess up the timing for the ADC. If the ADC and controller allow it you could use an I2S serial bus which runs constantly so the ADC conversions are always started at the same time. It only may take longer before the data is read but most some serial interfaces have FIFO and DMA capability.

[Rufus]

In my microcontroller projects I use a scheduler. Every module (you can call it a process) has an init function which is called once at startup and a run function which is called from main. Each run function must exit as soon as possible (so no while(wait for a pin)...). This scheme works excellent until you have to do a very long calculation which causes the system to stall. Waiting for I/O can be solved by using a statemachine which tests for a condition.

While with a multitasking OS you 'exit' as soon as you wait for a message or timer or flag and you come back right where you left so you don't have to implement state machines to remember and get back to where you were. You don't need separate initialisation functions either. If you have some very long calculation the OS will interrupt it so you won't starve other higher priority tasks and will share the CPU between equal priority tasks.

Task switching has overhead but so does all your tasks continually checking if they have something to do.

There is noting wrong with cooperative multitasking, there is nothing wrong with using a multitasking OS both require jumping through some hoops to make it work.

Getting back to the thread title I would say the more complex the system (more tasks) the more attractive an OS becomes. The OS overhead becomes proportionally smaller while the effort required to make cooperative multitasking work tends to scale with the number of tasks.

I'm sure somebody disagrees, but I think it solidly qualifies as a (very lightweight) OS.

(it being Arduino) I say if it doesn't support multiple tasks it is a fancy runtime library not an OS.

[c4757p]

I'm sure somebody disagrees, but I think it solidly qualifies as a (very lightweight) OS.

(it being Arduino) I say if it doesn't support multiple tasks it is a fancy runtime library not an OS.

I see your point. I'd say that if you consider MS-DOS a "true operating system", then the Arduino software is perhaps just one small step away from an OS (the closest DOS had to multitasking was TSR, and I suspect that would be quite easy to implement on an AVR with enough memory). If MS-DOS is more of a proto-operating system, then Arduino may well just be a fancy library.

So I have to concede that it's not an operating system, but depending on how you define OS it could be very close.

[IanB]

Multitasking has not always been a defining feature of operating systems, it is rather something that developed in sophistication as operating systems evolved.

Consider, for example RT-11, which was always considered an operating system, in spite of primitive multi-tasking features.

[c4757p]

Still, if multitasking were a defining feature, I think a background task would be close enough to a parallel process to qualify for the sake of definition.

RT-11 systems did not support preemptive multitasking, but most versions could run multiple simultaneous applications. All variants of the monitors provided a background job.

[Bored@Work]

I guess want I want most out of the OS is the multithreading and the file system.

Any thoughts?

Sounds like you are confusing an OS with libraries. Yes, typically an OS comes with  (but need not) libraries to make use of the OS' API. But stand-alone libraries are also very common in embedded systems.

In fact, multithreading is a typical job of a library.Even on "real" operating systems, with a bit of help of the OS. The idea behind multithreading is to be lightweight, running the threads in a single process, not involving the OS having to do full process switches. The price to pay for this is that threads (within a process) aren't isolated against each other. E.g. while processes have a hard time to mess up each other's memory, threads (within one process) share the same memory and can therefore easily mess it up.

Same for file system. Often they are implemented as libraries in embedded systems, not requiring an OS. This is different to real operating systems, where the OS manages the system's resources, and has the file system(s) build in, typically in the form of drivers.

Looking at your requirements, my decisions would be:

* No OS.

* No multithreading (and of course no processes).

* Classic embedded application design with
     ** a main loop checking states and reacting on state changes
     ** interrupts with very short interrupt handlers, changing system state (including a main timer interrupt to control the various timed functions)

* No C++. Within 32k you can't make much use of the STL (check the size of a typical STL std::string implementation and you get an idea why)

* C, because highest chance to get the libraries you need.

* Library for the SD card and file system. Annoying to do from scratch. Main loop decides when it is time to write to/read from the card.

* Manual coding for the ADC, interrupt handler filling a buffer. No big thing. Main loop checks the buffer and does whatever calculations are necessary.

* Manual coding for the RTC, interrupt handler. No big thing. Main loop checks if (soft) timer(s) has/have expired and initiates required action(s).

* If available, library for driving the particular display. Can be a bit of annoying to do from scratch, depending on display controller type. Updated from the main loop.

* Absolutely necessary: Library for USB. USB is a bitch to do from scratch.

* If available, a simple library for doing buffered serial I/O. If non available not a big thing to do your own. Interrupt handler working from/writing to buffer.

* LEDs: No big thing. Manually coded, updated from the main loop.


Showstopper: No USB library available. To get one I would be willing to change from "no OS" to "OS" if that OS contains such a library. Similar, I would be willing to change from C to whatever language for which I could get a USB library for.

[HackedFridgeMagnet]

Sorry to put you on the spot ntnico. Voice an opinion and people may remember.

Thanks all for the very helpful advice.

Sounds like you are confusing an OS with libraries.

I can't refute this, this is true.

Although the 32k is not mandatory it is only to save me time switching to standard eclipse and gcc etc etc.
I am currently up to 24k with only a portion of the code written but using RTOS.
But I think I will change direction and give it a shot with the classic embedded design in C, and because I have very little string manipulation to do,and I should easily make the 32k.

I have been playing around with the STM32 libraries and the USB slave seems to work ok, so hopefully I can just use this.

I guess I had been leaning toward an OS, c++, and STL more due to my experience then their appropriateness.

[nctnico]

In my microcontroller projects I use a scheduler. Every module (you can call it a process) has an init function which is called once at startup and a run function which is called from main. Each run function must exit as soon as possible (so no while(wait for a pin)...). This scheme works excellent until you have to do a very long calculation which causes the system to stall. Waiting for I/O can be solved by using a statemachine which tests for a condition.

While with a multitasking OS you 'exit' as soon as you wait for a message or timer or flag and you come back right where you left so you don't have to implement state machines to remember and get back to where you were. You don't need separate initialisation functions either. If you have some very long calculation the OS will interrupt it so you won't starve other higher priority tasks and will share the CPU between equal priority tasks.

Task switching has overhead but so does all your tasks continually checking if they have something to do.

There is noting wrong with cooperative multitasking, there is nothing wrong with using a multitasking OS both require jumping through some hoops to make it work.

Getting back to the thread title I would say the more complex the system (more tasks) the more attractive an OS becomes. The OS overhead becomes proportionally smaller while the effort required to make cooperative multitasking work tends to scale with the number of tasks.

I think scheduling or task switching cause an equal amount of overhead. What I do think is something to consider when  using an OS and running the processes semi-parallel is that you need to share the stack between the processes. With scheduling each process has 100% of the stack to itself. With an OS you need to share so if you have 5 processes then each process can use 20% of the stack. RAM is often a scarse resource so it is nice to be able to live with a very small stack.

The number of high priority tasks is often very limited. If scheduling is too slow or too uncertain I use a (timer) interrupt to execute a parallel task. Most of my digital signal processing projects work that way: the entire signal processing algorithm is executed from an interrupt (it is the highest priority process after all) and the rest from scheduling. In some cases 90% of the CPU time is spend in the interrupt. Still the stack sharing limits apply...

Scheduling can take you very far. I have worked on projects which compile into a binary close to 1MB with about 30 to 50 processes and scheduling still did the job.

All in all I'm not against using an OS. An OS allows you to program as you like 'while(pin is not high)... ' but you still need to take the limited resources into account and learn how to get the most from the OS. Maybe that can be more complicated than writing your software in a way more suitable for scheduling.

Having said that I still want to look into FreeRtos for an upcoming project which has to deal with ethernet. Its probably easier to just spawn off a new process for a connection then keeping track of all the connection in one scheduled task. Or still use scheduling but allow a task to execute in a certain context (=allocate some memory for variables and buffers and pass the pointer to the init and run function).

[HackedFridgeMagnet]

Scheduling can take you very far. I have worked on projects which compile into a binary close to 1MB with about 30 to 50 processes and scheduling still did the job.

I'm was a bit confused here, ntnico when you say scheduling do you mean threads within an OS or timer interrupts or maybe your own task switching library?
Actually I found the answer in your first reply.

In my microcontroller projects I use a scheduler. Every module (you can call it a process) has an init function which is called once at startup and a run function which is called from main. Each run function must exit as soon as possible (so no while(wait for a pin)...). This scheme works excellent until you have to do a very long calculation which causes the system to stall. Waiting for I/O can be solved by using a statemachine which tests for a condition.

So they are all just executing in the main thread but must obey the rule of always return asap. So basically out of an STM32F1 com port which has no buffer, you have to send one char at a time and then return. Next time around you test for TX clear and if clear send another char.
That sounds like an effective and consistent methodology, I will give it a try, seems to be very similar to what Bored was suggesting.


One thing, I guess we should be clear about the difference between threads and separate processes. https://en.wikipedia.org/wiki/Thread_%28computing%29 The main difference being the shared data space.

[nctnico]

In my microcontroller projects I use a scheduler. Every module (you can call it a process) has an init function which is called once at startup and a run function which is called from main. Each run function must exit as soon as possible (so no while(wait for a pin)...). This scheme works excellent until you have to do a very long calculation which causes the system to stall. Waiting for I/O can be solved by using a statemachine which tests for a condition.

So they are all just executing in the main thread but must obey the rule of always return asap. So basically out of an STM32F1 com port which has no buffer, you have to send one char at a time and then return. Next time around you test for TX clear and if clear send another char.
That sounds like an effective and consistent methodology, I will give it a try, seems to be very similar to what Bored was suggesting.

I'd use some buffers and use interrupts to handle the incoming and outgoing characters. What to do when the TX buffer is full is an interesting question. In some cases I throw the data away, in some cases I wait for the buffer to get empty (and effectively halting everything) and something I just make the buffer larger. It depends on what is most important.

[madires]

Multitasking has not always been a defining feature of operating systems, it is rather something that developed in sophistication as operating systems evolved.

Consider, for example RT-11, which was always considered an operating system, in spite of primitive multi-tasking features.

Operating systems are an abstraction layer between hardware and software to manage ressources and simplify access to hardware. Multitasking is a possible feature of an operating system but not a must have. And it's a very old concept. Before we had Unix there were time sharing systems. An IT veteran told me about a time sharing system with 12KB RAM supporting several users at the same time in the 60s.

[free_electron]

One thing that is often overlooked in embedded systems is optimizing the data handling. For small functions that require lots of parameter passing it is better to use global variables than to pass them in the call.

Lets take a look at that serial port example. A function that tests for data to exist in a buffer , check if tx flag is clear and then throw the character into the uart and return.

Sub transmit (buffer,txpointer)
   If txpointer>0
     If txflag =0 then
          Txrrgister =buffer(txpointer)
          Txpointer--
      End if
   End if
End sub.

Every time this thing gets called two variables are pushed on the stack by the caller and popped off by the function. Even if there is nothing to be done( buffer empty or tx still busy ) the cpu wastes time moving data.

That is a waste of cpu cycles ! Use global variables.

Now, you may say , but what if i want to send data from another stream into the uart ? I could simply change the origin during the call. True, but, you have to think the opposite way... 'Buffer' belongs to the uart handler. The outer process needs to fill it.

You also need to know your cpu and compiler and what it can and cannot do.
I remember an occasion on some PIC processor c compiler, Where, if you called print('hello world') it ended up being coded as a bunch of moves to ram buffer first and then print function was called on the ram buffer. The built in print could only take a pointer from ram. Printing astring stored in required copying the string over to ram buffer first .... That's kinda dumb ...

[andyturk]

For small functions that require lots of parameter passing it is better to use global variables ...

Gah! No! Please!

Every time this thing gets called two variables are pushed on the stack by the caller and popped off by the function.

Not necessarily. Depending on your architecture (and your compiler), the first few arguments are passed in registers. Whether these get pushed onto the stack depends on both the caller and the callee, but simple cases like your example can probably be in-lined by a smart compiler.

Use global variables.

Sure, use globals when you really and truly need something at the global scope. But be aware that globals suffer from namespace collisions and pretty much guarantee that your code isn't re-entrant. Plus, if your compiler is smart and the architecture has enough registers, you may be even slowing things down by forcing data out to RAM.

PS. I generally worship at free_electron's feet.

[free_electron]

note that i said depending on compiler and cpu architecture. not all cpu's are register based ....

[nctnico]

One thing that is often overlooked in embedded systems is optimizing the data handling. For small functions that require lots of parameter passing it is better to use global variables than to pass them in the call.

Wich is why people use pointers to structs. Using global variables is not a good solution because you loose oversight on which variable gets used where. A struct keeps everything in a neatly wrapped package.

[madires]

One thing that is often overlooked in embedded systems is optimizing the data handling. For small functions that require lots of parameter passing it is better to use global variables than to pass them in the call.

Wich is why people use pointers to structs. Using global variables is not a good solution because you loose oversight on which variable gets used where. A struct keeps everything in a neatly wrapped package.

But there's a trade-off. If you pass the pointer to the struct into a function the program has to add the offset to the element in the struct to be able to access the element. No free beer :-)

[nctnico]

On most modern CPUs (like ARM) that doesn't matter because they support indexed addressing with a constant offset. Because the compiler knows where the data is, accessing ptr[offset] is a single instruction. When passing lots of parameters on the stack you'll see the compiler using the stackpointer as a pointer to an 'on-the-fly struct'.

[andyturk]

Which is why people use pointers to structs...

Which is why other people use C++. 

C++ objects give you an implicit struct pointer passed into every method without the syntactic overhead. It's still a parameter behind the scenes, but since it's very common, most compilers will lock it into a register if possible.

[HackedFridgeMagnet]

I'd use some buffers and use interrupts to handle the incoming and outgoing characters. What to do when the TX buffer is full is an interesting question. In some cases I throw the data away, in some cases I wait for the buffer to get empty (and effectively halting everything) and something I just make the buffer larger. It depends on what is most important.

Yes that sounds far better than my previous proposal.
I'll figure something out to manage the overflow of the different I/O buffers, but it will probably just throwing stuff away. Luckily this should be extremely rare anyway.

[madires]

On most modern CPUs (like ARM) that doesn't matter because they support indexed addressing with a constant offset. Because the compiler knows where the data is, accessing ptr[offset] is a single instruction. When passing lots of parameters on the stack you'll see the compiler using the stackpointer as a pointer to an 'on-the-fly struct'.

Some CPUs and MCUs 20+ years old support that too. We had the same MCU wars we got today already back then.

[LeviathanX]

Using an embedded (RT)Os shall depend on the requirements set to the software.

Surely one can make main-loop software till he dies, but sometimes it's necessary to think about task switching based systems too.

E.g. you have a safety relevant software with watchdogs, fixed cycle times and error handling. Communication interface is CAN.
Hard requirements are: 10ms software cycles, be sensitive to bus faults, use a watchdog which will reset the µC in case of cycle failure; you're only allowed to trigger the watchdog once per window time.

We assume the worst case:

CAN Bus-Off occurs, due to safety requirements the customer wants you to fully re-initialize the CAN controller in the MCU. Say you chose a Microchip PIC, these guys can take up to 160ms for a full re-initialization!

Case 1:  You programmed the firmware as a main-loop variant.
You will hang up the entire software cycle and the watchdog will trigger for sure. Your MCU will be reset, internal diagnostics will write "watchdog reset" among other error codes into the EEPROM. Even worse: if bus off remains and the start up was not done correctly this might repeat endlessly until the bus off is cleared....
=> Customer in panic, software team in panic. 

Case 2: You programmed the firmware task based.
The CAN driver will start a re-initialization, the scheduler puts the CAN driver to background, rest of the software runs like normal, but being aware that the CAN bus went offline. No reset, safe state reached. Everyone happy...

These are non-fictional scenarios! I happened to see a lot of stuff during my career....

You can get at least one obvious conclusion out of this: if at least ONE module hangs in a main-loop program, your entire programm either will be off (by time) or hanging completely.

Though, task based systems are not free of disadvantages either. You need to plan in additional resources for your scheduler, the heartbeat timer interrupt (that is basically what re-arranges all your tasks execution order) and eventually some more function calls compared to your main-loop variant to get the tasks interact with each other.

Writing a task based OS from scratch is not much of black magic. If you know how to save a context with your MCU half of the way is done. E.g. get the OSEK/VDX specification (it's free), read the requirements and after three days you should've the system up and running. 

The fun stuff comes after you got it up and running: FINE TUNING. 

First you need to figure out the ideal heart beat. If your scheduler gets called way too often nothing will ever finish (or worse: on some MCUs it can disturb internal peripherals), if it's too slow your whole timings will be off and the system becomes far from being a "real-time" os.
Second is your scheduling algorithm. It need to fit your needs, but also need to be as small as possible. You can't necessarly call an algorithm each 100uS which takes up more than 100uS. You would leave the interrupt routine and immediately will be in it again -> system is dead.
Thus it is necessary to know your MCUs latencies for interrupts and general code execution (clocking speed is very important her, i.e. to calculate the time for the routine if you don't have an oscilloscope at hand)

As for the discussion about global variables... It should be avoid where possible. It leads to pretty bad spaghetti code and maintenance can be a nightmare. (Fun fact: I had to find a bug in an unknown code. A lot of global vars and what kind of unspeakable stuff. In the end I re-arranged the code and DELETED approx 3000 Lines (!!!) and it worked as before and even better! Though I was not allowed to commit the code "changes were too huge"  )

But, you can't avoid them if you use a configurable tasked based OS for example. You need to define your Alarms, Events, Task anywhere after all.. If you use Matlab for code generation, well it almost solely uses global variables throught the code. Though you don't need to maintain the resulting C-code here! Anyhow that's a matter you always have to discuss with customers... Most write "No external variables shall exists" in their requirements... but on the other hand they want you to develop a high reliable software and in case of a configurable task based OS you need to use external variables.

As for the buffer and parameter discussions, usually you should write your code in modules. Even in ANSI-C you can come through with the getter/setter approach; more over it's recommended.

So instead of

Uart.h

Code: [Select]

extern  uint8   buffer[];

Uart.c

Code: [Select]

uint8 buffer[MAX_BUFF_ELEM] = { 0 };

you should better do something like

Code: [Select]

extern FUNC( void, OS_CODE)   Uart_ReadBuffer(
                                           VAR(uint16, AUTOMATIC) offset,
                                           VAR(uint16, AUTOMATIC ) length,
                                          P2VAR(uint8, AUTOMATIC, AUTOMATIC ) PtrDataBuffer
                                       );

extern FUNC( void, OS_CODE)   Uart_WriteBuffer(
                                           VAR(uint16, AUTOMATIC ) length,
                                           P2CONST(uint8, AUTOMATIC, AUTOMATIC ) PtrDataBuffer
                                       );


Uart.c

Code: [Select]


typedef struct S_UART {
   uint8 buffer[MAX_BUF_ELEM];
} UartType;

_STATIC_ UartType Uart;

FUNC( void, OS_CODE)   Uart_ReadBuffer(
                                           VAR(uint16, AUTOMATIC) offset,
                                           VAR(uint16, AUTOMATIC ) length,
                                          P2VAR(uint8, AUTOMATIC, AUTOMATIC ) PtrDataBuffer
                                       ) {

   /* write internal buffer to given external pointer here */
   VAR( uint8, AUTOMATIC ) iter_i;
   VAR( uint8, AUTOMATIC ) iter_j;

   iter_j = 0;

  for (iter_i = offset; (iter_i < (offset+length)) && (iter_i < MAX_BUF_ELEM); ++iter_i ) {
     PtrDataBuffer[iter_j] = Uart.buffer[iter_i];
     ++iter_j;
  }
}

FUNC( void, OS_CODE)   Uart_WriteBuffer(
                                           VAR(uint16, AUTOMATIC ) length,
                                           P2CONST(uint8, AUTOMATIC, AUTOMATIC ) PtrDataBuffer
                                       ) {
   /* fill internal buffer here */
}
(Don't get confused, I'm using the AutoSAR compiler abstraction annotation)

You could even go further and do not use a buffer at all! Just using interrupts and pass the call through the layers.

Say you have the UART driver as lowest layer, an interrupt comes in, it calls the UART-Interface which is above the UART driver. this UART-Interface passes the lower layer HW-buffer forth to another layer or to the final destination. So you might have some call stacks more in the set up (initiated by the interrupt routine), but you save the RAM for the buffer, and the data gets "injected directly" into the destination module; therefore you don't need polling anymore. Read the AutoSAR CAN communication layer specification if you want that principle in finer details   (In case of UART, you might look in the AutoSAR SPI specification too, there's another buffer principle better suited for serial communications (but more complex)).

Anyhow... big  ...and I seem to have drifted away from the main topic?!

[HackedFridgeMagnet]

Though I was not allowed to commit the code "changes were too huge"

Commit first and say sorry later.
I work on the principle if I don't understand some bit of code, then it is no good to me, to logical conclusion is to get rid of it. I think on one project once I got rid of 6000 lines, a lot of it to do with multilingual support, it took a few days, and it still had multilingual support.

I am sure your AutoSAR macros would be good, especially in an environment where you have to prove safety but I am a bit wary of macros as they can also make simple things more confusing than they need to be.

But thanks for the post, a lot of interesting ideas, though I'll not be writing a Task switching OS anytime soon.