Using a microcontroller and FreeRTOS for a data logger

[Phil1977]

I know the problem: A lot of sensors, from my experience especially these with UART-interfaces, have sometimes unexplainable timing issues and then it´s often helps to have one controller with a low-level-microcode (like an Arduino) to talk to them, collect the data in the RAM and send it via I2C to the logging system. I can completely understand the setup you have chosen for the engine-logger.

Anyhow, there are several ways to do it better:
- Use asynchronous buffer functions of the uC whereever possible. Think about asynchronous buffer ICs if you need more ports
- Combine as many sensor readouts to one uC as it manages to read out the sensors serially. E.g. analogue signals are more or less gathered instantaneously, you can read out as many as the uC has ports. Some I2C readouts only take ms, some UART readouts can take hundreds of ms.
- Use a buffer for writing or sending the logged data. Arduino with its small RAM is not very well suited for this, RP2040 or ESP32 are better.
- Create timestamps at the first acquisition level
- Think about some shared medium data transfer as CAN/ethernet if the logging system gets bigger.

My list is definitely not complete and maybe outdated. I never professionally "learned" these things, I just write out of many years of hands-on-experience with various logging systems.

[pcprogrammer]

On the broader question, having shown that my data logger should be implemented differently, what sort of application would be suitable for an RTOS?

When you have a system that has different control loops that can be time shared, an RTOS makes it simpler to implement this.

For instance a watering and feeding system for growing plants. You can have multiple PID controls running, that basically use the same code with different parameters.

A real time operating system can be a very simple task switcher and not provide other support, but it can also be a full support operating system like Linux where timing is better guaranteed then on a bog standard Linux. Or something in between depending on your needs.

[brucehoult]

On the broader question, having shown that my data logger should be implemented differently, what sort of application would be suitable for an RTOS?

"RTOS" is a very grandiose term for what can be quite a simple thing.

It's not so much the application, it's how lazy you want to be in not having to decompose and invert the logic of your application to a main loop, polling, and a set of interrupt handlers and interleave the logic of all the different processes and protocols together into one mess.
 

[peter-h]

An RTOS is really brilliant. You can write loads of tasks and write each one as if it owns the machine.

This greatly simplifies the structure of the whole thing.

Obviously it brings lots of gotchas e.g. if you just wait in a loop then (depending on priorities) you will hang the machine until the next tick, which slows things down, so you need to keep it co-operative by instead calling e.g. osDelay(1) or some such. And if you have tasks passing messages to each other then you can have a whole load of fun too, with deadlocks and such. But a lot of RTOS products don't do much of that, or do it in a simple way.

Most products have code modules which interact little and when they do, via clear variables e.g. a user interface (keypad+LCD?) and some code which does real work. Perfect for an RTOS.

And once you develop a basic RTOS based platform then you can re-use it for any number of entirely different products. I've done just that with my 32F417 based one. It also has ETH and USB but that's another topic (a huge amount of work). Very easy to spin off custom products. Before RTOS, every new product (I've designed dozens of uC based boxes since 1980) was mostly starting again.

One still uses interrupts from things like UARTs even if there is an RTOS there.

[NorthGuy]

Having said that, measuring the fuel injection pulses to 100us seems like it might be hard while doing everything else as well.

No. You can use interrupts - when there's an edge, normal execution stops, you record the timing of the edge and then return back to what you were doing.

Moreover, MCUs have lots of different periphery modules. You may be able to delegate lots of functions to them - they may record the timing of the edges for you. To get a pulse width, all you need to do might be to subtract two values. When you select your MCU, you start from looking at the list of periphery and plan how you can use them to your benefit. In the end, the CPU may do very little - most of the job is done by periphery.

[peter-h]

Yes. With clever use of timers one can even get pulse width measurement done for you. If one is cunning one can even use DMA to copy timer values into RAM...

[SteveThackery]

I just want to say a big "thank you" to everyone who has contributed. I've learned loads and got some new insights into how to approach design challenges like this.

Thank you again - I really appreciate it.

[zilp]

As it seems noone has said it explicitly: The hardware solution for measuring pulses is something called the capture function of a timer, which will save the current counter value of the timer to a register on a signal edge, which you then can read out.

You might want to just take the datasheet of some microcontroller and read it to learn what kind of stuff the hardware is capable of. In the case of the "arduino AVRs", the 16 bit timer has a capture unit, for example, but you'll find similar things in most microcontrollers. But if you can accept a bit more measurement jitter, using interrupt to capture the timer value can work just fine as well.

[SteveThackery]

Thanks, @zilp. You are right - I don't know enough about the features already available in the hardware. This sort of stuff is pretty much completely hidden when you are messing around with Arduinos.

I have implemented a type of multi-tasking in Arduino by creating a number of 'C' functions - one for each task - and implementing them as state machines. Each is designed to do one simple thing (eg check the status of an I/O line, or a timer), switch to the next state, and return to the caller as quickly as possible.  The Arduino loop() then comprises little more than a list of calls to each function, so they are all called in turn, repeatedly, at maximum speed.

The secret to success is to keep the dwell time in each function as short as possible. I suppose it's a crude form of co-operative multi-tasking.  It is from this place that I began wondering about doing multitasking properly, with something like FreeRTOS.

I am pleased to learn that my data logger happens not to be a good project to implement on an RTOS. Combining the code from five Arduinos onto one, decently powerful, microcontroller will be an interesting task in itself, albeit not one I can use to learn about FreeRTOS.

[NorthGuy]

The secret to success is to keep the dwell time in each function as short as possible. I suppose it's a crude form of co-operative multi-tasking.

Typically, the things you do in the main loop won't be time critical, so you don't worry much about the timing. If you have something time critical, you would rather create an interrupt for it.

It's like in real life. If you need to give a medicine to your mother at exact time, you don't accomplish this by making all your tasks throughout the day very short. You accomplish this by setting an alarm clock and abandoning your routine tasks when the alarm clock rings.

[gpr]

I am pleased to learn that my data logger happens not to be a good project to implement on an RTOS. Combining the code from five Arduinos onto one, decently powerful, microcontroller will be an interesting task in itself, albeit not one I can use to learn about FreeRTOS.

I'm not sure how you've come to that conclusion.
Of course there is a learning curve if you've never used something like FreeRTOS but already know Arduino, this is a valid point.
But I don't see technical reasons to dismiss RTOS for this project.

[shtirka]

I am pleased to learn that my data logger happens not to be a good project to implement on an RTOS. Combining the code from five Arduinos onto one, decently powerful, microcontroller will be an interesting task in itself, albeit not one I can use to learn about FreeRTOS.

I'm not sure how you've come to that conclusion.
Of course there is a learning curve if you've never used something like FreeRTOS but already know Arduino, this is a valid point.
But I don't see technical reasons to dismiss RTOS for this project.

Even if there is no technical reasons to dismiss an RTOS, it doesnt mean you should stick it into all sorts of places: with OP's requirements on timing and limited functionality, an RTOS will do more damage than good

Ilya

[peter-h]

I wrote my own simple RTOS c. 1989 when I got deeply into a complicated Z280 based product. It was actually working without an RTOS (a main loop plus a 100Hz timer int plus comms ints plus some generous FIFOs built with an ASIC) but by the time I got around networking these boxes together (with a custom token-ring LAN I designed, Z180/85C30, 1553 PHY, etc - this was almost before ETH came in) it just got unworkable.

Since then I've used an RTOS for everything nontrivial. My current project (last 5 years maybe, arm32) is under FreeRTOS which works great and actually of all the external stuff in this box (LWIP, MbedTLS, ST ETH, ST USB, etc) it has been the simplest thing and the biggest productivity enhancer.

And one tries to re-use expertise, not design everything from scratch. If I now had to design some fancy digital clock I would just chop down my PCB and use that for it.

[gpr]

I am pleased to learn that my data logger happens not to be a good project to implement on an RTOS. Combining the code from five Arduinos onto one, decently powerful, microcontroller will be an interesting task in itself, albeit not one I can use to learn about FreeRTOS.

I'm not sure how you've come to that conclusion.
Of course there is a learning curve if you've never used something like FreeRTOS but already know Arduino, this is a valid point.
But I don't see technical reasons to dismiss RTOS for this project.

Even if there is no technical reasons to dismiss an RTOS, it doesnt mean you should stick it into all sorts of places: with OP's requirements on timing and limited functionality, an RTOS will do more damage than good

It has been an interesting discussion with a few good suggestions and clarifications from people. And eventually OP concludes that this is not a good project for RTOS, which was a bit unexpected. I'm just wondering what was the logic behind that conclusion and what damage RTOS can do here. Probably I missed something important in the thread.

[SteveThackery]

It has been an interesting discussion with a few good suggestions and clarifications from people. And eventually OP concludes that this is not a good project for RTOS, which was a bit unexpected. I'm just wondering what was the logic behind that conclusion and what damage RTOS can do here. Probably I missed something important in the thread.

As a newbie, it is perfectly possible that I have misunderstood the advice. If so, I'll change my mind again.

The messages I'm picking up suggest that a stupidly fast task switching clock tick would be required to time event durations in software to a resolution of 100us. On the other hand, many micros have sophisticated timer features built in which could do a similar job with only a trivial software overhead.

The other message I'm getting is that the time-critical event handling would be much more subject to an interrupt-based architecture.

As I say, I might have misunderstood the comments in this thread, and I remain totally open to mew lessons and changing my mind where appropriate. Please - if you feel I've come to the wrong conclusion, please say so. I'd be delighted to continue this thread.

[Phil1977]

As a newbie, it is perfectly possible that I have misunderstood the advice. If so, I'll change my mind again.

The messages I'm picking up suggest that a stupidly fast task switching clock tick would be required to time event durations in software to a resolution of 100us. On the other hand, many micros have sophisticated timer features built in which could do a similar job with only a trivial software overhead.

Especially with a multitasking OS you should use the hardware timers and buffers where ever possible. And if you do so you can easily get good resolution.

[dietert1]

Arduino is a little boring, isn't it?
Nowadays we have very powerful MCUs for little money that easily run open source packages like FreeRTOS, LwIP or a USB stack. Every embedded developer should use those packages once in his/her life. One needs to take a closer look in order to know what those tools are and how to use them. FreeRTOS isn't just task switching. It is task switching based on queues, semaphores and timers. It works very well with interrupts and depending on what MCU you have, 100 usec may be 100 000 cycles, no problem at all.
I am also integrating MbedTLS into a project. That STM32 MCU has a true random number generator based on analog hardware. No more pseudo random numbers!

Regards, Dieter

[SteveThackery]

FreeRTOS isn't just task switching. It is task switching based on queues, semaphores and timers. It works very well with interrupts and depending on what MCU you have, 100 usec may be 100 000 cycles, no problem at all.

I am keen to learn how to use an RTOS, and FreeRTOS does seem very suitable for microcontrollers.

I am also integrating MbedTLS into a project. That STM32 MCU has a true random number generator based on analog hardware. No more pseudo random numbers!

My first adventure into an RTOS was the mBed environment, but I soon realised all was not well. The development UI was pretty rubbish, and I was soon asking on the forums about whether it was abandon-ware. Lo and behold, my fears came true, and now it really is officially abandon-ware (sorry, "deprecated").

[gpr]

As a newbie, it is perfectly possible that I have misunderstood the advice. If so, I'll change my mind again.

The messages I'm picking up suggest that a stupidly fast task switching clock tick would be required to time event durations in software to a resolution of 100us. On the other hand, many micros have sophisticated timer features built in which could do a similar job with only a trivial software overhead.

You can measure duration with hardware timer, either purely in hardware which will just give you pulse width in some register, or with some help from interrupts, when interrupts can be triggered on pulse edges for example. I doubt you can find some MCU nowdays without any support for this.
Then why bother about tick clock? It is just a frequency how often scheduler decides which task to run next. Set it to 100Hz and forget about it, until you have a good reason to change it.

The other message I'm getting is that the time-critical event handling would be much more subject to an interrupt-based architecture.

It depends what alternative you compare with interrupt-based architecture. Probably you mean polling, when you check some condition periodically with some delay, or even without any delay in a busy loop. Both approaches are valid, interrupts are generally have better latency. But interrupts don't come from nowhere, something should trigger it, for example:

• sensor toggles dedicated pin, interrupt is triggered by GPIO (pin peripheral), you know sensor has something to say, you read data from sensor
• i2c peripheral (for example) detects i2c activity, triggers interrupt, you read the data from sensor via i2c
• DMA controller (configured beforehand for peripheral, i2c for example) triggers an interrupt, saying "I've completed reading data from i2c, data is ready in the memory you configured for it"

About FreeRTOS (and probably any other embedded RTOS), there is a common misunderstanding that it limits what you can do in you code. No, if you use FreeRTOS doesn't mean you cannot use interrups or anything else. You still have unlimited full access to all your hardware, maybe except of a couple of system interrupts or a timer used by rtos for its own purposes. You can even continue enjoying your favorite state machine loop running in a separate task, concurrently with other tasks you might have. In that sense FreeRTOS adds you a possiblity to use multitasking with inter-task communication without any big disadvantages.
If you are working on something hard realtime with tight timing requirements (not in this case), there can be some subtleties. But otherwise FreeRTOS is a very good tool to have in your toolbox.

As you can see, this arduino project can be transformed to something really cool where you can learn a lot.

[shtirka]

I am pleased to learn that my data logger happens not to be a good project to implement on an RTOS. Combining the code from five Arduinos onto one, decently powerful, microcontroller will be an interesting task in itself, albeit not one I can use to learn about FreeRTOS.

I'm not sure how you've come to that conclusion.
Of course there is a learning curve if you've never used something like FreeRTOS but already know Arduino, this is a valid point.
But I don't see technical reasons to dismiss RTOS for this project.

Even if there is no technical reasons to dismiss an RTOS, it doesnt mean you should stick it into all sorts of places: with OP's requirements on timing and limited functionality, an RTOS will do more damage than good

It has been an interesting discussion with a few good suggestions and clarifications from people. And eventually OP concludes that this is not a good project for RTOS, which was a bit unexpected. I'm just wondering what was the logic behind that conclusion and what damage RTOS can do here. Probably I missed something important in the thread.

The big difficulty with an RTOS in this context is that it would only primarily be used for a relatively simple task of logging - which not only defeats the whole point of an RTOS (usually it is used for use when more than one "logical" (to us humans) task needs doing, and also the effective throughput may well be well be well below the effort necessary to have an rtos in place for the task of logging in this context), it also adds quite a bit of complexity to an application (which in this case would defeat the point of an rtos simply for the purpose of making a smart logger). RTOS would make sense in a case when you add additional requitements to the logging application or if you restrict the timing or the effective throughput, but as it stands an RTOS would scare the heck out of the OP to begin with - youre trying to REDUCE complexity of an application, not add to it lol

Ilya

[gpr]

FreeRTOS isn't just task switching. It is task switching based on queues, semaphores and timers. It works very well with interrupts and depending on what MCU you have, 100 usec may be 100 000 cycles, no problem at all.

I am keen to learn how to use an RTOS, and FreeRTOS does seem very suitable for microcontrollers.

I am also integrating MbedTLS into a project. That STM32 MCU has a true random number generator based on analog hardware. No more pseudo random numbers!

My first adventure into an RTOS was the mBed environment, but I soon realised all was not well. The development UI was pretty rubbish, and I was soon asking on the forums about whether it was abandon-ware. Lo and behold, my fears came true, and now it really is officially abandon-ware (sorry, "deprecated").

Alright, I see a bit of difficulty with FreeRTOS here. mBed has harwdare abstraction, you can use it without any hardware knowledge. Simply put, it has "I2C" class which you use, pass some parameters, and it just works. FreeRTOS doesn't have any built-in peripheral support (unless you use some third-party libraries, or FreeRTOS-based SDK from espressif for example), it is just a scheduler which switches tasks and allows them to communicate with each other.
Personally, I prefer to use hardware directly without any libraries, maybe just CMSIS headers to use C structs instead of hardware addresses. This might sound complicated, but once you read datasheet and understand how some periprheral works, I find it much easier than HAL library.

There are other choices if you want built-in peripheral support: espressif SDK, zephyr (which I don't reccomend as a first choice though), a bunch of python-based enviroments. Other people can give you better advice here.

[peter-h]

mBed has harwdare abstraction, you can use it without any hardware knowledge. Simply put, it has "I2C" class which you use, pass some parameters, and it just works

That will seriously bite somebody in the bum one day.

What do they do about /CS timing for example?

Also, RTOS support for hardware devices is obviously tricky. Hardware is obviously not thread-safe So how do you do that? One has to mutex each device, and then you (depending on detail) need to re-issue initialisation to it if the current one is for a different process, etc. I've done all that and it does work but there is a lot to know about what goes on behind the scenes.

[Siwastaja]

Most I2C sensors decouple the actual sampling from reading it out, anyway - either by implementing a FIFO which can hold many samples, or at very least buffering one sample (so fixed FIFO length of 1). This means that exact timing is non-critical as long as you read out the sample before the next one arrives*, which is pretty easy if sample rate is low.

*) if that next sample is relevant. If you just want data at say 10Hz and sensor samples internally at 100Hz, it's usually not a problem to let the data register overrun and lose 90% of the samples, although aliasing might be an issue if input values are changing faster than your final sample rate.

Same is true for sampling analog signals - triggering ADC sampling can be done accurately and jitter-free e.g. by a timer, so that readout timing is irrelevant (as long as it's before the next sample).

DMA, if available, extends this even further, so that you don't need to process between every two samples, but you can collect many samples in memory automagically for later batch processing. Although, even if you could, it's another question if you should; on microcontrollers, timing is usually easy anyway, so if you just guarantee doing all the work between two samples, then job done, no need to think about how to process them in batch. Again, like in real life: if you work at a box factory assembling cardboard boxes that come in on a conveyor, it's so much easier if you are quick and dedicated enough to finish each before the next one arrives. Failing that, you can collect them, but that brings new management tasks and requires area to store half-finished boxes; and eventually the conveyor needs to stop so that you can finish your work. And microcontrollers do not need coffee breaks, so simple "do it fully once it arrives" works very well.

But pure datalogger is actually one of the examples where DMA and batch processing probably does make sense, and life easier - significantly decoupling sample generation from storage. This is useful because on datalogger, latency usually is irrelevant - data is looked at hours, not nanoseconds, later. DMA into memory allows leeway for stuff like SD card / filesystem latency / timing uncertainties.

Lacking DMA, you basically do the same by reading a single value in an interrupt handler, writing it in a memory buffer. At trivially slow data rates, this isn't a problem.

And RTOS solves none of this. If you use RTOS, you have exact same choices to make: whether to use interrupts for data collection, or configure DMA for the same job and then manage the DMA in interrupts.

With an OS, you have a third option of creating abstract concept of "threads" that all busy loop looking at the inputs and let the scheduler to swap between these threads, but it's highly inefficient way to do it, and probably not even useful from code clarity viewpoint. I suggest event-driven programming paradigm, and it maps pretty well on hardware interrupts, especially on modern microcontrollers that implement pre-emptive (meaning: interrupts can interrupt other interrupts in priority order) interrupt controllers on hardware. If you choose this paradigm, then you end up using very few of RTOS features. Maybe some FIFOs / locking primitives offered by OS. Which you can do without the OS pretty easily.