The Imperium programming language - IPL

[DiTBho]

I have a feeling this language will suck more than what (C?) is going to replace 

[Kalvin]

I have a feeling this language will suck more than what (C?) is going to replace 

It is very hard to create a new, useful programming language which would gain a wide user base.

As we have now over 70 years of experience in different kind of programming languages, the author should really take a deep look into the most popular languages used in embedded software design, and extract the best features from those languages into the new language, while avoiding pitfalls in the existing languages as much as possible.

This discussion thread has some very good set of features that could be provided by the new language.

Since implementing a programming language is a big task, the author might want to introduce the new language in smaller steps, starting from a simple core language, and implementing more advanced features in small steps. However, in order to avoid duct-tape syntax and grammar blunders, the author should have a fairly good and mature plan for how to implement the advanced syntax and grammar features as a whole in the future language releases.

[DiTBho]

This discussion thread has some very good set of features that could be provided by the new language.

Seen nothing good, too much nonsense repeated over and over again, and the latest*hit seems to be written by someone who has never planned anything serious in his professional life.

Good show

[DiTBho]

the author should really take a deep look into

Frankly, the author should really get a job. I mean programming real stuff, from home automation to avionics. Only then can he come back here and talk. Otherwise, sugar crap is and will be the only main core of everything.

[SiliconWizard]

the author should really take a deep look into

Frankly, the author should really get a job. I mean programming real stuff, from home automation to avionics. Only then can he come back here and talk. Otherwise, sugar crap is and will be the only main core of everything.

That was fun.

To be fair, we don't know much about the OP's background and I wouldn't be so harsh about it, maybe he actually has accomplished a lot, we don't know. You may be surprised seeing how many professional software engineers 1/ have an obsession with designing a new language, 2/ have a very personal view of programming, and 3/ tend to focus on specific points that are likely stuff that has bitten them in the arse during their career.

So when they design a new programming language, they tend to come up with a pile of ad-hoc concepts and features, rather than take a step back and look at the big picture. Actually, most programming languages ever designed either fall into this category, or were designed in an academic context. In the third category, which is an exception, I would put languages such as Ada or Modula-3, which were designed by groups of professionals with definite goals, rather than individuals focusing on their own perceptions.

Note that, as we often discuss, this certainly has an impact on the intrisic qualities of a language, but little on its popularity. A number of languages that have become very popular were just a bunch of ad-hoc stuff stitched together. Sad but just how it goes.

[Nominal Animal]

Note that, as we often discuss, this certainly has an impact on the intrisic qualities of a language, but little on its popularity. A number of languages that have become very popular were just a bunch of ad-hoc stuff stitched together. Sad but just how it goes.

PHP is perhaps the best example of this.

Considering that, why would anyone start designing a language from the ground up, starting with syntax or grammar, instead of examining how expressions and algorithms would be best expressed in the new language?

For example, if one intends the language to be used for linear algebra type stuff, then array slicing – which goes back to at least Fortran, so nothing new! – is particularly important.  If one intends the language to be used for server-side HTTP stuff, then efficient and effective I/O is paramount, as well as interfacing to external datastores (files, databases, external servers).  For the tasks microcontrollers (as opposed to embedded systems like SoCs) are used, latencies and overhead should be minimal.

For microcontrollers, we have many imperative languages, but very few event-based languages, even though many microcontrollers basically just respond to events, rather than follow imperative logic.  The imperative languages (C/C++, Ada/Spark, Forth, etc.) have decades of work behind them, so unless one creates a variant by only replacing the parts they believe are keeping developers back, starting from scratch like Go, Julia, Rust did will involve at least a decade of development before it becomes comparable; this seems to apply regardless of what kind of programming language one is developing.

One intriguing approach would be to design a language better suited for event-based programming, suitable for use on a microcontroller.  (Can you even create an event-based language without a significant runtime?)  Things like interrupts do map very well to events, but I'm talking more about the internal code structure: instead of the code being based on an imperative sequence, it would be based on posting and consuming events.
Rather than syntax or grammar, one should first examine how one would implement typical tasks in such event-based code, and then compare the maintainability, robustness, and reliability of that code to the same implemented in one of the imperative languages.  How obvious is the event-based code, compared to the imperative code?  How many ways one can write a buggy version of the same without the bugs being obvious?

At the bottom, there is always the machine code implementation.  Languages with a significant runtime have their place (in desktop applications, mostly), but in a microcontroller, it seems like a waste of resources: using abstractions where abstractions are not needed, just to help unskilled programmers produce low-quality implementations.  So, a microcontroller programming language should be efficient wrt. resource use, and the abstractions used map cleanly (without complicated translation layers) to underlying hardware.  This means along with the example code implementation in the new language, one should carefully consider and show the expected machine code the new programming language code might compile to.  (Of course not exactly or perfectly; but approximately, showing how the entire high-level abstract concepts get mapped to the new programming language, and that eventually to the underlying machine code.  If you do need a significant runtime, say a bytecode interpreter or JIT compiler in between, you really need to drop the 'microcontroller' part out; most microcontrollers don't have the resources to support such.)

[tggzzz]

One intriguing approach would be to design a language better suited for event-based programming, suitable for use on a microcontroller.  (Can you even create an event-based language without a significant runtime?)  Things like interrupts do map very well to events, but I'm talking more about the internal code structure: instead of the code being based on an imperative sequence, it would be based on posting and consuming events.

Rather than syntax or grammar, one should first examine how one would implement typical tasks in such event-based code, and then compare the maintainability, robustness, and reliability of that code to the same implemented in one of the imperative languages.  How obvious is the event-based code, compared to the imperative code?  How many ways one can write a buggy version of the same without the bugs being obvious?

Those are, IMNSHO, very sensible and important questions, and ones that do lead to practical advances in the field.

On example is xC, where there are no interrupts[1] because input events (in the form of messages from an I/O port or processor) are handled by a "superloop". There is no RTOS per se, because those benefits are achieved by hardware and multiple independent processors. It enable FPGA-like hard realtime guarantees. When "kicking the tyres" of the technology, I found it stunningly easy to use and without surprises.

There are also many event-based DSLs. While many are pretty scrotty hacks, ISTR one that was used in safety critical avionics. (?Kennedy Carter, kc.com? I came across it at the turn of the milennium!)

[1] a slight lie, but accurate to all intents and purposes

[DiTBho]

Can you even create an event-based language without a significant runtime?

Probably ... no 
But it would be WOW!

[SiliconWizard]

Can you even create an event-based language without a significant runtime?

Probably ... no 
But it would be WOW!

I think this could be achieved with a relatively small runtime.
Message passing schemes is one of my "favorite" topics in programming lately, and it could be applied to passing and handling events.
Sure some hardware support would help getting better performance, but even on a classic architecture with interrupts you could implement that.

I've implemented some of that with pthreads. Haven't done this for small targets yet, but it should be achievable. And achievable using almost any programming language, certainly C. Maybe with some assembly for the "runtime".
Integrating within the language itself being mostly syntax sugar coating. That has always been Wirth's opinion too. Anyone's free to differ.

[tggzzz]

Can you even create an event-based language without a significant runtime?

Probably ... no 
But it would be WOW!

Of course you can.

Since you have complete control of the machine, it isn't difficult, but you benefit from having tiny parts in the relevant machine code, to  swap stacks, avoid priority inversion, ISR exit/entry, and atomic mailboxes.

Have a look at how simple RTOSs are implemented. Ignore those that try to do everything in C, since you can't (unless you are prepared to have severe constraints with programming style).

Ignore POSIX; it is too heavyweight.

[tggzzz]

Can you even create an event-based language without a significant runtime?

Probably ... no 
But it would be WOW!

I think this could be achieved with a relatively small runtime.
Message passing schemes is one of my "favorite" topics in programming lately, and it could be applied to passing and handling events.
Sure some hardware support would help getting better performance, but even on a classic architecture with interrupts you could implement that.

Exactly!

Message passing isn't well taught in traditional undergrad computing courses, for several reasons...
It isn't a computer science topic, cf lambda calculus, compilers, a standard algorithm.
It comes from hardware engineering.
Mainstream languages don't have it as a feature; it is relegated to libraries.

OTOH those that have to push hardware and software to achieve scaleable hetereogeneous long-lifetime high-complexity high-availability systems absolutely rely on message passing. Obvious examples are high performance computing (HPC traditionally stresses computing hardware and software so far that they break in unanticipated ways), telecoms, avionics, etc.

[Nominal Animal]

Can you even create an event-based language without a significant runtime?

Probably ... no 
But it would be WOW!

I think this could be achieved with a relatively small runtime.

I happen to agree.

One approach I use extensively in Linux programming is a binary heap of expiration times (timed events) with an associated array of event descriptions, managed by a dedicated pthread.  Given uniform random expiration times, adding or removing an event involves just e≃2.718 percolation events on average, so both adding and removing new events is quite lightweight and doesn't pollute the cache too much.   The event descriptions stay put, and contain an index to the heap; each entry in the heap is a tuple of the expiration time and an index to the description.  The descriptions vary by what the program does, but typically contain an expiration flag or a POSIX semaphore (sem_t).  To me, this shows that time-related events are not a problem even on small microcontrollers.

One realization I had when writing that post was that in the vast majority of cases, event handlers do not need to be re-entrant, and they either return nothing, or a flag indicating whether the event should not be propagated (or possibly postponed).  (I've written quite a bit of event-based code, from Macromedia Lingo to Javascript handlers, as well as GUIs using GTK, Qt, Tk, etc. toolkits.)  If this is considered in the internal ABI –– for example, making "functions" and "event handlers" ("procedures"?) have completely separate calling mechanisms –– it opens up a lot of possibilities.  For example, consider the case where event handlers have a separate "state context" or object they refer to (indirectly via a pointer register) to record state.  An event is passed in the registers, so that stack is not really used (except for the call chain return addresses and interrupts).
The translation layer (for example, interrupt handler preamble) is very thin, and minimal amount of RAM is needed.
To dispatch an event, you need to know the event handler, its state context, and what to pass in the registers.

The largest part of the runtime would be the superloop event dispatcher.  Depending on the internals, it likely would be quite simple; possibly quite similar to the binary timeout heap thread I mentioned above, augmented with a priority queue of events to be dispatched as soon as possible.

Thing is, what would the actual code look like?  My own opinion is colored by my experience, so I have strong opinions on some facets, but I am absolutely certain I alone would not end up with the best possible syntax.

Also, is a single global event queue sufficient, or are there useful patterns where multiple independent event queues are needed?

Lots of questions to be explored, for sure; and a lot to gain, compared to current programming languages used on microcontrollers, for sure.
Which makes me so darned frustrated to see someone concentrate on syntax and grammar first, before even discussing the target programming model or paradigm.

[DiTBho]

Have a look at how simple RTOSs are implemented. Ignore those that try to do everything in C, since you can't (unless you are prepared to have severe constraints with programming style).

Yup, I have already ported ucOS/2 and XINU to m68k (Gcc) and MIPS5++ (my-c).

my-c adds support for tr-mem, which is useful to implement semaphores and message passing between cores and cops, but I am not prepared to have more constraints with programming style, thus, in both cases specific modules written in assembly cover all the machine-specific code.

I cannot add any event-driven feature to my-c, anyway.
Only libraries.

[tggzzz]

I'm not particularly familiar with ucos, but it seems to be much more than necessary for an event driven kernel (e.g. a GUI, FTP, DHCP etc).

But at least you realise your statement "Probably ... no  But it would be WOW!" is unduly pessimistic.

[Nominal Animal]

Message passing schemes is one of my "favorite" topics in programming lately, and it could be applied to passing and handling events.

Message passing isn't well taught in traditional undergrad computing courses, for several reasons...
It isn't a computer science topic, cf lambda calculus, compilers, a standard algorithm.
It comes from hardware engineering.
Mainstream languages don't have it as a feature; it is relegated to libraries.

I've used MPI a lot in both C and Fortran, but my use cases are HPC and distributed computing, and the messages contain shared simulation data.
I'm also familiar with message mailbox techniques used in e.g. Linux kernel drivers, but haven't really explored microkernels using message passing.
Simply put, my own experience in message passing is too one-sided –– distributed data as messages –– to know much about how one could treat events as messages at the low (machine code) level in an efficient manner.  I am sure it can be done, I just don't know exactly how.

(In MPI, each message is identified by a tuple (sender, tag), and messages with different identifying tuples can be received in any order.  This makes asynchronous/nonblocking message passing (as in MPI_Isend() and MPI_Irecv()) especially powerful.  The per-process runtime in OpenMPI is a helper thread, which coordinates the messaging between processes, and is surprisingly lightweight.)

Simple events like timeouts and interrupts do seem quite straightforward, but when we look at cases where the message involves significant amounts of data –– say, a DMA completion interrupt –– we'll likely find we need to also consider zero-copy techniques; not so much for speed or efficiency, but because RAM and buffer space is quite limited in microcontrollers.

One possibility for handling zero-copy buffer events is to have "acknowledged events": When the buffer is full, the related event is dispatched.  The buffer is "owned" by the event handler, until it acknowledges the buffer-full event by a corresponding buffer-now-free event.  Obviously, there is nothing special in such acknowledgements, it is just a programming paradigm.  We just know from existing programming languages that the approach must be consistent and useful, or we'll end up with footguns like gets() in C.  This is also why I believe development through real-world examples/problems is the way to go.
(I am fully aware that I just described a completely asynchronous/non-blocking I/O or buffer-passing mechanism.  This was deliberate, as we really do need this to make better MCU firmware.  Less buffer-bloat, and so on.)

If I were to try and develop such a programming language, I would start by creating a simple IP stack and an example application, perhaps a HTTP server displaying a static page (with GET and HEAD support), in this new language.  The language, the IP stack implementation, and the low-level machine code generated (roughly – no need to have an actual compiler, just the rough intent suffices, perhaps call ABI on a specific architecture), would all be developed in parallel.  One could examine its readability and maintainability by giving the language spec and a snippet of code with a bug in it (something the compiler would not complain about; a thinko rather than a typo) to a suitable victim test person, describing the effect of the bug, and asking them to find the bug.  If the person finds the bug without having to consult the language spec beyond the introduction, we'd be on the right track.

Things like keywords or reserved words, operators, exact syntex, and so on, are secondary, because the point is to develop a language to effectively express the concepts.  You could even have antagonistic teams, finding ways of obfuscating code and hiding bugs in it, to arrive at the most robust syntax.

That the large organizations like Google do not seem to grok this at all, and instead focus on churning out yet another imperative object-oriented language after another, really depresses me.  So, perhaps I'm a bit hard on OP for going down the exact same unfruitful road, but I definitely have a reason to here, it's not just "my opinion" here.  The entire method is at fault, and a completely different design approach should be taken.

[SiliconWizard]

I'm not particularly familiar with ucos, but it seems to be much more than necessary for an event driven kernel (e.g. a GUI, FTP, DHCP etc).

But at least you realise your statement "Probably ... no  But it would be WOW!" is unduly pessimistic.

I've written a small preemptive scheduler (priority-based + per-task tick period) for my RISC-V core, which would lend itself well to event-driven stuff. One C file and one assembly file. Total object code is about 5KB with optimizations enabled. There is no specific feature for passing messages at this point, but it wouldn't be too hard to do and would probably take no more than 1 extra KB or so. I've put this on hold, but I was considering what kind of hardware blocks I could add to my core to make this more efficient. Sure there is context switching which takes a few tens of cycles. Could be avoided if I didn't make this preemptive. In order to avoid latency due to cache issues (on systems with a cache), one can define a core-coupled area of RAM dedicated to context saving/dedicated stacks/messages, so it doesn't have to go through any cache.

[tggzzz]

I'm not particularly familiar with ucos, but it seems to be much more than necessary for an event driven kernel (e.g. a GUI, FTP, DHCP etc).

But at least you realise your statement "Probably ... no  But it would be WOW!" is unduly pessimistic.

I've written a small preemptive scheduler (priority-based + per-task tick period) for my RISC-V core, which would lend itself well to event-driven stuff. One C file and one assembly file. Total object code is about 5KB with optimizations enabled. There is no specific feature for passing messages at this point, but it wouldn't be too hard to do and would probably take no more than 1 extra KB or so. I've put this on hold, but I was considering what kind of hardware blocks I could add to my core to make this more efficient. Sure there is context switching which takes a few tens of cycles. Could be avoided if I didn't make this preemptive. In order to avoid latency due to cache issues (on systems with a cache), one can define a core-coupled area of RAM dedicated to context saving/dedicated stacks/messages, so it doesn't have to go through any cache.

Sounds right.

I'm not a fan of pre-emptive event processing. I prefer a superloop where an event is consumed and the actions run to completion before processing the next action. The action can be short duration computation/io/etc, and usually concludes by generating another event that moves the FSM or FSMs into another state. That's sometimes called the half-sync half-async design pattern; where the "sync" computation is followed by the "async" event.

If the processing is going to take too long, then encapsulate it in a separate start.event plus completion event, and modify the FSM accordingly.

Caches kill hard real-time guarantees stone dead; that's why xC on xCORE processors avoid them as well as interrupts.

[SiliconWizard]

I understand your point about preemption, but cooperative multitasking can get messy pretty fast, and the cost of keeping context for each task can be every bit as expensive, if not more in the end, than just saving and restoring all registers. You end up with pretty intricate state machines. I've written a FAT32/exFAT library in this cooperative way (in order to have no blocking call whatsoever), and it was a real PITA.

And if you stick to messages/events as synchronization means, then a lot of the hard stuff about synchronization becomes a lot easier.

Meeting deadlines is also easier in general with preemption, as long as priorities and tick periods are handled appropriately (so not in the way general-purpose schedulers usually do.)

[tggzzz]

I understand your point about preemption, but cooperative multitasking can get messy pretty fast, and the cost of keeping context for each task can be every bit as expensive, if not more in the end, than just saving and restoring all registers. You end up with pretty intricate state machines. I've written a FAT32/exFAT library in this cooperative way (in order to have no blocking call whatsoever), and it was a real PITA.

I don't disagree

The ameliorations are good taste and clear thinking, neither of which are in common supply. Even then, sometimes the application doesn't suit the design pattern; have many tools in your toolbox, and pull out the right one for the job.

And if you stick to messages/events as synchronization means, then a lot of the hard stuff about synchronization becomes a lot easier.

Exactly. And given the strong similarities between timeouts, inputs arriving, outputs completing, messages to/from other processes/cores, there seems little reason not to have one mechanism for the whole lot.

Meeting deadlines is also easier in general with preemption, as long as priorities and tick periods are handled appropriately (so not in the way general-purpose schedulers usually do.)

My tastes are otherwise, but no doubt I could be convinced in some cases.

[DiTBho]

But at least you realise your statement "Probably ... no  But it would be WOW!" is unduly pessimistic.

I mean I don't think I am good enough to make my-c an event-oriented language.
I can use my-c to write event-oriented software, like I can use C, but there is no improvement between them, except my-c has tr-mem support, so if your hardware has tr-mem (POWER10), it can simplify some inner message passing mechanisms.

Mailboxes can have support as well, but it hasn't been designed to serve any event-oriented paradigm, so it's nothing new, nothing revolutionary (like xc), you still have to write a lot of assembly. Just a bit less.

[Nominal Animal]

It is funny how it all really boils down to RAM use, on the smallest microcontrollers.

Even on 32-bitters, a lot of time and effort is spent to find out exactly how much RAM to reserve for each task stack.
Even in Linux, the default stack size is so large (2-8 megabytes, depending on the 32-bit architectures) that it is the first limiting factor on the number of parallel threads a process can have.  (On most 32-bit architectures in Linux, 16384 bytes is fine for non-recursive threads.)

The very first things I'd drop from an ABI is the concept of a function stack frame.  The less stack is used, the more you can do with limited RAM.  (Anyone who suggests a red zone should be shot outright.  That damn thing makes any kind of stack use tracking via the stack pointer unreliable –– useless, really.)

It also means it is best to pass data by value in registers, so that data already in registers does not have to be written to RAM just to call a function.
Passing data by reference is an utter pain.

Many microcontroller tasks are easier to describe in terms of generators and coroutines.  For example, a COBS stuffer/unstuffer can be implemented as a simple state machine FIFO.  By their very nature, they're non-reentrant.  So, if a language –– be it event-based or imperative –– supports multiple forms of "functions", one could be a closure-like coroutine/generator/producer-consumer, which uses a state object to record its own state, and stack only for calling functions; "consume"/"produce" ("next"/"yield") would only be allowed in the function itself, and not in any functions it calls.  This way, one would not need to swap stacks to call these, since they may not use stack across a "consume"/"produce" event.  Also, the ABI can define which registers are saved/lost across "consume"/"produce", further reducing the cost of using such coroutines.
I haven't yet discovered a really good way to express such sequence-processing "functions", but they definitely need to be handled differently to ordinary functions.  (Even Python 3 is still struggling a bit wrt. generator functions; see e.g. yield expressions.)

The above also means that recording programmer intent, similar to how one can use function attributes with GCC and Clang, so that the compiler can enforce the rules when compiling the code, is important.  I personally am having trouble thinking up good descriptive names for the things, much less worry about how to syntactically express the darn things.. 

[Kalvin]

From time to time I have been missing a [C] preprocessor that would be able to generate hygienic macros, and would be coupled to the compiler more tightly so that the macros could determine and check the types of the macro parameters. In that way it would be possible to extend the core language, and for example create framework for implementing events, state machines, patterns, and templates without making the core language too complex.

[tggzzz]

...
which uses a state object to record its own state, and stack only for calling functions;
...

That's a very good technique, especially where there are many independent FSMs running simultaneously, e.g. telecom phone calls, or web connections.

Have a single FIFO containing events yet to be processed. Have one "worker thread" per core/processor dedicated to consuming an event, doing the actions, and creating another event. Each worker thread sucks the next event from the FIFO. Doesn't require co-routines. It all leads to good scaleable high performance applications.

[tggzzz]

From time to time I have been missing a [C] preprocessor that would be able to generate hygienic macros, and would be coupled to the compiler more tightly so that the macros could determine and check the types of the macro parameters. In that way it would be possible to extend the core language, and for example create framework for implementing events, state machines, patterns, and templates without making the core language too complex.

I hate that kind of thing

Need to read/parse all the codebase to understand what's happening.

Templating languages end up being Turing complete monsters - like the C++ monstrosity did by accident. https://en.wikibooks.org/wiki/C%2B%2B_Programming/Templates/Template_Meta-Programming#History_of_TMP

There are several excellent design patterns for implementing FSMs in conventional languages. I haven't found a reason not to use them.

[Kalvin]

...
which uses a state object to record its own state, and stack only for calling functions;
...

That's a very good technique, especially where there are many independent FSMs running simultaneously, e.g. telecom phone calls, or web connections.

Have a single FIFO containing events yet to be processed. Have one "worker thread" per core/processor dedicated to consuming an event, doing the actions, and creating another event. Each worker thread sucks the next event from the FIFO. Doesn't require co-routines. It all leads to good scaleable high performance applications.

Miro Samek's freely available ebook Practical UML Statecharts in C/C++, 2nd Ed Event-Driven Programming for Embedded Systems
https://www.state-machine.com/psicc2
describes an Active Object pattern decoupling the event producers and consumers:
https://www.state-machine.com/active-object

This Active Object-pattern can be used as an alternative to / with the system-wide event-queue.