text command parsers?

[westfw]

Is there a a defacto standard for interactively parsing text command lines in microcontroller code?
(NOT the "unix command line"; there are plenty of libs for that.  I'm talking about a program with a prompt that you type text commands at, like "set pin 13 = 1" or "clear uart 2" or "send westfw Hi Bill!"
Preferably something Twenex-like (or cisco CLI), with context sensitive help, keyword completion, history and line editing.  C or C++ preferred; (large-ish) microcontroller sized (I thought unix gettext might be a starting point, but it seems to focus on internationalization...)
I could dust off CMU's "CCMD", but it's one of those 20y-old programs whose last edits were the addition of prototypes and modern C functions (vs K&R) and hacks for non-standard environments, and it was never tweaked to be modular on systems without various major features.

[magetoo]

Don't know about anything commonly used in microcontrollers, but as a starting point: On the Unix side the usual suspects would be readline (GNU GPL) or editline (BSD).  That'll get you line editing and history.

The readline Wikipedia page mentions a library called linenoise at the end, which claims to be much smaller (and BSD licensed), that might be worth a look for a microcontroller where size is an issue.  On the other hand you mention some pretty advanced stuff in your wishlist, so that might not be right for you either.

Another option is having only the simplest possible code in the microcontroller, and then run a shell around it on the controlling computer, if you have one; I sometimes use a small tool called "ile" (Input Line Editor) which provides editing and command history for dumb programs (like e.g. netcat).

Hope that helps.

[andyturk]

Not standard, but here's what I do to implement a command line shell. It's written in C++ (of course) and has ties to ChibiOS, but could be ported fairly easily to other environments.

https://github.com/andyturk/libakt/blob/master/akt/shell.h
https://github.com/andyturk/libakt/blob/master/akt/shell.cc

Some sample output from a project using this via a USB CDC serial connection:

Code: [Select]

> ?
exists   -- test if a file is present
type     -- type file contents
rm       -- remove a file or directory
pwd      -- print current directory
mkdir    -- create directory
ls       -- list files
df       -- show disk info
cd       -- change directory
i2c      -- read/write to the I2C bus
sensor   -- show sensor info
cal      -- show/clear sensor calibration info
9250     -- Invensense MPU-9250 commands
reset    -- force hard reset
memory   -- display RAM/ROM info
threads  -- list ChibiOS threads
info     -- program/toolchain information
help     -- list available commands
time     -- show/set time of day
id       -- show CPU unique id
date     -- show/set calendar date
buttons  -- show button state
bat      -- show battery charger state
> df
FS: 237998 free clusters, 64 sectors per cluster, 7615936 kbytes free
card is writable
> bat
external power is present
full
battery voltage: 5246
> i2c scan
  found device at 0x60
  found device at 0x68
> threads
    addr stku prio refs     state     time name
20001AD0 0000   64    1   WTOREVT 00000032 main
20001988 0000    1    1     READY 00047662 idle
20003A18 0000   64    1  SLEEPING 00000000 button-debounce
20003BD8 0000   64    1   WTOREVT 00000002 card-mounter
20001F30 0000    2    1     READY 00000003 usb_lld_pump
20002E58 0000   64    1   CURRENT 00000015 shell
>


[hamster_nz]

You could most likely hack my tiny basic port for Arduino to do something useful for you.

http://hamsterworks.co.nz/mediawiki/index.php/Arduino_Basic

[westfw]

Thanks Magetoo!  "readline" was what was on the tip of my tongue, but couldn't find ("Comfortable terminal input library" hmmph!)  I don't like the license, though.  I'll have to look into the alternatives that you and wikipedia mention.

Thanks to others, too, but I was hoping for something well organized as a partitionable/extensible library, (don't need date/time parsing?  Leave it out!) rather than examples of how to implement parsing in general.  Rewriting CCMD might be reasonable, yet.

[andyturk]

A basic shell is not that hard to write yourself. The first task is to accumulate a line of text from a keyboard, UART, USB connection or whatever. Depending on how much line editing you need, it can be half-a-page of code (mine does backspace/delete and ^U to nuke the whole line). The trick to making it truly portable is is factoring the code so it'll work with blocking calls like getc() and also with interrupt routines that give you a character at a time.

Then you'll need some kind of service entry point to see if a complete line of text has been collected, parse it, and look up the command to execute in some sort of list. Figuring out how to collect/represent the list of all commands known by the shell is the interesting part, IMO. Most implementations I've seen have commands appear in a monolithic table, which means that part of your code needs to know about *all* the commands supported by the system. That's a pain because when you want to add some new whizzy command for your latest project, you have to patch that in somehow to the list of boilerplate commands you want to be available in every shell. It's not conceptually difficult, it's just messy from a code management point of view.

My approach was to skip the monolithic table and build up a linked list of available shell commands at runtime (via C++ constructors). I don't really care if looking up a shell command by name is O(N). The advantage is that simply constructing an instance of the shell command class adds it into the list, so the generic shell source code doesn't need to know about what commands are available. In fact, there's no list of commands in the source code at all--it's built at runtime.

For instance, whenever you implement a new device driver, you can also add a shell command to talk to it. The shell command is closely associated with the device it's talking to and usually lives in the same source file as the driver. Add that driver to a new project, and you automatically get the shell command with it. Very cool.

The first thing I do in a new project is blink an LED. The next thing is to bring up a command-line shell. It's a great way to get stuff running quickly.

[amwales]

If you are using C, you could take a look at lex, yacc or flex, bison, these are lexers and parsers.
You normally start getting a little annoyed at the amount of code being generated and end up rolling your own lexer and parser.

Alternatively, write up a simple grammar in BNF ( http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form),
use a simple set of rules to translate that into code ( http://en.wikipedia.org/wiki/Recursive_descent_parser look at the C example).

So long as you stick to the rules for translating it, coding it become a very mechanical process, it starts looking very verbose but is simple to write and extend later if you don'f fall into the trap of trying to optimize it.

I hope that was helpful.

[andyturk]

If you are using C, you could take a look at lex, yacc or flex, bison, these are lexers and parsers.

I love flex/bison, but they seem a little heavy for a command line shell, especially on a microcontroller. All you really need is to break up words by whitespace. Add quoted strings to that if you're going to be really fancy.

[zapta]

I love flex/bison, but they seem a little heavy for a command line shell, especially on a microcontroller. All you really need is to break up words by whitespace. Add quoted strings to that if you're going to be really fancy.

+1

BTW, I liked your idea of independent commands register themselves using static constructors and having a common base class.

[amwales]

If you are using C, you could take a look at lex, yacc or flex, bison, these are lexers and parsers.

I love flex/bison, but they seem a little heavy for a command line shell, especially on a microcontroller. All you really need is to break up words by whitespace. Add quoted strings to that if you're going to be really fancy.

This is why I said in my post - "You normally start getting a little annoyed at the amount of code being generated and end up rolling your own lexer and parser."

[legacy]

Hi
in this thread i presented my little interpreter written in c++, sources is provided following the git repo.

feel free to modify it as you want 

[Erwin Ried]

Have you checked "bitlash"?
http://bitlash.net/

[codeboy2k]

For instance, whenever you implement a new device driver, you can also add a shell command to talk to it. The shell command is closely associated with the device it's talking to and usually lives in the same source file as the driver. Add that driver to a new project, and you automatically get the shell command with it. Very cool.

The first thing I do in a new project is blink an LED. The next thing is to bring up a command-line shell. It's a great way to get stuff running quickly.

I agree with the turk   Getting an interactive monitor/shell running is a best practice first step.

I sometimes have done this with a forth shell. The basic primitives are there in the default interpreter, lots of libraries are reusable, and I can create device drivers interactively in forth to test out stuff immediately.  It's really great for embedded work and a seriously lost art.

[Rigby]

I don't know a lot about microcontrollers or microcontroller programming, yet.  When I write a 'shell' in C# or Java, though I use what is called the "command pattern" and if you're using an OO language on your micro, such as C++, then the pattern would work there, as well.  The command pattern can be used to do parse just about anything that comes in a string, even weird things like Roman numerals.

[linux-works]

the high level things in c++ and c# don't really belong in controllers.  there is VERY limited memory there and even limited code space in the arduino, for example.  using OO things starts to bring in too many libs and dependancies.  something is classed off of something else and so they all get pulled in.  blech!  not good for low mem controllers.

(OP: is this bill from the old cisco days?)

[Rigby]

I have a micro on my desk with 1MB of program space.  I'm sure I could fit a few objects in there.  I'm not talking about C# at all beyond it being an OO language - it doesn't really suit my needs as a development platform for microcontrollers.

You're definitely right for most micros, though.  There's probably a way to do it without objects, though, and in quite small space, which I guess is what OP is asking about.

[linux-works]

the arduino 328 chip (most popular of the arduinos) has 32k (k-bytes, not m-bytes!) of program space!!

that amounts to about 10k lines of C code.  that's all.  I filled my arduino up almost to the last byte with about 13k lines of c code on a project I was on.  I had to remove some things in order to add others; I kept running out of the 30k (2k goes to bootloader) that the arduino allowed.  I even had to remove floating point calls (libs) since that ate up 4k or so of code.  4K!!  not meg.

c++ is totally wrong for small footprint embedded use.  you can pull all kinds of things in by mistake and not even realize it.

[Rigby]

Yeah, I know.  I'm agreeing with you.

My micro with 1MB of program space is the TI Tiva-C Connected Launchpad.  It's actually a pretty great board, and only $20 shipped.

[linux-works]

ah, to have a full megabyte.

if only to dream...



my code was so full of ifdef's since if you wanted this feature, it would be at the expense of another and you'd have to be clever with how you ifdef'd things.  there simply was not enough code space in the controller and so it had to be structured as optional modules that you would include only at build time, certainly not at runtime.

a lot of libraries and systems for unix don't take into account low mem footprint size, for target binaries.  you have to pick very low dependancy libs if you want them to fit in tiny controllers.

when I have to parse strings inside controllers, I just use the string.h primitives and go from there.  its more work, but I have no lib dependancies other than libc, which has to be there anyway.

[Rigby]

sounds like a sound approach to me.  aren't there microcontroller libs available now that have been honed a bit to produce smaller binary sizes?  github probably has a bunch of microcontroller gems littered about in random places.

[senso]

The arduino IDE produces a lot of crap, blink a led 1.x k, do it with avr-studio/c+makefiles, and its 184 bytes, a little diffference.
I'm also dipping my toes in this subject, so keep posting about this, so I can get some ideas, with luck it will work at the 100 try

[westfw]

(OP: is this bill from the old cisco days?)

Yep.  Do I know you?  It's ... annoying to have all this experience with a proprietary infrastructure that I no longer have access to :-(

One doesn't usually do much of a CLI parser on small microcontrollers, but now that they're getting up to the 256k to 1M code size, it's worth thinking about again.  Although the other annoying differences a modern microcontroller with 1M of memory and a original PC with 640k of memory is that the PC memory was all RAM, and people wrote code without paying any attention to which parts of their data could/should be read-only.  (Slightly interesting: the original tops20 comnd% jsys was better than the CMU CCMD libraries, out of desire to keep code pages shareable...)

[mariush]

If you're low on actual space in the flash, you could always store the commands in the eeprom - 1024 bytes on the ATmega328, 512 bytes on the ATmega168 and ATmega8, 4 KB (4096 bytes) on the ATmega1280 and ATmega2560

you can store it for example like [1 byte for command length] [ x bytes for command 0] , [1 byte for command length] [ x bytes for command 1] ... repeat  [ null to tell you end of commands]
then you have a switch or a bunch of ifs in the code.
when reading from eeprom, if the command you parsed has less or more characters than the value you read from eeprom, you can just skip to next command in eeprom

[westfw]

One of my early attempts at fame was a Serial Comms program for the IBMPC.  It had to run in 256k of RAM, and from a single 360k floppy.  Lots of 8086 assembler.  Unlike a lot of SW from that timeframe, it had extensive help for its (also extensive) configuration menus.  But the help text was all stored on the floppy till needed.  That actually worked pretty well (relative to the standard of the day...)

(That never did see the light of day.  What an education in business details and sleazy-acting partners.  And lawyers.  Oh well...)  (and looking back, it had some really cool security holes (like an H19 escape sequence that displayed internal memory values...))

[mrflibble]

(and looking back, it had some really cool security holes (like an H19 escape sequence that displayed internal memory values...))

IBMPC, are you still there? If so, reply "HAT" (500 letters).