Graphics on an embedded system, TFT display - need help with some pointers

[peter-h]

I have not done this sort of thing, since the 1980s when I used a Z80 driving a graphics coprocessor https://en.wikipedia.org/wiki/Thomson_EF936x or an NEC UPD7220 https://en.wikipedia.org/wiki/NEC_%C2%B5PD7220. The 9365 project ended up an obscure but successful commercial product, while the 7220 one was never finished because we spent a year trying to decipher the NEC manual It was all done in assembler, of course, and actually worked quite well, using video buffer swapping. Then the PC came, everybody used 80x86 boards for this stuff, and the graphics coprocessor business died because those chips were not any faster.

I am just looking for some pointers on present-day software techniques.

Let's say I have a 256x256 TFT and I want to draw a clock. One can develop primitives like draw a filled rectangle or a filled polygon and use these to draw the hands. Then as the hands rotate, and bear in mind one hand will overlap the others, you have to draw the "last time image" in the background colour (to make all the hands vanish) and then draw all the hands afresh using the desired colour. This works ok because you aren't changing all that many pixels. It works much better if you can chuck the new image into a "background buffer" and then swap them. The clock is actually a bad example because you will obviously just swap the buffers at 1Hz, or maybe 5Hz, so this is a trivial way to achieve visual perfection, but not trivial if you are drawing a continuously sweeping hand(s).

More clever ways redraw just the parts which have changed, which generally means fewer pixels need writing, but the software is much more complex; you now need to have algorithms which deal with polygon overlap etc. This is obviously how computer games are done but they have fantastic power, in the CPU and even more in the GPU. I just have a 168MHz ARM 32F4 and a 21mbps SPI feeding the display

Do people do this stuff the slow (but intellectually obvious, and fast enough for simple graphics) way described above, or are there easy to use C libraries where you would define shapes, an overlap order of the shapes, and orientation (origin and rotation) of each shape, and the library draws it all for you, by updating only the pixels which actually changed?

Thank you for any hints.

Obviously I am not building a clock but the graphics complexity is very similar.

[capt bullshot]

Well, that depends.
Especially on the memory bandwidth of how the display is hooked to the controller, and if you can switch framebuffers.

It's quite easy if you have a fast memory mapped framebuffer and can use double buffering:
Erase framebuffer 1, redraw everything on fb1, switch visible display to fb1
Erase framebuffer 2, redraw everything on fb2, switch visible display to fb2
rinse and repeat ad infinitum
works best if you can synchronize the switching to frame rate
Use your own code or any library you can find.

If you got only one framebuffer, but still memory mapped (fast access):
Erase and redraw small portions (so flickering won't be visible) or draw over small portions in fg/bg colour (depending on the object shown, overwriting works quite well for e.g. text fonts, geometric shapes work well by drawing the old in bg colour and the the new one in fg colour, element by element)

For SPI attached framebuffers:
check if the display can handle two frame buffers - then use the first strategy, otherwise the latter. If your SPI bandwidth is low, I'd recommend the latter anyway.
If your MCU has plenty of RAM, one can draw in internal memory first, and the transfer the delta only (you'd need a second internal buffer then), or transfer the whole framebuffer in one piece (maybe by DMA in the background). Depending on the memory size and bandwidth of the display interface, I found the bulk copy (by DMA if possible) method quite effective.

I've done most of these approaches before, chosen depending on the available HW / bandwidth. For example, a complete framebuffer bulk transfer to a SPI (or slowish parallel bus) display module might be faster and easier than drawing single pixels through individual commands, even with drawing reduced to changed pixels only.

There's a lot of free and commercial libraries available (don't ask me for examples), some of them might even offer ready made drivers for your particular displays. I don't know about any library that uses your "intelligent" approach, afaik they all draw the primitives completely in the order you call the API. Some STM32F4 and higher chips offer the DMA2D that can offload the CPU from e.g. copying font bitmaps or sprites to the frame buffer (works with the frame buffer in CPU / DMA addressable memory only, not through SPI).

Here's one example of a free library for smallish displays:
https://github.com/olikraus/u8g2
Most probably not suited for your needs, it uses a clever approach to run on low MCU ressources.

[peter-h]

One interesting angle is that even a 256x256 px display, 24 bit colour, needs 196k of RAM, which is what the whole 32F4 has

One could do it using 8 bits/colour, perhaps which a colour mapping LUT in the display - assuming the system wasn't doing much else.

So even drawing a trivial clock, in 24 bit colour, is not feasible using a buffer in the CPU. Inside the CPU it all has to be "objects".

Fonts can also take up a lot of space, if you want nice ones (not the crappy 7x5 sort). Fortunately in my application there will be very little text.

One of the ST 32F4 development boards, the DISC1 Discovery Kit, came with an LCD of about 5x6cm but it was connected using a parallel interface. I can see a fossil entry in the linkfile of my project
MEMORY_B1 (rx)      : ORIGIN = 0x60000000
which is believed to relate to that. However that uses up a huge number of pins. But ST did supply some sort of library so you could do a printf() to the LCD.

[ajb]

This changes a bit depending on what sort of display you're driving and what you're trying to show. 

If the display has no internal frame buffer, then the frame buffer has to be in local memory, which is expensive in terms of memory consumption, but the tradeoff is that the local memory is fast to read as well as write, and in particular is a LOT faster in random access patterns (at least compared to the interfaces typical for these sorts of displays that DO have internal frame buffers).  That speed makes it a lot simpler to do more complicated graphics, because it's easier to update arbitrary sections of the screen.  Having internal memory also allows for DMA, and in fact some parts like the STM32F7/H7 family have DMA engines specifically designed to support graphical interfaces, to reduce CPU overhead.  If you have enough memory to double buffer, then there's a bonus advantage that you can swap the entire image at once and get nice crisp updates with no tearing.  However this sort of thing locks you into higher performance MCUs that can support the attendant memory and interface speed requirements (not to mention pin counts).

When the frame buffer is in the display, by contrast, you generally have a much more limited rate at which you can write any changes to the screen, because not only is the interface data rate lower but typically you have to transfer commands before writing or reading, which means that random accesses are particularly expensive.  This isn't such a problem for simple character/glyph/fill types of screens because those work well with writes to rectangular areas, which allows for more efficient bulk writes to the display.  Doing anything that involves partial transparency within a rectangular area, even something like a circle drawn on a background color, or text drawn over something other than a basic background color, is more complicated.  That sort of thing kind of needs a local frame buffer, which gets back to needing a bunch of memory.  Transferring the whole buffer to the display would still be expensive, especially on a particularly slow interface, and can result in visible tearing, so it can be beneficial to have all of the display module track the 'dirty' areas of the frame buffer and then only write those areas when it's time to update the display.  I've done this on OLED displays where the display only allows page-wide writes, the interface is slow, and the memory requirement is low, it's not that hard and has a significant performance improvement.  Below a certain display size the frame buffer can be a reasonable tradeoff for not having to keep track of what logical objects are supposed to be on the screen anyway, and it can greatly simplify the rest of the graphics software.

So you kind of have to do the math on how fast you need to be able to update the display and how long it will take to do various size writes to the display and see how that fits in your application requirements.  A full write to a 256x256x16bit display at 21Mbps ends up taking ~52ms (a little longer due to overhead, possible framing requirements, etc).  If that's acceptable, and you can spare 128kB for a frame buffer, then it would be pretty easy to do arbitrary redraws in the local buffer and just kick off a DMA transaction when you're ready.  If that's not fast enough, you can track the areas that are changed and only rewrite those.  If you don't have the memory for a frame buffer, then you're going to need to compensate with some extra computational complexity or simplify your graphics.

There are certainly some embedded graphics libraries out there, since your other threads have mentioned STM32 projects you might look at their libraries.  I don't know how good their graphics libraries in particular are, but if you're using an MCU that has their graphic-specific DMA engine you probably want to take advantage of that. 

[SiliconWizard]

I have done this kind of things both with my own libaries, and with third-party ones.
(I do admit that I have mostly used my own - and as with anything else, this is code built over the years and that can be reused. This is because I've had a rather strict policy of using the least third-party code possible. But that has evolved a bit.)

With that said, there are nice libraries these days for what you want to do, and if you have no existing code base and no time or will to write all this yourself, we can suggest a few options. One pretty nice option is LVGL: https://lvgl.io/ - I highly recommend it.

[ajb]

One of the ST 32F4 development boards, the DISC1 Discovery Kit, came with an LCD of about 5x6cm but it was connected using a parallel interface. I can see a fossil entry in the linkfile of my project
MEMORY_B1 (rx)      : ORIGIN = 0x60000000
which is believed to relate to that. However that uses up a huge number of pins. But ST did supply some sort of library so you could do a printf() to the LCD.

It sounds like that board is using the external memory interface (not the RGB/Hsync/Vsync TFT interface, to be clear), which can be a good way to speed up writes to displays that support parallel data interfaces.  You can often trade off between pin count and speed by selecting the width of the interface, many TFT controllers support variable interface widths in parallel mode.  One gotcha with this on Cortex M7 parts is that you need the core to treat that address space as 'device' memory.  The M7 core is dual issue and doesn't guarantee that memory accesses will happen in program order or quantity for standard memory, and since the LCD doesn't have address lines and relies on the word sequence to put the right data in the right place this will cause weird and inconsistent things to happen.  This is independent of the optional cache as well as compiler optimization.  IIRC STM32F7/H7 parts allow the external memory to be mapped to two different address ranges, and one of those is already tagged as device memory, or you can use the standard location and use the MPU to tag it appropriately. 

[x86guru]

I didn't read all of the details so maybe this is not viable for you, but another technique that may work for you is color pallet cycling. Or RAM-DAC reprogramming. For example, if you have a ROM/nvram/UFM where you can store a static 256x256 raster image with 8-bit values, you can setup that 256x256 raster image with all of the different objects you want to selectively display by reprogramming the RAM-DAC to turn on/off specific objects that index into the RAM-DAC. For overlapping objects, you'll want to use a unique 8-bit RAM-DAC index for the overlapping pixles so you can still program the RAM-DAC to fully display individual objects even though they overlap.

For example, color pallet cycling was used a lot back in the day to create the notion of animation even though pixel values (pallet indices) were not changing. Only the color pallet (ram-dac) was changing.

Here is an example of color pallet cycling. Only the color pallet is changing

[strawberry]

ST uses TouchGFX for this task but I dont get it  so I made my own library that works on STM32F103RB 320x480 16bit 8080 interface 13ms refresh rate

[Nominal Animal]

Since you are using SPI, you will be using a display controller that includes a full framebuffer.

A key concept here is damage: the pixels that need to change between the current and the next frame.

Because there is overhead in setting up the region to be updated, damage is usually not tracked in single pixel resolution, but in multi-pixel sized units.
The exact size and shape of such units depends on your display controller.  If the controller has a preferred data direction, it is common to use units that are a small power of two sized in that direction (2, 4, 8 are typical), but a single pixel in the other.

For example, for a 240×240 or 256×256 display, you could use 8×1 pixel units, and a damage bit map of 32×240 or 32×256 (960 or 1024 bytes).  Whenever you draw something at (y,x), you also set (y/8,x) as damaged.

When an element is removed from a frame, it gets drawn to the damage bitmap.  When an element is added to a frame, it gets drawn to the damage bitmap.  When an element moves, both the old position and the new position are drawn to the damage bitmap.

Only the damaged pixel units are actually transferred to the display device.

Exactly how you construct the pixel data is a separate question.  Indexed-color tiles (say, two pixels per byte, with a separate 16-entry palette; or four pixels per byte, with a separate 4-entry palette) are often used.  For anti-aliased text, instead of palette, you can use opacity (00 = transparent, 01 = 33% opaque, 10 = 67% opaque, 11 = fully opaque), and keep a tile in Flash for each glyph you need, using half a byte per pixel in storage.

A common approach is to have a "buffer" of one scan line at full supported color resolution (full color, not indexed color), which is constructed for the next frame if any of the pixels on that scan line are damaged.  You start with the background color, and then blit each needed tile and/or vector element, in the correct order.  Then, for the pixels in damaged regions, you convert to the final color format, preparing the command sequences, and continue on for the next scan line.
Things like anti-aliased lines of specified fractional thickness become much simpler, when you only need to calculate a single pixel wide slice.  Approximately anti-aliased circles are easy, if you adjust the pixel opacity as a function of r², and compare the distance squared at both "edges" of each pixel to determine whether the pixel is outside, at the inner edge, on, or at the outer edge of the circle.  Lots of interesting math here, if you want to do it yourself.
Note that the glyph tiles should be oriented so that the correct single-pixel row/column is easy to scan to the buffer.  For two pixels per byte, it can be cheaper/faster to scan non-consecutive bytes, and have two different routines depending on whether one is scanning an odd or even row/column from the glyph.

On a 32-bit microcontroller, using 15-bit color opens up some fascinating possibilities.  In particular, if you keep the three color components in bits 0-4, 10-14, and 20-24, you only need two 32-bit multiplications (one by a and the other by 32-a), one addition, one bit shift (right by 5 bits), and a binary AND (with 0x01F07C1F), to blend any two such colors.

I don't think a single library can fit even my own needs in different situations.  For example, even among circular dials, a tiny difference, whether the dial arm/indicator/arrow is constant width or triangular/polygonal, would completely change how I'd implement anti-aliasing, and therefore the core operations.  If the dial covered the entire screen, I might only use one bit per scan line for the damage.

That's why I start by drawing some example graphics I'd might want to see, preferably using a simple firmware to display each on the actual final display unit to evaluate ideas and compare usefulness (especially using unsuspecting victims friends and cow-orkers to evaluate their response to the display).  Our desktop machines have ample power to draw to a framebuffer (even at 2×2 or 4×4 oversampling for even better antialiasing) using existing graphics toolkits, and convert the data to e.g. 15-bit color, and push the 115200-byte (240×240) or 131072-byte (256×256) frames to a simple firmware that just displays the host-provided data on the actual display module.
I have learned that what I can imagine is usually only the first step towards what I actually want, and experimenting with the actual display unit is a core part of the work flow that enables me to arrive at a result I can be proud of.

Also, this way, I know what I need to implement, roughly, before I start working on the code.  It is much more efficient than trying to cater to vague ideas and wishes.  I know others are happy to limit their implementation options to what is already available, but I'm not.

[T3sl4co1l]

Well not even games, just basic operating systems handle all of that -- for example Windows GDI, or whatever the hell x or other *nix graphics APIs use.  Many (PC grade) languages support such features, often as mere shims on top of the OS itself -- for example, Java frames, labels, canvas, etc. being basically adaptations of the respective objects in Windows or etc.  These would be, understandably quite difficult to implement in an embedded context, without an OS to handle the multiple layers from hardware interface (drivers) to API.

Most likely, you'll assemble such objects yourself, from bespoke modules that do most of the work, plus a little logic, that's likely internal to your main program, rather than associated to the objects as such.  For example, a checkbox typically has its internal logic (concerning a whopping one bit of information), and interfaces automatically (i.e., on creation) with user input and graphics.  The same functionality can be built from program logic and a couple images, not needing to be a whole object at all; you likely aren't even using a language (i.e. C, or even ASM) that has the same fleshed-out notion of objects as in OOPL.

For your purposes, "canvas" is probably the keyword you'll find most useful.  Compositing is what you're after: you want to draw graphics on top of others, and especially if that needs to be done with alpha (as in antialiasing or etc.), the underlying graphics need to be known as well.  This is easiest when a frame buffer is handy (just redraw the damaged sections, and for alpha, just read the previous pixels), but harder when not (perhaps you have to compute the underlying graphics instead).

Libraries are handy, like, as I recall a basic canvas (I recall Adafruit has one) takes just a little too much RAM to be practical on the smaller AVRs, but 8k+ SRAM is enough.  And for sure, anything that can host a framebuffer (like a ST32F4, even without external RAM) makes this much easier; if you're using an unbuffered display (i.e., 24-bit RGB+sync or the like) and the onboard panel controller, you also save trouble with an external one.


When you are using an external controller, there's its own limitations to consider.  For example, in recent memory, I've worked with two controllers that have the following limitations:

ST7735:
- SPI interface, 10MHz max (or around there, I forget what it is exactly)
- 128 x 160 px typical display size
- 16bpp graphics only
- Single onboard frame buffer
- Arbitrary read or write to a rectangular region of pixels; corners at any pixel location
- Arbitrary direction, up/down/left/right raster within region.

Here you can do single pixel writes, or read modify write (RMW) to take advantage of its frame buffer, but note that there is a huge amount of overhead on changing the location: it's a 128x160 screen so both X and Y are uint16_t.  Setting a new region costs ten bytes, and at 10MHz that's quite slow (so, approx. 100k regions/sec, flat out).  Full motion video is not possible, but partial redraw (small animations, etc.) can be done above refresh rate.

Notice if it were a 16-bit parallel interface, this wouldn't be much of a problem (and indeed such are available, like IL9325).

A nice feature about the direction control is, you can write text left-to-right (or anything else) in order, drawing a column of pixels at a time (which can in turn be streamed from a compression routine, at reasonable savings).

S1D13517:
- 8/16-bit parallel interface
- Up to 960x960 display size (800x480 in my case)
- 16 or 24 bpp graphics
- Arbitrary write to rectangular region, x position in steps of 8px, any y position
- Settable transparent color
- Multiple framebuffers
- Limited compositing ("picture in picture", alpha blend mode)
- Left-to-right, top-to-bottom raster fill pattern only
- No framebuffer reads allowed

The transparent color allows compositing well enough, but the limitation of x address makes writing horizontal text a pain.  If you don't mind using it as a rigid column format, not so bad I guess; but nice, variable-width fonts are a pain, either requiring a ton of overdraw (transparent padding around the arbitrary-sized character) or

No read from framebuffer, means you need a local buffer, or canvas, to do alpha or AA.  (Alpha can be done in limited ways thanks to the hardware, but it's only on rectangular areas (again by the same x mod 8 = 0 limitation) and the transparent color isn't used for the operation, it's simply pixel to pixel.)

The multiple frame buffers, does at least allow seamless double/triple/n-buffering, and also compositing from regions on other buffers, to a target one.  I put like 16 freakin MB of SDRAM on my test board (because why not, it's basically zero added cost), which holds almost 13 full frames at 800x480.  It's more than enough that you could, for example, hold all eight pixel rotations of a given graphic, and composite them sequentially to the destination for smooth (single pixel) motion.



One thing I haven't seen controllers do much of, is paletted (indexed) graphics.  Seems they prefer mostly to store verbatim frame buffers.  I'm sure they're out there, shop around; I haven't come across many though.  So, although you can do some lovely work with them, it may be hard to do so today (short of the hard way, translating it all in software).

Tim

[peter-h]

Really interesting - thank you all.

Not a lot of mention of graphics libraries. LVGL looks viable. 64k of FLASH and 8k of RAM is ok. It sounds like it does what I referred to earlier i.e. you can draw objects made up of polygons and it takes care of the rest.

But also ISTM that just doing it the simple and obvious way is not that hard - if you can get the speed up and have two buffers in the display. As I said I can get 21mbps on SPI (the CPU can do 42mbps on SPI1 but those pins have been lost to other stuff   and anyway few displays can run that fast. DMA is quite easy to set up too if going direct to hardware and not using the bloated HAL code; I got my wave generator working in no time. But a 32F4 at 168MHz can also service a 21mbps SPI by polling very easily if not doing much else.

The concept of "damage" is something I did but never thought about it that way. I implemented a "VT100" terminal once, using the font built-in in the graphics controller. To erase each line, I just drew a rectangle of the right size. It was never fast; that kind of thing just never looked good unless the controller implemented hardware scrolling-up. Fortunately my project will not involve more text than just a few labels in fixed positions.

I asked about displays here
https://www.eevblog.com/forum/projects/small-tft-display-5x5cm-for-moving-graphics-over-spi/
and there is plenty to dig into. The TFTs are amazingly cheap. I don't need a touch screen.

There are some curious things e.g. the ST7789 controller on this display
https://www.buydisplay.com/1-54-inch-tft-lcd-display-ips-panel-screen-240x240-for-smart-watch
has two different SPI max speeds



These displays are not trivial. A 317 page data sheet and it doesn't look like there is any easy way around reading all of it

[Dave8266]

Thank you for any hints.
Obviously I am not building a clock but the graphics complexity is very similar.

If you don't need hundreds of units, you can save yourself a ton of software time/effort by just using a display controller that handles graphics primitives natively.  Then your MCU software becomes very lightweight, I/O decreases dramatically, higher resolutions become possible, etc.

For example, the FTDI/Bridgetek EVE line (EVE2/EVE3/EVE4) or the RAiO RA8875. 

[peter-h]

It's funny to see that basically these are the 1980s chips like the UPD7220 but with a load more functions, and I won't even try to guess whether the RM has more pages than the 32F417 RM which is around 2000 The UPD7220 RM was not merely big but contained references to lots of concepts which were unexplained, which made it hard to do anything with it.

IME the speed of deployment of a device depends much on the size of the RM. The 2k pages of the 32F4 is already crazy and few would use it if it wasn't for ST providing the Cube "code generator" which despite generating massively bloated code does help a lot of people to get started.

I always read the data sheet fully (300 pages for the 32F4) because one has to, to design a working PCB.

I think if I was drawing moving graphics on a VGA+ screen I would need some parallel bus approach, plus a LOT of RAM in the CPU. Fortunately I am looking at around 256x256 only.

[SiliconWizard]

The EVE controllers work very well and will give you antialiased graphics. That can be an option if you do NOT want to do this in software. But something like LVGL will be a much more flexible option.

[peter-h]

As a volume production user of FTDI chips I am not sure I would want to use anything else from them. The company has basically packed up and sold out to some owner in China or Taiwan. There is a guy in the UK (originally the company was in Scotland) who still answers the phone but can't do anything if you need the chips but can't get them. The company is apparently trying to get out of selling chips and trying to move to finished products using the chips.

[peter-h]

This thread was picked up here
https://www.eevblog.com/forum/projects/small-tft-display-5x5cm-for-moving-graphics-over-spi/msg3860261/#msg3860261