Advice on a microcontroller for a camera application

[Boscoe]

Hi all,

I wonder if anyone could give me some advice on which microcontroller family to use. I have a 25MP 12bit per pixel RAW image sensor outputting its data over MIPI CSI2 at an overall rate of 43MP/s or 86MB/s.

I seem to be in a middle ground, here. I don't want to use Linux as it's very complicated for custom HW setups and is too slow at booting for my application. I want to use FreeRTOS or a bare metal application however this is not trivial on the more power A Cortex processors.

So I ask here to see if there's an obscure Chinese or other microcontroller that could fit my needs.

I would like to have:

- MIPI CSI2
- MIPI DSI or an RGB888 interface
- Support for DDR2 or faster
- Compatible with gcc or similar comprehensive toolchain
- Ethernet support

Now the NXP RT crossover processors seem to be a good fit however the RAW12 CSI2 interface is broken and so it only support colour based inputs.

Many thanks for any pointers!

[fchk]

You could use NVidia Jetson Modules. They have plenty CSI and DSI ports, Ethernet, a GPU for image processing, debayering hardware etc etc. You get CPU modules where NVidia has done the difficult things for you - using the modules is not that difficult.

Ok, it's Linux, but it's powerfull. And instead of a full boot maybe you can get away with hibernation etc.

fchk

[rooppoorali]

How about Raspberry Pi Pico?

[rstofer]

You could use NVidia Jetson Modules. They have plenty CSI and DSI ports, Ethernet, a GPU for image processing, debayering hardware etc etc. You get CPU modules where NVidia has done the difficult things for you - using the modules is not that difficult.

Ok, it's Linux, but it's powerfull. And instead of a full boot maybe you can get away with hibernation etc.

fchk

This is a really good idea!  I have a couple of Jetson Nanos and I'm using one with the Jetbot and a camera for learning routes (once I take a lot of photos for the model).  NVIDIA has an ACTIVE forum for Nano related stuff.  They are prolific in the AI department.

I use the Waveshare incantation of the Jetbot:

https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetbot-ai-robot-kit/

I bought an HP laptop with an NVIDIA RTX 3070 graphics chip specifically to get the 5,888 CUDA cores.  That's a LOT of parallelism!

[peter-h]

That MIPI bitrate is huge. You will need something with custom hardware, like the CPUs which have built-in ETH controllers which do all the dirty work and have a dedicated DMA shoving the data into buffers from where you can extract it at your leisure.

Also, unless you structure your code in a very specialised way, your processing will be very slow. Even a 168MHz ARM32 will be slower than you think unless you really optimise it. As an example I am currently getting 140kbytes/sec ETH transmit speed on mine, LWIP, FreeRTOS, FatFS reading of > 1MB/sec. The bottleneck is probably LWIP but it shows how much care you will need.

Cameras all use specialised processors.

[nctnico]

You could use NVidia Jetson Modules. They have plenty CSI and DSI ports, Ethernet, a GPU for image processing, debayering hardware etc etc. You get CPU modules where NVidia has done the difficult things for you - using the modules is not that difficult.

Ok, it's Linux, but it's powerfull. And instead of a full boot maybe you can get away with hibernation etc.

fchk

This is a really good idea!  I have a couple of Jetson Nanos and I'm using one with the Jetbot and a camera for learning routes (once I take a lot of photos for the model).  NVIDIA has an ACTIVE forum for Nano related stuff.  They are prolific in the AI department.

Hmmm, no. NVidia's documentation and support are not even close at the level that NXP or TI are. If you run into problems with NVidia modules, you could end up on your own (even with a decent sales volume) and very little is documented. My suggestion would be to go for an iMX6, 7 or 8 series chips from NXP. There is support for CSI and video in u-boot for the iMX series so that could be something to try if you don't need a full blown OS.