Where to put the data?

[sgpee]

Fellow masters,

here is a rookie, asking probably a stupid question.

- I have a system that generates data, roughly 60KB/sec (coming from peripherals, industrial application)
- I need to ship this data to a PC using USB.
- For some reason, I need to align this data in my MCU every second to a correct format, add time stamps etc. before ship out to PC.

Now, the MCU I am using has only 8KB of memory, ideally, I would have written all data to RAM and fiddle with it and than ship to USB buffers and problem solved, however not in this case since I have a tiny RAM. (Or change PC software to fix this, however PC software is legacy and customer is not interested in me fiddling with PC).

I have some flash memory available that I can use as a circular buffer to write and read etc, however I find this not very elegant as I am afraid to miss data during this long arduous write operation as the data keeps coming. Now, I am considering couple of options and I wanted to get some ideas:

- Use SPI ram 1Mbit or so: There is a problem here as I use two available SPI ports to collect data from peripherals. If I multiplex the SPI, I am afraid I may loose data. (Even a  single bit should not be lost, the peripheral just puts data to fifo, if you run the fifo, data is gone). Question: How do I calculate whether I run out of BW and loose data here?

-  Use a second processor: Dedicate this one for data collection. Choose this one wisely. Size is a concern here and I cannot tolarate 60-70pins, need to be tiny and a lot of ram etc. Couldn't find one yet?


Anyway, hope you guys could offer some help.

Cheers, sgpee

[alm]

Now, the MCU I am using has only 8KB of memory, ideally, I would have written all data to RAM and fiddle with it and than ship to USB buffers and problem solved, however not in this case since I have a tiny RAM. (Or change PC software to fix this, however PC software is legacy and customer is not interested in me fiddling with PC).

Can you modify the algorithm to be streaming (i.e. process every 'record' as it comes in, not wait for a full second of data)? Otherwise, the cheapest and simplest solution would probably to change to an MCU with plenty of RAM, it's not like that's expensive. It sounds like you have to modify the software and board layout anyway.

- Use SPI ram 1Mbit or so: There is a problem here as I use two available SPI ports to collect data from peripherals. If I multiplex the SPI, I am afraid I may loose data. (Even a  single bit should not be lost, the peripheral just puts data to fifo, if you run the fifo, data is gone). Question: How do I calculate whether I run out of BW and loose data here?

Can your FIFO's run at a higher SPI clock, or are they limited to 60k*8 Hz or so? How long until your FIFO overruns? Can you ship the data in and out during this time at the highest clock rate the devices allow?

-  Use a second processor: Dedicate this one for data collection. Choose this one wisely. Size is a concern here and I cannot tolarate 60-70pins, need to be tiny and a lot of ram etc. Couldn't find one yet?

I'd look into an ARM Cortex M3 controller (eg. NXP LPC1xxx, or ST M32). Getting these with some horse power/RAM is usually cheaper than getting a top-of-the-line 8-bit MCU. Use some tiny TQFP package if size is an issue. However, I'd look into replacing the first processor before adding a second, since two processors would increase complexity, and it shouldn't be an issue to do this with one (appropriate) MCU.

[Psi]

It does sound like you're expecting a bit much from what sounds like an 8bit mcu with 8KB sram.

There is the storage issue with trying to handle 60KB/s data with only 8KB of ram but also 60KB/s is 480,000 bits a second, so your data is coming in at around half a million bits per second.
Even if your 8bit mcu was running at 20MHz (depending on mcu type and instructions per clock cycle) your only going to get
something like 10 -> 40 cpu instructions per bit of data.

Of course you can load an entire 8bit port of data in one instruction if your I/O is matched nicely between the mcu and external sensors and doesnt need to be bit merged back into useful variables in software.

And on top of that you need to convert the data format and add time stamps.

[mikeselectricstuff]

Use SPI ram 1Mbit or so: There is a problem here as I use two available SPI ports to collect data from peripherals. If I multiplex the SPI, I am afraid I may loose data. (Even a  single bit should not be lost, the peripheral just puts data to fifo, if you run the fifo, data is gone).

It's trivially easy to do SPI on normal IO ports - much easier than sharing a hardware SPI.

[Zad]

If you have to stay with that micro, stream it to a big cheap SD card over serial. I would be going the ARM/Cortex method though, it will be cheaper and faster all round.

[migsantiago]

If you use an HID USB interface you can transmit data at speeds up to 64kB/s.

The data is transmitted in packets which hold 64 bytes and that are sent every 1ms. You only require 64 bytes of RAM in order to send 64kB/s.

There's no need to have that huge amount of RAM you're looking for. 

[TechGuy]

How many free I/O Pins do you have on the MCU? You might be able to add a small SRAM chip to store the data. That would have pretty quick I/O. You need at least 21 free I/O pins (8 for databus and another 12 for Address for 4K and another bit for R/W). To read/write you would have to set the Address bus in your code then set the R/W pin and then Read or Write the data.  If you can tolerate a few extra instructions per I/O cycle, and room for a couple of external logic gates you can use a shared Data/Address bus. You use a an a pair of 8 bit registers (ie 74HCT273) to hold the address data, and write to it using a 8 I/O pins. You would only need 8 I/O + 2 ADDR Latch pins (ADDRHI ADDRLO) and RAM R/W for a total of 11 I/O pins. Memory I/O would require you to Set the Low Address bits each cycle, and the high Address latch every 256 bytes. BTW: This was the method the Intel 8086 processor used. It had a shared 16 bit data/address bus so that the pin count could be kept to only 40 pins.

I would consider going with a beefer MCU, you can get 32bit MCU in 64pin QFN or 64-pin TQFP packages. The QFN is only 7 mm wide, which is pretty tiny, which I suspect would have a smaller PCB footprint then your existing MCU.

PIC32MX695F512H (64 pin QFN/TQPF)
80Mhz, 512KB Flash, 128KB RAM, 3 SPI, 1 USB 53 I/O Pins,
http://www.microchip.com/wwwproducts/Devices.aspx?dDocName=en545654

[sgpee]

Thanks masters.. One thing that become apparent is 8 bit MCU i have is probably not going to be able to handle the tasks. So, I have a different question, which ARM to use. I will post a new message asking that..

Thanks a lot.

Best, sgpee

[sgpee]

Ok.. A follow up question. I have come across at ST's web site that there are a lot of spi and i2c EEPROMs with various sizes. My application requires 50K lifetime read/write, this is above flash limits but for EEPROM should be a piece of cake to handle.

The question is speed, if I use 400Khz I2C, what type of utilization I should expect 70%? How about for SPI? (datasheet says 16Mbit is supported)

Do you guys have experience with this?

thx,
sgpee

[migsantiago]

An EEPROM writes a byte every 3-5ms. They also write an array of bytes (page write) at the same speed. Read the eeprom datasheet and find out if that timing is acceptable for you.

[TechGuy]

Thanks masters.. One thing that become apparent is 8 bit MCU i have is probably not going to be able to handle the tasks. So, I have a different question, which ARM to use. I will post a new message asking that..

Thanks a lot.

Best, sgpee

First see if you can afford the IDE cost for ARM software development\programming ($3K to $6K)

[Mechatrommer]

arm is overkill from the op description. and funny he want it cheap. the problem is he didnt mentioned his detailed mcu spec. arduino mega can easily do this, if i understand it correctly.

[sgpee]

You may very well be right..  The problem is I have no sense of ARM vs. 8bit. I have never done code at 8 bit but I have worked with complex SOCs with ARM9 and plety of memory and small OS. etc. This case, I don't even want an OS. So, I rather err on the side of CPU horse power than to find out I have to optimize my code to death to get "ok" performance out. Similar for the RAM. I just don't know how to model without doing the actual implementation. Do you have any suggestion on the modelling side?

sgpee

arm is overkill from the op description. and funny he want it cheap. the problem is he didnt mentioned his detailed mcu spec. arduino mega can easily do this, if i understand it correctly.

[Mechatrommer]

Similar for the RAM. I just don't know how to model without doing the actual implementation. Do you have any suggestion on the modelling side?

whats your incoming data rate/bw? i understand there are two of them. if you can explain more true nature of your collected data, it can give us better picture. so far i'm half half so so.

[sgpee]

I have 3 ADCs. First two are at 1Khz @16 bit, the third one is 2Khz @ 12 bit. These are my data sources.

Due to legacy on PC side, I need to collect the data for about 2 seconds before I ship it. Therefore, I need to store it somewhere. uP RAM is the obvious source. I also need to run my program (uP has a flash so code is flash but all other stuff, variables etc will be at RAM).

Thx,
sgpee

[Mechatrommer]

assuming the 2 1khz is in spi, and the 2khz adc is handled in io, there are 8000 ins/op to handle that (16mips), i'm not sure about the bps, but assuming it is continuous in the 2khz time, then its 2*12 = 24Kbps, 666ops to handle each bit. i think its alot of code and room, i'll try arduino first for the challenge. i'm not sure about exernal ram/flash design as i never done that, but i think synching read/write ram between collecting and sending mcus will need a carefull thought, but i think do it all in arduino (collect/send) is still doable. nothing beats doing it and finding out. collecting and sending algol code, should be stupidly simple, even if including delta compression discussed in another thread

[Mechatrommer]

for more bulletproof system, you can try using all the serial port available, ie spi, uart, mega got more than 3 iirc.