Fast data transfer between FPGA (Xilinx Spartan 6) and 30 microcontrollers

[GeorgeOfTheJungle]

To send the PWM values to the µCs through a single comms channel, you could do something similar to DMX512: send all the PWM values in order, one after the other, in a single frame, and every µC reads the data that corresponds to its slot/µC ordinal/id number.

In the other direction, to send the ADC values from the µCs to the FPGA with a single comms channel you'd need something like that too, but it's not easy to do because every µC would have to insert its data at its slot and there's no built-in ad-hoc comms peripheral to help with that. Ethernet can't do it, CANBUS can't do it, spi, i2c, uart/serial etc, aren't designed to do that. It can be bodged in software with a serial channel and interrupts at low(er) speeds, but there's no (sane) way to do it at 50 Mbit/s.

I think you can do with only one channel to send data from the FPGA to the µCs, but you're going to need more than one comms channel to send the data that goes in the other direction (from the µCs to the FPGA), or help from additional, dedicated hardware.

[NorthGuy]

But I want to have the number of possible cells as big as possible. I will build it with 30 cells.

Just imagine you wanted to sell something on eBay and, of course, you would want to get as much money for it as possible, so instead of giving a price you would list it for "as much money as possible". Designing for as many cells as possible is equally silly.

You first need to decide on the maximum number of cells you're willing to support and then you design everything else around this number. Since your control block is already designed and made, what is the maximum number of cells it can handle? Is this acceptable for you?

[Momchilo]

You first need to decide on the maximum number of cells you're willing to support and then you design everything else around this number. Since your control block is already designed and made, what is the maximum number of cells it can handle? Is this acceptable for you?

The control block could handle basically, infinitely many. That's the whole point of this multilevel converter concept. The number of cells doesn't matter for the DSP. The FPGA could handle many thousand cells. There is only one limitation in the number of cells and this is the data transfer, which increases with every added cell. Either the I/O pins of the FPGA are the limitation (without using MCUs) or the data rate from the bus.

[PCB.Wiz]

You seem to have not looked at separate links ? -  if you send 16b each way in 10us, that's a fairly modest 1.6Mb/s duplex link.

That would be for one cell. But there are 30 and it should work with more cells. The faster the better.

Yes, the idea is you add more simple but  identical links, rather than try to connect all those slaves onto a single bus.  KISS is usually better.
FPGAs have plenty of pins.  Each MCU only needs 3 or 4.

Search Digikey for LVDS, Multipoint and you will find a list of suitable cable driver candidates, that can do half or full duplex.

[nick_d]

Based on the previous thread and this thread it would seem that the design is not really set in stone at this stage. It would be a good idea to go back a few steps and explain what you are really trying to achieve. Wikipedia has an article on HVDC converters which went some way to enlightening me, but it talks about thyristors, whereas I understand that you're making a multilevel modular converter which is apparently similar but uses transistors?

Are the levels isolated? In that case, how are you planning to get your bus isolated? And would it not make sense to use a separate wire(s) for each level in such case?

What we really need to know is the "givens" of the design. Sometimes these are set by customer, sometimes by hardware or software or physical considerations, sometimes by politics... but the proposed designs (FPGA, 30 uCs, ADCs etc) are from what I understand, not givens. It's certainly good to try to solve the problem yourself, but this shouldn't unduly constrain the advice asked for/received.

cheers, Nick

[NorthGuy]

The control block could handle basically, infinitely many. That's the whole point of this multilevel converter concept. The number of cells doesn't matter for the DSP. The FPGA could handle many thousand cells.

The biggest Spartan-6 has 180 DSP blocks. If you run at 200 MHz, this is about 3 million operations per your 12 kHz cycle (if you manage to utilize every DSP at every cycle). If you have 3000 cells, it's only 1000 operations per cell - top max - is this enough?

If you only need 1000 operations per cell per cycle, it's roughly 12 MOp/s per cell, which can easily be done by rather small CPU. May be you can off-load this (entirely or partially) to MCUs and decrease traffic?

[Momchilo]

Thanks a lot for all your help. I'm happy to share my thoughts. I'm not an expert, I'm learning. Please point out every possible mistake/oversight I made and make You learn from mistakes, but there will be several, few less would be nice

Based on the previous thread and this thread it would seem that the design is not really set in stone at this stage.

You're right, everything except the control board (DSC and FPGA) can be changed. 

Wikipedia has an article on HVDC converters which went some way to enlightening me, but it talks about thyristors, whereas I understand that you're making a multilevel modular converter which is apparently similar but uses transistors?

Exactly. Here you can see the output of one phase. With every additional level (each level is built by 6 cells; 1 positive, 1 negative for every phase) the voltage steps become smaller at the output. With enough levels, you have a superb sinusoidal output without harmonics and additional filters (big expensive heavy inductors) aren't needed. Few years ago the breakdown voltage wasn't high enough of transistors and/or R_on was too high, instead they used thyristors. But with modern IGBTs it's possible to use transistors.

Are the levels isolated? In that case, how are you planning to get your bus isolated? And would it not make sense to use a separate wire(s) for each level in such case?

Each level (more precisely every cell) has an own floating power supply for the ICs. The measured voltage is galvanic isolated (optical) and the gate driver is also galvanically isolated. Only the digital side of each level/cell has a common ground with the control board. 

What we really need to know is the "givens" of the design. Sometimes these are set by customer, sometimes by hardware or software or physical considerations, sometimes by politics... but the proposed designs (FPGA, 30 uCs, ADCs etc) are from what I understand, not givens. It's certainly good to try to solve the problem yourself, but this shouldn't unduly constrain the advice asked for/received.

The DSC and the main FPGA are given. It's highly desired to use a MCU or FPGA on every cell with a shared bus/buses, but if not possible without extremely high effort, other options are possible.

For the ADC with the required isolation an ACPL-C87B (Optically Isolated Voltage Sensor) and an ADC (ADS8320) with differential inputs could work.
But there aren't finished boards or even layouts. So everything can be changed with the exception of the control board.

If you only need 1000 operations per cell per cycle, it's roughly 12 MOp/s per cell, which can easily be done by rather small CPU. May be you can off-load this (entirely or partially) to MCUs and decrease traffic?

Less traffic than receiving one measured voltage per cell and send one pwm signal to each cell isn't possible. There is no MCU with at least 30 PWM outputs. Both values are critical. The cell voltage is critical for the control and for emergency stops. All PWM signals are based on the cell voltages, phase currents and leg currents (3x positive, 3x negative). 

[NiHaoMike]

Have the FPGA directly generate the PWM signals and interpret PDM, to allow the use of simple isolators. TI makes isolated ADCs specifically for that purpose.

[Momchilo]

Have the FPGA directly generate the PWM signals and interpret PDM, to allow the use of simple isolators. TI makes isolated ADCs specifically for that purpose.

Yes, this would be possible or isolated Delta-sigma modulators like AD7401 with the demodulation in the FPGA. But if possible I would use some kind of busses and separate MCUs/FPGAs.

I will look into all your suggestions. But I need some time to do this

Best regards
Momchilo

[free_electron]

DMA bulk transfer linked to an external clock.
the FPGA supplies the external clock.
A frame is fixed length.
you use a wired-or mechanism (READY signal)
once all microcontrollers are ready the FPGA begins transport. The first clock cycle after ready generates an interrupt to the microcontrollers that they can start acquiring new data. Once they are done acquiring they need to wait for the DMA transfer to complete. once complete they load the memory with new acquired data and set the ready flag.

so in flowchart

cpu's acquire 30 readings .
cpu copies 30 readings into DMA channel.
cpu sets READY HIGH and waits for CLK to go high.

when all CPU's are ready the ready pin will go high. ( the slowest one is the lst one to release the 'ready pin'. ready pin is open collector )

the FPGA waits for ready to go high.
if ready goes high the FPGA starts clocking.

the first rising edge of the CLK the CPU clears the ready pin ( makes it low ) and starts a new sequence of acquiring measurements into a buffer.

After the last transport clocktick the DMA controller will notify the cpu 'transport complete'. the cpu now copies its buffer into the DMA block and sets its readay pin high.

the cycle begins anew....

This gives you 2 control signals ( CLK and READY ) and one data line from CPU to FPGA.

you can use the DMA controller in conjunction with a USART ( not a UART , you need the SYNCHRONOUS type , most CPU's have such a mode where the send in synch with an external clock ) Most modern cpu;s can do this under DMA control. here is a block of bytes : send using usart under control of clock.

Minimal wiring , minimal code, everything synchrounous and self timing.

[PCB.Wiz]

Exactly. Here you can see the output of one phase. With every additional level (each level is built by 6 cells; 1 positive, 1 negative for every phase) the voltage steps become smaller at the output. With enough levels, you have a superb sinusoidal output without harmonics and additional filters (big expensive heavy inductors) aren't needed. Few years ago the breakdown voltage wasn't high enough of transistors and/or R_on was too high, instead they used thyristors. But with modern IGBTs it's possible to use transistors.

Each level (more precisely every cell) has an own floating power supply for the ICs. The measured voltage is galvanic isolated (optical) and the gate driver is also galvanically isolated. Only the digital side of each level/cell has a common ground with the control board. 
..
Less traffic than receiving one measured voltage per cell and send one pwm signal to each cell isn't possible. There is no MCU with at least 30 PWM outputs. Both values are critical. The cell voltage is critical for the control and for emergency stops. All PWM signals are based on the cell voltages, phase currents and leg currents (3x positive, 3x negative).

What charges these cells (batteries?) ? What appx voltage are they ?
There were Audio Amplifiers that used multiple power rails, and switched them in and out to lower the power losses, but I think they rather faded with Class-D technology.

You mention 16b PWM, but do not show exactly what that controls ?
Does each cell have an IGBT Pair and an inductor, and the PWM sets the % of full cell out ? 
How do the ON and OFF energies flow ?

If they switch at ~12KHz, that indicated ~800MHz of edge placement, which limits MCU choice more, so you may be better sending modulation info from the FPGA, and skip the local PWM entirely.
I'd expect you want to have tight control over every edge, not have some serial link added into the mix.

A device like EFM8LB1 could read 14b of Analog, and transmit serially at 18MBit over a Digital Isolator, for ~1.1 us link times.

The TI isolator ISO7721 can manage info flow both ways, and specs typical skews of 0.5ns, for precise edge control.

[dmendesf]

Seems you're doing a multilevel inverter. I helped make one before. Used a uC for each cell + 485 bus to drive them from a stm32. Used 3 buses, one for each "leg" (phase).

[David Hess]

You seem to have not looked at separate links ? -  if you send 16b each way in 10us, that's a fairly modest 1.6Mb/s duplex link.

That would be for one cell. But there are 30 and it should work with more cells. The faster the better.

Yes, the idea is you add more simple but  identical links, rather than try to connect all those slaves onto a single bus.  KISS is usually better.
FPGAs have plenty of pins.  Each MCU only needs 3 or 4.

Search Digikey for LVDS, Multipoint and you will find a list of suitable cable driver candidates, that can do half or full duplex.

I agree with the above except for the need for dedicated drivers and receivers.  The FPGA has plenty of I/O to divide the MCUs into individual channels or small groups.  The much lower data rates, 16 bits in 10 microseconds is 625 nanoseconds per bit, combined with the transmission line delay of roughly 3 nanoseconds considerably relaxes the driver and receiver requirements if some care is used.

[PCB.Wiz]

I agree with the above except for the need for dedicated drivers and receivers.  The FPGA has plenty of I/O to divide the MCUs into individual channels or small groups.  The much lower data rates, 16 bits in 10 microseconds is 625 nanoseconds per bit, combined with the transmission line delay of roughly 3 nanoseconds considerably relaxes the driver and receiver requirements if some care is used.

Yes, the OPs' original post did not mention any voltage offsets, but later posts suggest this certainly will need voltage isolation.
In that case, the TI isolator ISO7721 I mentioned is likely good enough, and LVDS would not be required.
LVDS-M  was suggested from a view of EMC and noise rejection, as much as 'can it work', plu sit is lower power then RS485 alternatives.

[NorthGuy]

I think you overcomplicated things a bit.

I don't think it's a good idea to isolate ADCs. I would have few ADCs with ground on the bottom rail, which would be constantly measuring voltage at all the necessary points through resistive dividers. Each ADC can handle several channels. Even one ADC with a high sample rate would do. There are MCUs which have enough channels and speed for adequate sampling.

Then I would have wires going through optoisolators to switches.

So, a cell would contain optoisolators, switches, voltage divider for ADC, temperature sensor and interface.

IMHO, an MCU could do all this quite well. Certainly won't be a problem for FPGA.

[Marco]

A resistive divider might be too slow depending on the voltages involved.

[nick_d]

About multi-drop buses:

10 Mbit/s Ethernet, PCI and PATA are examples of multi-drop buses. They originally ran at 10 MHz (Ethernet) or 33 MHz (PCI, and intermediate versions of PATA). Enhanced versions have gone up to 66 MHz (PCI) and even 133 MHz (PATA with the special 80 pin ribbon cable). In the PCI case, it's all within the same PCB, which makes things a lot easier. PATA uses a ribbon cable. So in theory, if they can do that, you can do it too. However I suspect that some black magic is involved, I personally wouldn't vouch for a multi-drop bus running over any kind of cable at more than 10 MHz. But, remember this is MULTI-DROP. If you go to point-to-point links, the rules change.

100 Mbit/s Ethernet, PCIE and SATA are examples of point-to-point buses. By using a point-to-point link, you have much better control over the inductance of the link and issues like termination and reflection. This is good for 100 MHz without significant problems, even several Gbit/s if special techniques are used. However, note that you should really use dedicated transceivers for this kind of thing, if you just took a 74AC gate and tried to send 100 MHz data over it with a long cable, it would be problematic for a number of reasons. Noise pickup, ground issues, etc. So you should definitely be going differential, and preferably it should be isolated, as in Ethernet.

Now with that background, considering your problem, it would be 100% easier and better to have a separate connection or connections from your FPGA to each converter level. You've specified 100 MHz, and this just won't work well over a shared cable. Even if you used special transceivers and cables to make it work, there is also the problem of the connector, which is always going to add a lot of stray impedances, and a loose connector could bring the whole system down. If there are wiring considerations that make the single cable approach unavoidable, the best I could suggest would be a daisy-chain, that is, a series of point-to-point links (repeaters) from each level to the next.

Another way to do the multi-drop bus (if it absolutely must be a multi-drop bus) would be to put everything on the same PCB: the DSP, the FPGA, and the 30 levels. It would be a big PCB but quite workable. If you have a multi-drop bus WITHIN the same PCB, then with good layout techniques you could make it work up to 100 MHz. However, there'd be no point because the PCB would allow a star topology just as easily as the multi-drop bus.

So clearly the right thing to do is a star topology: reserve a couple of FPGA pins for communicating with each converter level, and have a big bundle of wires going to the level PCBs. Given that modern FPGAs have hundreds of pins, it would not place significant limitations on your design. And, the data rate would be so much reduced (kbit/s not Mbit/s), that no special techniques would be required, you could just connect the FPGA pins directly to the cable with say a 1k resistor and potentially a couble of diodes (one to each rail) for protection.

About isolation:

Isolated ADCs are going to be very expensive. Isolated PWM output slightly less so, since in general the high side driver needs to be isolated anyway, but even so, it would be much easier to put the isolation in between the micro and the FPGA, rather than between the micro and the PWM + ADC. A good way to isolate the communication between FPGA and micro would be to do it like the old MIDI standard: an opto-isolator is placed on the receiving PCB, and the sending PCB gets a pair of wires connected to the opto, which it can use to turn on the LED of the opto. (The opto can also be placed on the sending PCB and the open-collector output sent along the wire to the receiving PCB).

Another approach might be to use fibre-optics instead of wires. So basically the LED is on the sending PCB and the receiver (light detector) is on the receiving PCB. Look up S/PDIF with TOSLINK connectors for an example of this in practice.

Also, the micro could possibly be deleted if you use the right ADC. Certain kinds of ADC only require a clock input, and will regularly initiate a conversion and clock out the results. For example, many audio ADCs work like this, and can send the results in S/PDIF over a TOSLINK connector. I don't suggest to use an audio ADC as such, because they often can't handle DC and you need to measure the DC, but I am sure that an appropriate ADC could be found. If the conversions need to be synchronized, you'd need an extra opto (or fibre-optic cable) to distribute the conversion clock. As to the PWM, how about having one opto (or fibre-optic cable) for the low side and another for the high side, it has the advantage of simplicity and that the FPGA can generate the needed dead time, but may be unsafe as both can be turned on simultaneously. Or possibly just have an opto letting the FPGA drive the low side, and generate the high side with appropriate dead time locally based on what you see at the low side.

cheers, Nick

[Rerouter]

I'll start out simply, you want to do as much as possible in the least amount of effort possible

The simplest way would indeed be to give each micro its own SPI line, dedicated comms, and the data rate can be very high, while keeping to really cheap micro's, It would only take 16 SPI clocks to cycle in and out the values you needed, as long as its a 16 or 32 bit device, you write your ADC value in the SPI register, then as it writes in the new PWM value it clocks out the current ADC reading

Next step up involves the same design, but with cost reduction in mind. have a SPI ADC and a 8 pin Micro with an SPI port and a timer that can handle your desired frequency and resolution.
So you have 5 wires running through an isolator of your choice. SCLK, MOSI, MISO, CS, PWM_CLK, 4 in, 1 out, You have the ADC and the Micro share the Chip select line, but with opposite activation states, Select the ADC, clock out the reading, Select the Micro, clock in new PWM value.

The second option would be a pair like so
Micro: https://au.element14.com/microchip/attiny402-ssnr/mcu-8bit-avr-20mhz-soic-8/dp/2915533
main pain then is just finding the ADC, 12 Bit is easy, but 16 bit at low cost is a right pain, Is there a reason for the 16 bit requirement? its seriously hard to layout circuits to make proper use of the last bit.

[vealmike]

Sounds like a job for Daul / Quad SPI, one link to each endpoint.
For the processors, take a look at the PSoC. The integrated CPLD will allow you to have minimal processor intervention in your control loop and to implement your own FPGA comms if necessary.

[PCB.Wiz]

A resistive divider might be too slow depending on the voltages involved.

Yes, plus you lose precision, and a local MCU is likely needed for failure handling and overcurrent reaction. MCUs are very cheap these days.
Of course, a host side MCU could be added as well, as another system-monitor task, but it's unlikely to be fast enough for control and protection.

[NorthGuy]

A resistive divider might be too slow depending on the voltages involved.

Yes, plus you lose precision, and a local MCU is likely needed for failure handling and overcurrent reaction. MCUs are very cheap these days.
Of course, a host side MCU could be added as well, as another system-monitor task, but it's unlikely to be fast enough for control and protection.

You need a voltage divider anyway. As to the resolution, you're unlikey to get accuracy better than 0.1%, which, in case of a 16-bit ADC, leaves you 6 bits of extra resolution.

But there's a huge advantage. When they're all referred to common ground you don't need isolation whether you use one MCU or a separate MCU for every cell.

[Marco]

A resistive divider is probably an option at 1kV, probably not at 10 kV.

[Scrts]

Sounds like a job for Daul / Quad SPI, one link to each endpoint.
For the processors, take a look at the PSoC. The integrated CPLD will allow you to have minimal processor intervention in your control loop and to implement your own FPGA comms if necessary.

QSPI over 0.5m cable will end up with garbage. Been there. We had to use LVDS drivers/receivers for such application and it still wasn't 100% robust.