Lattice ECP5: opinion?

[BrianHG]

The limits colored in green for almost every IO on the IC.  That's almost every IO on the left and right side of the device.
Obviously the dedicated transceiver devices & their pins will do the high GHz...

[BrianHG]

Do not underestimate me.  However, yes, wiring to the DDR3 does use one (3 times) of Altera's phy, Lattice's phy and will use one of Xilinx's DDR phys, one of 3 selected by the first line parameter.

I still don't see a point in using your controller over standard MIG. I think you should focus more on Antel and Lattice platforms as they don't provide free one.

Yes, mine has limitations being built around all the different FPGAs, but this is not the point.  I doing this for the puzzle fun and to make something which may not reach Altera's hardware phy on their slower parts, but it has functions which roast their software phy and are more capable than their dedicated hardware phy.  It is designed to operate uniformly across any FPGA and it is properly documented so that it can be adapted to others I haven't written for.  I'm not trying to sell or profit from this.  Having 3 FPGA vendors just demonstrates how the 2 sections which needs plugging into another vendor's compiler & my explanations in the code will allow meaningful changes anyone desires.

[SiliconWizard]

The limits colored in green for almost every IO on the IC.  That's almost every IO on the left and right side of the device.
Obviously the dedicated transceiver devices & their pins will do the high GHz...

I guessed so but I think it's not particularly clear in the datasheet. Where are those dedicated pins described? (And sorry if I missed it!)

And that said, can't you use those dedicated pins for HDMI?

[SiliconWizard]

No no, I was talking about LPSDR as an alternative. I know LPDDR is probably very similar to regular DDR. LPSDR is just SDR SDRAM, but lower power. Much easier to use than DDR. Of course not the same kind of bandwidth, but if you go for large data busses such as 32-bit, or even 64-bit in 2 chips, you can still get a pretty usable bandwidth with them. It will eat up a lot of I/Os though.

You can have a look there: https://www.alliancememory.com/products/low-power-mobile-synchronous-dram-sdr/
or at Micron.

No, that's too slow. Let's hope Brain will design working DDR3 controller by the time the board will actually exist

I guess this was kind of ironical or something. So 1/ no it's not "too slow", it's actually fine for many purposes - and so much simpler; 2/ you seem to doubt BrianHG here, and I agree a DDR3 controller is not a simple task, but it's definitely doable. As I said earlier, others have done it. One may take a look at the controller that's implemented in nMigen. I know it works. Now whether you'd manage in reasonable time to design one that is as efficient as the commercial/vendor IPs, that will remain to be seen.

For point 1/, it really depends on what your use case is. As it happens, the current board I'm using has a SDRAM chip with a 32-bit databus, 166MHz max. Many FPGA dev boards only have a single DDR3 chip with a 16-bit data bus. Although DDR3 has a few access optimizations, the effective throughput will not be that far apart between the two at the same frequency. But the main benefit was that I can actually use it. If it had a DDR memory chip, it would be as good as dead beef to me at this point.

[asmi]

I guess this was kind of ironical or something. So 1/ no it's not "too slow", it's actually fine for many purposes - and so much simpler; 2/ you seem to doubt BrianHG here, and I agree a DDR3 controller is not a simple task, but it's definitely doable. As I said earlier, others have done it. One may take a look at the controller that's implemented in nMigen. I know it works. Now whether you'd manage in reasonable time to design one that is as efficient as the commercial/vendor IPs, that will remain to be seen.

No, I don't doubt him, but it's going to take a while no matter how good he (or anyone really) is. And then there is always a question of what UI is supposed to look like. For example Lattice's DDR3 controller is fairly barebones - you send in address/command, it passes it along, and at some point it asks for data (for writes) or make data available (for reads). There is no reordering, no burst merging, no pipelining, the only "optimization" it's got is that it keeps track of opened rows for each bank, and also sends in bursts of REFRESH commands when required.
Compare that to advanced controller provided with MIG - it's got reordering, request pipelining, in and out FIFOs, and even optional AXI4 slave interface block.

I expect homegrown controller to be closer to the former rather than to latter.

For point 1/, it really depends on what your use case is. As it happens, the current board I'm using has a SDRAM chip with a 32-bit databus, 166MHz max. Many FPGA dev boards only have a single DDR3 chip with a 16-bit data bus. Although DDR3 has a few access optimizations, the effective throughput will not be that far apart between the two at the same frequency. But the main benefit was that I can actually use it. If it had a DDR memory chip, it would be as good as dead beef to me at this point.

DDR3 can be ran at 400 MHz for SG8, which gives 400*2*16 = 12800 Mbps for 16 bit data bus, 32 bit bus doubles that. SDRAM running at 166 MHz gives 166*32=5312 Mbps for 32 data bus, or less than half of what 16bit DDR3 provides. Crucially though, a single 1080p@60 stream requires 1920*1080*32*60/(1024^2) ~ 3800 Mbps, so 32 bit SDRAM will not be sufficient for a framebuffer as you need to read and write at the same time at the very minimum (realistically write bandwidth needs to be much higher if you want to implement any sort of GPU functions). 16 bit DDR3 at 400 MHz is enough for basic double-framebuffer implementation with some bandwidth to spare for CPU cache reads/writes. 32 bit gives you some headroom for GPU functions like layer composition and maybe some basic texturing/3D graphics pipeline, but for advanced GPU functions you will likely need 64 bit DDR3.

[SiliconWizard]

No doubt about using DDR3 @400MHz and its usefulness, but most of the basic dev boards only sport a 200MHz max chip. Also, as you said, Lattice's DDR3 controller is not even that good and the probability of getting/writing something better is doubtful, at least if you don't have years ahead of you and/or big bucks to shell out.

I'm not looking to develop the kind of applications you mentioned, at least not with such a board and an ECP5. So that would be much more modest.

[asmi]

No doubt about using DDR3 @400MHz and its usefulness, but most of the basic dev boards only sport a 200MHz max chip.

200 MHz DDR3? Such chips don't exist. DDR3 standard defines a minimum frequency of 303 MHz, in reality you won't be able to find chip slower than 800 MHz for sale (at least not via official channels).

Also, as you said, Lattice's DDR3 controller is not even that good and the probability of getting/writing something better is doubtful, at least if you don't have years ahead of you and/or big bucks to shell out.

If PHY that Clarity generates is any good, I don't doubt that it's possible to design a full controller yourself. It will take some time for sure, but it seems that Brian is determined to make it happen (and his track record for achieving stuff he wants is excellent if Z80 thread is anything to go by), so if we all help him out a little, we can have something really good in the end. In fact this board might be a good platform for hardware validation of this project.

I'm not looking to develop the kind of applications you mentioned, at least not with such a board and an ECP5. So that would be much more modest.

That's why I think 2 x16 bit DDR3 is the optimal choice. Regular SDRAM is too much of a stone age tech for me to seriously consider.
Another choices are DDR2 or LPDDR1, which are a bit simpler it a sense that they don't require write/read leveling, but they require balanced tree layout for address/command and clock, so no way this can be accomplished on 4 layers. As a matter of fact, at this point I'm not convinced that it's possible to do a DDR3 flyby on 4 layers, but that is definitely easier than balanced tree.

[SiliconWizard]

No doubt about using DDR3 @400MHz and its usefulness, but most of the basic dev boards only sport a 200MHz max chip.

200 MHz DDR3? Such chips don't exist. DDR3 standard defines a minimum frequency of 303 MHz, in reality you won't be able to find chip slower than 800 MHz for sale (at least not via official channels).

You're right, I was probably thinking of DDR2 instead here... Regarding availability, the lowest grade DDR3 available at Digikey is listed as 533MHz, but no stock, the lowest in stock are 667MHz chips.

That said, whatever DDR3 chip you use, I highly doubt you'll manage to get higher than 400MHz on an ECP5. Lattice's DDR3 controller IP states that 400MHz is the highest clock rate achievable, and with the highest speed grade ECP5...

And with a hand-written controller, I wouldn't be hugely surprised if you really struggled to reach those 400MHz.

We're again talking about the ECP5 here. If I want to be able to use DDR3 at 800MHz+, I'll go see elsewhere.

Also, as you said, Lattice's DDR3 controller is not even that good and the probability of getting/writing something better is doubtful, at least if you don't have years ahead of you and/or big bucks to shell out.

If PHY that Clarity generates is any good, I don't doubt that it's possible to design a full controller yourself. It will take some time for sure, but it seems that Brian is determined to make it happen (and his track record for achieving stuff he wants is excellent if Z80 thread is anything to go by), so if we all help him out a little, we can have something really good in the end. In fact this board might be a good platform for hardware validation of this project.

I don't doubt it either, as I said it has already been done (and with 100% open-source tools on top of that.) But it's going to take time, and performance and Fmax are likely going to be a little disappointing.

I'm not looking to develop the kind of applications you mentioned, at least not with such a board and an ECP5. So that would be much more modest.

That's why I think 2 x16 bit DDR3 is the optimal choice. Regular SDRAM is too much of a stone age tech for me to seriously consider.
Another choices are DDR2 or LPDDR1, which are a bit simpler it a sense that they don't require write/read leveling, but they require balanced tree layout for address/command and clock, so no way this can be accomplished on 4 layers. As a matter of fact, at this point I'm not convinced that it's possible to do a DDR3 flyby on 4 layers, but that is definitely easier than balanced tree.

I get it that SDR SDRAM is not for you. But it's still useful for a range of applications, and I for one see no reason to switch to a MUCH complex overall design if I don't need the performance.

Note that I'm now figuring YOU would actually be interested in designing such a board yourself (I remember you've designed other FPGA boards before, the latest being a Spartan 7 one?) It's obvious that if you embark on this project, the choice will be entirely yours. I was just listing what *I* would currently be happy with, but I'm not going to make those choices for you!

[BrianHG]

A sneak peek: (Yes, this is putting my ass on the line)
(I need a few days of concentration to finish, so no questions...)

[BrianHG]

The limits colored in green for almost every IO on the IC.  That's almost every IO on the left and right side of the device.
Obviously the dedicated transceiver devices & their pins will do the high GHz...

I guessed so but I think it's not particularly clear in the datasheet. Where are those dedicated pins described? (And sorry if I missed it!)

And that said, can't you use those dedicated pins for HDMI?

In the Diamond's Pin Planner, clicking on the ram  DQpins, and Show 'Differential Pairs' will show you the IO pins you can achieve 800megabaud.  For HDMI, you only need 4 pairs.  With the NXP IC, we got it to work even ignoring the EDID, but you should still wire those on cheap IOs so you may query the display if you like.

[asmi]

You're right, I was probably thinking of DDR2 instead here...

200 MHz DDR2 are nowhere to be found either. Maybe they were available like 15 years ago (or whenever DDR2 became a thing), but now I think they start at 400 MHz.

That said, whatever DDR3 chip you use, I highly doubt you'll manage to get higher than 400MHz on an ECP5. Lattice's DDR3 controller IP states that 400MHz is the highest clock rate achievable, and with the highest speed grade ECP5...

And with a hand-written controller, I wouldn't be hugely surprised if you really struggled to reach those 400MHz.

Lattice's controller guarantees 400 MHz on the top speed grade for 64bit DIMM, that means there is quite a bit of margin and hence I don't think there will be a lot of problems achieving that since we only want half of bus width and soldered-on memory chips (so traces are much shorter than typical DIMM + connector).

We're again talking about the ECP5 here. If I want to be able to use DDR3 at 800MHz+, I'll go see elsewhere.

Well nobody here talks about 800 MHz DDR3, Artix-7 can't do that either, you will need to go for Kintex and use HP banks to achieve up to 933 MHz DDR3.

I don't doubt it either, as I said it has already been done (and with 100% open-source tools on top of that.) But it's going to take time, and performance and Fmax are likely going to be a little disappointing.

But that's a nice challenge isn't it? Also knowing how controller works inside out will allow using it more efficiently.

I get it that SDR SDRAM is not for you. But it's still useful for a range of applications, and I for one see no reason to switch to a MUCH complex overall design if I don't need the performance.

This will be a serious limitation - you have 1080p@60 HDMI Out, but you can't do a framebuffer due to insufficient bandwidth. If I were to see such a board, this oddity would jump right out. And what are you going to output via HDMI if you can't implement a framebuffer?

Note that I'm now figuring YOU would actually be interested in designing such a board yourself (I remember you've designed other FPGA boards before, the latest being a Spartan 7 one?) It's obvious that if you embark on this project, the choice will be entirely yours. I was just listing what *I* would currently be happy with, but I'm not going to make those choices for you!

I am interested, though I'm leaning towards going all in and using the largest package and a largest density because the top SG parts in 554 package are kind of hard to find in stock. Or going the opposite way - halving DDR3 bus to 16 bit and using FT600 instead of 601. This should leave enough pins for a nice expansion port, though we'll have to think about what kind of connector to use to make sure it won't be yet another pain in the butt a-la FMC.

The fact that I'm doing the design doesn't mean I can't ask other opinions and they might change my mind, and/or find a way to accommodate various requirements as long as they don't conflict.

[SiliconWizard]

You're right, I was probably thinking of DDR2 instead here...

200 MHz DDR2 are nowhere to be found either. Maybe they were available like 15 years ago (or whenever DDR2 became a thing), but now I think they start at 400 MHz.

Pretty much everywhere?

https://www.digikey.com/en/products/filter/memory/774?s=N4IgjCBcoBwAxyqAxlAZgQwDYGcCmANCAPZQDaIATAgLIASAXiALpEAOALlCAKoB2ASw4B5NDTwYcAVwBOeEAF8FRSuRAATdTNXMFQA

But anyway, those are just figures I had in mind. Not saying we should use DDR2 at all.

That said, whatever DDR3 chip you use, I highly doubt you'll manage to get higher than 400MHz on an ECP5. Lattice's DDR3 controller IP states that 400MHz is the highest clock rate achievable, and with the highest speed grade ECP5...

And with a hand-written controller, I wouldn't be hugely surprised if you really struggled to reach those 400MHz.

Lattice's controller guarantees 400 MHz on the top speed grade for 64bit DIMM, that means there is quite a bit of margin and hence I don't think there will be a lot of problems achieving that since we only want half of bus width and soldered-on memory chips (so traces are much shorter than typical DIMM + connector).

Yeah. I'll be curious to see that though.

I don't doubt it either, as I said it has already been done (and with 100% open-source tools on top of that.) But it's going to take time, and performance and Fmax are likely going to be a little disappointing.

But that's a nice challenge isn't it? Also knowing how controller works inside out will allow using it more efficiently.

Sure is.

I get it that SDR SDRAM is not for you. But it's still useful for a range of applications, and I for one see no reason to switch to a MUCH complex overall design if I don't need the performance.

This will be a serious limitation - you have 1080p@60 HDMI Out, but you can't do a framebuffer due to insufficient bandwidth. If I were to see such a board, this oddity would jump right out. And what are you going to output via HDMI if you can't implement a framebuffer?

Or use lower resolutions... (In particular, I have specific ideas using industrial panels that are not Full HD. But other than this, HDMI would just be a bonus. So admittedly for people willing to do Full HD graphics, that would be a problem.)

[asmi]

Pretty much everywhere?
But anyway, those are just figures I had in mind. Not saying we should use DDR2 at all.

I meant to say "Micron parts" as I only ever use their parts for reasons I described elsewhere.
Whatever - it's not really important for this discussion as we're talking 400 MHz max, and there are both DDR2 and DDR3 parts available supporting this (or higher) frequency.

Yeah. I'll be curious to see that though.

I think you have a very good chance to see that Honestly part of me wants to just jump to the 756 package and route a full 64bit SODIMM connector. That package has a ton of IO pins (365), there will be enough for everything.

Or use lower resolutions... (In particular, I have specific ideas using industrial panels that are not Full HD. But other than this, HDMI would just be a bonus. So admittedly for people willing to do Full HD graphics, that would be a problem.)

I'm currently on a contract with one of major automakers (as software developer) supporting their production-related IT systems, and I can say that they use regular COTS 1080p TVs hanged to a ceiling all over the shop floor. So 1080p is an important resolution to support. Also that's the reason I would really like to have 4k support, as 4K TVs now cost dime a dozen, so they recently started deploying those. But that's not going to happen for this FPGA as it requires MGTs, and pretty fast ones at that, at least four DisplayPort HBR2-capable at 5.4 Gpbs per lane, or three 5.23 Gpbs ones for HDMI 2.0.

[SiliconWizard]

I meant to say "Micron parts" as I only ever use their parts for reasons I described elsewhere.

I probably missed it. I know it's not the thread to discuss this - would you have some link to another so I can understand why?

Also that's the reason I would really like to have 4k support, as 4K TVs now cost dime a dozen, so they recently started deploying those. But that's not going to happen for this FPGA as it requires MGTs, and pretty fast ones at that, at least four DisplayPort HBR2-capable at 5.4 Gpbs per lane, or three 5.23 Gpbs ones for HDMI 2.0.

Well, as far as I saw, there are a few HDMI TX over there that support 4K, such as the Sil1136. That would definitely be a case for using one here. Additionally, as I mentioned earlier, I'd be more confident using that as a PHY for HDMI rather than directly route FPGA I/Os to the HDMI connector...

[asmi]

I probably missed it. I know it's not the thread to discuss this - would you have some link to another so I can understand why?

It's very simple really - they provide excellent HDL models for their parts without going through seven circles of bureaucracy, and for that I'm grateful to them. So me using their parts is my way of thanking them for supporting small guys like me.

Well, as far as I saw, there are a few HDMI TX over there that support 4K, such as the Sil1136. That would definitely be a case for using one here. Additionally, as I mentioned earlier, I'd be more confident using that as a PHY for HDMI rather than directly route FPGA I/Os to the HDMI connector...

4k@60 has a pixel clock of over 500 MHz, so ECP5 can't output even parallel pixels (unless you use DDR and half pixel clock) as their IO limit is 400 MHz.

[SiliconWizard]

I probably missed it. I know it's not the thread to discuss this - would you have some link to another so I can understand why?

It's very simple really - they provide excellent HDL models for their parts without going through seven circles of bureaucracy, and for that I'm grateful to them. So me using their parts is my way of thanking them for supporting small guys like me.

Well, as far as I saw, there are a few HDMI TX over there that support 4K, such as the Sil1136. That would definitely be a case for using one here. Additionally, as I mentioned earlier, I'd be more confident using that as a PHY for HDMI rather than directly route FPGA I/Os to the HDMI connector...

4k@60 has a pixel clock of over 500 MHz, so ECP5 can't output even parallel pixels (unless you use DDR and half pixel clock) as their IO limit is 400 MHz.

Well, I do think this transmitter (and some others) supports DDR. So that should be ok using DDR? I didn't count though - maybe you have and have figured there wouldn't be enough DDR blocks inside the ECP5 for both DDR3 memory and HDMI?

[asmi]

Well, I do think this transmitter (and some others) supports DDR. So that should be ok using DDR? I didn't count though - maybe you have and have figured there wouldn't be enough DDR blocks inside the ECP5 for both DDR3 memory and HDMI?

No, I did not look that far yet. The reason I'd rather go with MGT's direct drive approach is that HDMI transmitters have unique challenges to them because of HDCP, so you can't even buy certain parts unless you are a member of HDMI org. But in this case we simply don't have a choice. Another challenge is that 4k@60 stream requires 3840*2160*32*60/(1024^2) ~ 15188 Mpbs of bandwidth, which is A LOT and pretty much requires 64 bit DDR3 if controller can only reach 400 MHz (800 MT/s). That is, unless you are OK with streaming lower-resolution image from memory and upscaling it in real time. So for this board I think 4K is going to be a bit too much because other components are going to be the limiting factor, even if we ignore the memory for a moment - what exactly are you going to display, and where this data is going to come from? Also part availability might be a problem - for example that Sil1136 you've mentioned is not stocked by Mouser nor DK as far as I can tell, so it might as well don't exist for practical purposes.

[SiliconWizard]

The Sil1136 is available at Digikey: https://www.digikey.fr/product-detail/fr/SII1136CTU/220-2183-ND/6596585
The only transmitter I coud find at Mouser is the TDA19988, but it's much more limited - it can do Full HD, but not 4K.

But yes anyway, not sure what you would do with 4K@60 on such a board? That goal may be a bit too much.

[BrianHG]

DDR3 update: https://www.eevblog.com/forum/fpga/fpga-vga-controller-for-8-bit-computer/msg3497050/#msg3497050