BrianHG_DDR3_CONTROLLER open source DDR3 controller. NEW v1.60.

[BrianHG]

That CE path is cycled once during system reset.

Ok,, this looks like the way to go, so long as we can make the 'delay' primitive work for us.

Right now, the DDR3 controller will function, but, the write data phase will be too soon, and the read data phase will be un-tunable.  You need to find out how the delay element works and what is its range to see if we can make it fit our needs.  If good, and you want to make a tiny module to software tune it controlled by my phase_step and phase_updn, then this will be the next step just before you check to see if you can shrink your PLL down to one unit.  Doing so may further improve your FMAX range eliminating those 2 minute timing errors at 432MHz.

And finally, other than the IOs and PLL tuning limitations, we can now say that my DDR3 code can achieve the necessary 400MHz rates.


If tuning the delays cannot be done, then there is the last option to decode Gowin's DQS mem-phy clock system and use it to auto generate the read clock phase.

(Wait, did you say fast cold model?  Actually, we only are interested in the Slow Hot 85degC model.)

[SpacedCowboy]

I think I added the 'fast cold' model when I was trying everything to try and get something to work. I can take it out and see where we land, but it might not be pretty

Also remember there's no input/output delays in the SDC right now.

Ouch. Yeah...

[BrianHG]

Set the PLL for ~300MHz and see what happens.
Show me the setup slack log.
Also, try the -8 or -9 chip.

[SpacedCowboy]

Ok, yes, that is a lot closer.. On 'slow' I'm getting the attached. I get identical results for grades 8 and 9, oddly enough. And I re-ran it a couple of times, clearing the data in-between runs, just to make sure that was the case.

[SpacedCowboy]

FWIW, It's still inferring 3 RAMs, and I haven't found any way to stop it from doing so as yet.

Extracting RAM for identifier 'PIN_OE_WDQ'("ddr3_io_port_gowin.sv":184)
Extracting RAM for identifier 'bank_row_mem'("BrianHG_DDR3_CMD_SEQUENCER_v16.sv":117)
Extracting RAM for identifier 'vector_pipe_mem'("BrianHG_DDR3_CMD_SEQUENCER_v16.sv":180)

Funnily enough, I seem to recall reading that the fMax of the RAM was 320MHz. Ah, no not quite right, see attached.

[BrianHG]

The RAMS are being infered at the 0.50clk freq, 162Mhz, so, you are safe.  Except for the OE path which is running at the 324MHz.  However, none of your failed paths are in the 324MHz domain to the DQ DDR port like before.

This is something we can work with as the worst 6 paths at ~ -0.2ns is part of a combinational logic block to decide when to output the next DDR3 command.

Do you have a compiler option to adjust the effort for placement and routing.
Can you enable or disable any smart state machine or mux options.
Auto infer options for infering clock enable or other smart timing analysis during compile to improve FMAX.
Like auto duplicate or re-time registers to improve FMAX and slack?

Try going down to 1 pll if it can be made to just generate the DDR_CLK, _50 & _25.  This should help improve routing between the _25 and _50 domain maybe giving you a little more breathing room.

Also, remove the _RDY <-> _WDQ false_path in your .sdc.  Maybe that is why we cleared the timing way too easily.

[SpacedCowboy]

Compared to other synthesis/P&R options, the Gowin one is very limited. Attached is every option available in settings...

• We're already running with Timing-driven synthesis, I can't believe it gets better without.
• The 'Place' option of '1' is basically 'Place option 0, except try a bit harder'. The only options are 0 and 1
• The 'Route' option of '1' is 'Route according to timing', 0 is according to congestion, and 2 is 'make it take as little time to route as possible'. Those are the only options

In general, the '1' options seem to produce the best results so far. None of them are what you might call intensive, From-scratch synthesis, P&R, and bitstream generation takes ~13 seconds in all

The only thing that approaches smart state-machine/mux options is "gray" | "onehot" in

Code: [Select]

verilog object /* synthesis syn_encoding = "setting_value" */;

In the same docs, there's mention of a

Code: [Select]

Verilog object /* synthesis syn_tlvds_io = setting_value */;
.. but I'm instantiating using primitives, so that's not really relevant. There's also dspstyle, but I doubt you're multiplying much [grin].

Overall, it seems to be very much 'click-and-go synthesis' as opposed to the fine-tuning you can do with the major players.

There's a control for fan-out, and hopefully it'd do duplication if the fan-out was exceeded, but I tried that earlier (reducing the '23' default to '10') and it actually got worse timing....
 

[BrianHG]

FWIW, It's still inferring 3 RAMs, and I haven't found any way to stop it from doing so as yet.


Extracting RAM for identifier 'bank_row_mem'("BrianHG_DDR3_CMD_SEQUENCER_v16.sv":117)

Un-infering this reg from ram will probably make the difference.
The other 2 you do not need to worry about as they have double write / read latching.
Bank row mem is being used directly with logic without such a read stage latch.  This is why I have the attribute (*preserve*) which in Quartus means to preserve the logic register and not infer anything at all like a ram block.

You will need to find Gowin's equivalent to prevent the ram infer of the logic bank_row_mem array..

[SpacedCowboy]

Ok, so it's now definitely not using block-rams. I can't be absolutely sure that it's not using what it calls 'shadow ram' (using the LUT as an SRAM) but it ought to be using registers now - the directive is /* synthesis syn_ramstyle = "registers" */

Looking at the resource usage I can see there's 10 of those SSRAMs in use, and no BSRAMs (the block-rams). Adding that directive took the register usage from 1832 to 1904, so I think it must be having the right effect.

We still don't quite make 324MHz (but we get 317) and we don't quite get 162MHz (but we get 156), so perhaps we can claim 300MHz...

There are the obligatory setup path violations because we're not making the "required" clock, attached.

[BrianHG]

You cleared out the timing for the S3_BANK_0_s0, but now, the slow paths are in the S3_BANK_1_s0.

Take a look at which logic I used the (*preserve*) in the 'BrianHG_DDR3_CMD_SEQUENCER_v16.sv' and attach your same attribute to all of the same logic regs.

[SpacedCowboy]

Doing that made things a lot worse (>100 setup paths failing, and lower fMax). Seems to be linked to 'preserve'ing BANK_ACT_ANY. Preserving the others didn't make much difference  -still got the same 23 violations and the same slack values.

In the interests of full disclosure, I was using "/* synthesis syn_preserve = 1 */" rather than the syn_ramstyle directive because there's no RAM being inferred on any other line apart from line 180.

Anyway, another midnight, and I'm up again at 5:30 so I'm off to bed...

[e2020k]

Do you still have plans to look into Lattice & Xilinx chips?

[BrianHG]

Xilinx shouldn't be a problem, but I will need to learn their IDE.  At least there are many dev boards with DDR3 ram on them.

I started looking into Lattice.  Their PLL is as limited as Gowin and I need to clock transition the data for their 4:1,1:4 DDR IOs to achieve their max 800mtps.  (Lattice's 2:1,1:2 mode is capped at ~500mtps) All I'm looking for is a cheap dev board with a wired DDR3 chip on it.  Right now, the cheapest is around 350 usd.

Lattice has the dirt cheap ECP5 series with 45kle and 85kle, so it is better to focus my attention there.

As for Xilinx, it does have a free DDR3 controller already.  What I may do is just tie my 'multiport' module of my DDR3 to their controller replacing my 'PHY' stage.  My multi-window VGA display processor just needs the 'dual-clock dual-port' ram assigned to Xilinx's and Lattice's counterparts to make my video system function.

[iMo]

I may try in turn with the ML605 board (w/ 512MB DDR3 and Virtex-6 there), provided I get something which builds in ISE 14.7 straight, without too much hassle (too complex stuff here in this thread for such an eternal rookie like me).. 

[SiliconWizard]

For Lattice, yeah, I remember we talked about designing a dev. board with DDR3, but ECP5s are currently as unobtainium as the rest and the very little stock there is here and there is with outrageous prices, we're talking like $100 for a ECP5-85.

[mopplayer]

During an unbroken burst, you send a command every 4 clocks to maintain optimum efficiency.  This means with a full rate controller, you can stuff 3 new commands in-between.  With a half-rate controller, your controller can only be fast enough to add 1 command in-between. 

I think the number of in-between commands shall not be limited by whether it is full-rate or half-rate controller.
Since it would only be using simple if-else clocked logic (inside fast clock domain, maybe 500MHz in your case), your controller should be able to achieve such goal without suffering from STA setup timing violation.

Please correct me if wrong.

Well, nothing bad. This is just an open source project.

And in fact, the bandwidth depends on the user interface issuing a new command. We do not need to concern what exactly is the half-rate/full-rate or a hybrid. The random access will depend on that bandwidth.

For example, if a controller runs at 533MHZ, the user interface is 533/2= 266, so bandwidth is  266x2Byte = 533MB/s, therefore 500(250) is 250MB/s. Please remember that is not actually the same as PC data rate.

I recall I implemented that Scrypt algorithm on DDR3 bandwidth bottleneck issue, and more cost on HBM.

I also have a Anlogic EG4S20 FPGA (with embedded SDRAM) to do some project, that user interface transfers can achieved at 200MHz, but that bus is 32 bit, so we get the 800MB/s bandwidth, which match the official datasheet described.

[BrianHG]

Well, nothing bad. This is just an open source project.

And in fact, the bandwidth depends on the user interface issuing a new command. We do not need to concern what exactly is the half-rate/full-rate or a hybrid. The random access will depend on that bandwidth.

For example, if a controller runs at 533MHZ, the user interface is 533/2= 266, so bandwidth is  266x2Byte = 533MB/s, therefore 500(250) is 250MB/s. Please remember that is not actually the same as PC data rate.

I recall I implemented that Scrypt algorithm on DDR3 bandwidth bottleneck issue, and more cost on HBM.

I also have a Anlogic EG4S20 FPGA (with embedded SDRAM) to do some project, that user interface transfers can achieved at 200MHz, but that bus is 32 bit, so we get the 800MB/s bandwidth, which match the official datasheet described.

The controller rate is the DDR_CK command port clock rate.
IE: If run at 400MHz, the DDR rate is 800MTPS.

If my interface runs at 200Mhz, I still need to command my DDR command port at the 400MHz clock and the data is still coming in and going out at 800MTPS.

This should be called a quarter rate controller.

Though, the full bandwidth of the memory is still available.  Example, if my ram is 16bit at 800MTPS, my 200MHz controller interface needs to run at least at 64bits to maintain unbroken full 800MTPS bursts.  I went 2x further, 128bit to allow unbroken full speed bursts even running my controller interface at 1/4 mode, 100MHz.

With this setup, the core of my controller, even when the user interface is set to 100MHz, will still stuff commands inbetween, IE Command timer part of it still runs at the 400MHz command port rate and at the end or beginning of a burst cycle, to open/close/precharge rows in an attempt to cut down on as much wasted dead time as possible when a new row needs to be opened/closed.  Once said rows are open or closed, you can read/write back and forth between them with no wasted clock cycles except for the mandatory buss-turnaround cycles in the changeover between a read to write to read in the DDR3 spec.

[asmi]

Hi Brian,

Since Lattice ECP5U parts are now available, would you be willing to port your controller onto that device? If you will provide me a pinout and commit to making a port, I will design and assemble the board for HW validation as well as provide an assembled board for you (and others if they will contribute to development and/or will be willing to buy them). We can discuss & debate specifics of what else will be on that board in a dedicated topic, as I'm open for suggestions in that regard. But we need to decide on a part fast, so that I would be able to buy them while they are still available. Right now I can see 85F device in 381, 554 and 756 balls packages are available (285 are available too, but they have 0.5 mm pitch which is too fine for PCB manufacturers I used in the past). I'm inclined to go with the largest 756 balls part because "MOAR is better" , but I'm open to suggestions.

[BrianHG]

Give me a day to get back to you.  I've already did some research a few months back.  It seems it is best for me to use Lattice's free DDR# PHY in place of my 'Altera DDR IO Port' .sv module, so, copying an example schematic from one of their dev boards for wiring the DDR3 itself is a good starting place for research to maintain hardware backwards compatibility with Lattice's full paid version of their DDR3 controller.

Unless you are going for a compact 1 chip DDR3 solution, I also recommend going to 2 if not 4 DDR3s, or a laptop sodimm module.  4 DDR3s/Lapto SODIMM 64 bit module give us enough speed for basic real-time 3D texturing graphics engine, or, ~ 10x 1080p 60 real-time bandwidth.

Also, remember, for the high speed GBPS transceivers, you need to paid version of Lattice Diamond.  This is useful for true 1080p60/4k30p HDMI out on 8 IOs.  Not everyone has this version of the Diamond Studio.  Unless Lattice changed this policy, the best you can officially get is 800 MBPS on a DQ buss and the cheaper FPGAs.  (I think in the 480 & above pin count, you may have a cross compatible with and without GTPS transceivers.)

Also, be careful, there are different peak maximum MBPS on the horizontal rows VS the vertical columns IO banks.

[asmi]

Give me a day to get back to you.  I've already did some research a few months back.  It seems it is best for me to use Lattice's free DDR# PHY in place of my 'Altera DDR IO Port' .sv module, so, copying an example schematic from one of their dev boards for wiring the DDR3 itself is a good starting place for research to maintain hardware backwards compatibility with Lattice's full paid version of their DDR3 controller.

Are you sure they provide a free PHY for DDR3? I can see it in their IPs list, but so is the "full" DDR3 controller, and description doesn't say anything about licensing.

Unless you are going for a compact 1 chip DDR3 solution, I also recommend going to 2 if not 4 DDR3s, or a laptop sodimm module.  4 DDR3s/Lapto SODIMM 64 bit module give us enough speed for basic real-time 3D texturing graphics engine, or, ~ 10x 1080p 60 real-time bandwidth.

I think 64 bit is going to be too much for that FPGA, it will consume almost half of all available IO pins even in the largest package. So I'm thinking two 16bit devices for 32 bit wide data bus would be optimal.

Also, remember, for the high speed GBPS transceivers, you need to paid version of Lattice Diamond.  This is useful for true 1080p60/4k30p HDMI out on 8 IOs.  Not everyone has this version of the Diamond Studio.  Unless Lattice changed this policy, the best you can officially get is 800 MBPS on a DQ buss and the cheaper FPGAs.  (I think in the 480 & above pin count, you may have a cross compatible with and without GTPS transceivers.)

Forget about transceivers, the version I'm looking at is ECP5U which doesn't have any, and it's the only version of ECP which can be used with free license.

[BrianHG]

Give me a day to get back to you.  I've already did some research a few months back.  It seems it is best for me to use Lattice's free DDR# PHY in place of my 'Altera DDR IO Port' .sv module, so, copying an example schematic from one of their dev boards for wiring the DDR3 itself is a good starting place for research to maintain hardware backwards compatibility with Lattice's full paid version of their DDR3 controller.

Are you sure they provide a free PHY for DDR3? I can see it in their IPs list, but so is the "full" DDR3 controller, and description doesn't say anything about licensing.

Ok, their PHY is nothing more than a pre-configed IO buffer DDR port for the command and DQ lines.

It has no intelligence and just calls and configures the physical buffers, see here:
https://www.latticesemi.com/products/designsoftwareandip/intellectualproperty/ipcore/ipcores02/ddr3phy

This is Lattice's full DDR3 controller: https://www.latticesemi.com/products/designsoftwareandip/intellectualproperty/ipcore/ipcores01/ddr3sdramcontroller

As you can see, the full controller also uses the top 'PHY' to drive the IO buffer pins.

I know the full controller is paid, but what about that PHY.

If not, I basically need to manually instantiate the IO pin's DDR primitives like what I have done in my 'BrianHG_DDR3_IO_PORT_ALTERA.sv' source file.  Lattice seems to have similar PLL capabilities to what 'SpacedCowboy' was doing with Gowin, except, I am forced to use a 4:1 and 1:4 serializer to achieve the the full 800mbps instead of Altera's 2:1 and 1:2.  This isn't horrible as I have already a second layer 1:2 / 2:1 after my Altera IO port's 2:1 / 1:2, but, I will need to remove some code in my main PHY section.

Properly instantiate the DQ bus to the DQS lines instead of using a tuned PLL and using the IO buffer's delay lines for the write 90deg phase will also save a second PLL from being used since I must use the PLL's main clock output on any IO buffer to achieve the 800mbps unlike with Altera where any PLLs clock can run the IO buffer at full speed and there I also get 6 tunable outputs instead of 1 tunable output, plus 3 additional integer divided outputs per Lattice PLL.

2x 16bit DDR3 will still be plenty speed for a 720p 3D accelerator, or 1080p @ 16bit color.
1x 16bit DDR3 will give us the same as the Arrow DECA board, double the necessary bandwidth for 1080p60 at 32bit.

The 'LFE5U-45' and 'LFE5U-85' should both have a variant with pin-pin compatibility.
There should be enough in the 85k chip to include a 1080p MJPEG2000 player with 2x 16bit DDR3 chips, just barely enough with the 45k chip.

I will look tonight at the documentation about the Lattice's QDR IO buffer (4:1 / 1:4) and see how they handle the write delay and DQS read latching.  I'll also read up on their PLL capabilities.  Limiting myself to 1 PLL just means my controller will also be meaningful for Lattice's 25k / 12k fpga.  (only 2 PLLs) (Note that their 12k is actually the same die as their 25k.)

[SpacedCowboy]

I haven't completely given up on the Gowin implementation, btw. I just had something come along that's taking all my time. I *really* need to retire soon...

[asmi]

Ok, their PHY is nothing more than a pre-configed IO buffer DDR port for the command and DQ lines.

It looks like they do provide a block called "ddr_mem" in Clarity, which is free and can be used to create a PHY layer for the memory controller. It's kind of described in chapter 7 of "ECP5 and ECP5-5G High-Speed I/O Interface" document, but only in very general terms. So I suspect one would require quite a bit of simulations to figure out how it works and how to interface with it.

2x 16bit DDR3 will still be plenty speed for a 720p 3D accelerator, or 1080p @ 16bit color.
1x 16bit DDR3 will give us the same as the Arrow DECA board, double the necessary bandwidth for 1080p60 at 32bit.

16bit DDR3 running at 400MHz should be enough for double-buffering of 1080p@60 (meaning reading one frame and writing another one in parallel) with some margin to spare. So 32 bit interface would have a huge margin. Or we can instead go for 4 x 8 bit interface, this will allow to have higher memory capacity (4 x 4Gb = 2 GB vs 2 x 4Gb = 1 GB, or 4 GB vs 2 GB for largest 8Gb devices). Since DDR3 uses flyby routing, it shouldn't be much harder to route four 8 bit memory devices as opposed to two 16 bit ones.

As for HDMI out, it would make sense to add something like TFP410 to implement a 1080p@60 output without having to mess around with LVDS, or, since these FPGA seem to natively support MIPI, we can simply implement a DSI out port.

The 'LFE5U-45' and 'LFE5U-85' should both have a variant with pin-pin compatibility.

I'm not so sure about that one, as both 381 and 554 packages have different amount of IO pins for 45F and 85F devices as per datasheet, and 756 package only exists for the largest device (85F).

[BrianHG]

The 'LFE5U-45' and 'LFE5U-85' should both have a variant with pin-pin compatibility.

I'm not so sure about that one, as both 381 and 554 packages have different amount of IO pins for 45F and 85F devices as per datasheet, and 756 package only exists for the largest device (85F).

In the 381, 2 IOs are missing on the 45F version which I believe are VCC / GND or NC.  Just double check.  With Altera, I would usually just wire the appropriate power pins to the power and tristate on the hardwired power on the larger FPGA.

DDR rates on their data sheet V3.0 on page 66 & 67:
https://www.latticesemi.com/-/media/LatticeSemi/Documents/DataSheets/ECP5/FPGA-DS-02012-3-0-ECP5-ECP5G-Family-Data-Sheet.ashx?document_id=50461

Only the Left and Right banks support the full 800mb/s.

Isn't the TFP410 only DVI, IE, no HDMI sound.  You might as well add a HDMI/DVI cable driver NXP's PTN3366BSMP wired to a 800mb/s DQ8 group as well is not fitted under the TFP410 as an option.  Though 720p, 1280x960, or 1080p30 will be the max, you have the option for HDMI audio.

[BrianHG]

@asmi, I'm sorry, but I cannot guarantee I will have enough free time over the next 3 months to do work on adapting my DDR3 controller to Lattice.  It's a shame that we didn't do this back in September.