BrianHG_DDR3_CONTROLLER open source DDR3 controller. NEW v1.60.

[SpacedCowboy]

Can you test a 90degree WDQ clock?
You need to change your PLL (i forgot if it is auto...), the parameter 'DDR3_WDQ_PHASE' at the top of your top hierarchy, and your .sdc file.

Ok, so the 90-degree phase is actually worse - see attached. I'm seeing -3.366 as worst-case slack rather than -2.496. This is with changing the SDC file clock definition and also "DDR3_WDQ_PHASE" in top.sv, which passes it through to BrianHG_DDR3_PLL, which passes it through to gowin_ddr_clocking, which uses it to index the phase table.

Running with 0-degree phase (as above, in the SDC and top.sv), the results are identical to the 90-degree results. Both of these are runs using the "clean and re-run all" option. Just to make sure there's no stale data.

I'm getting the feeling that I'm missing something fundamental here - I would have expected more change...

Also find out the output buffer's delay element range and test a write data clock of 0 deg.  If 0 deg really helps and the programmable delay element can be made large enough to shift the write data by 90 deg, then we might have to go that route

The IODELAY module is 18ps per step, with a max step-count of 127, ==> 2.286ns. a 400MHz signal is 2.5ns, so we can do a delay of ~90% of a period. But I guess that doesn't help much if the 0-degree phase still has all the setup violations...
 

[BrianHG]

Ok, try this:

At the PLL, the write clock output, assign it to the DDR3 clock.
This will force every associated ounce of logic in the write section to be tied to the first PLL.

You might need to remove the set_output_delay in the .sdc.

If this doesn't work, I don't know what will.  We put all of the write onto the same system clock as all the other outputs.  It may be boiling down to something with the clock switch we are using to swap from read to write.

[SpacedCowboy]

Ok, so with that I have no clk_ddr3_wr setup violations. There are some clk_ddr setup violations though (attached)

I also have 4 hold violations (attached)

If we can remove/ignore these, perhaps then use the DELAY to get the write-phase working...

[edit: meant to say, this is with the SDC containing no set_output_delay statements, but keeping the set_input_delay statements]

[BrianHG]

Though we did cut down from ~2.5ns to 1.5ns slack, it is still not good enough.
Though, the funny thing, this path should not be a problem.
The write DQ path is nothing more than a DFF -> DFF, a number of fold.

Ok, let's try changing this parameter in the IO port source HDL:

line 72:

Code: [Select]

parameter int        WDQ_SYNC_CHAIN          = 3 + CMD_ADD_DLY - WDQ_CLK_270, // + ((CLK_KHZ_IN*CLK_IN_MULT/CLK_IN_DIV)>=450000), 
Try changing the '3' to a '2' and a '4'.

This shifts the clock domain transition position in the middle of the write pipe FIFO from the DDR_CLK to the DDR_CLK_WDQ.

You may want to try changing everything back to 270deg and also just try 0deg.

[SpacedCowboy]

Ok, so I tried pushing it as far as it needed to go, in order to pass muster. Setting the PLLs to x19/2, and changing the (default) calculation of 'slow, hot' to 'fast' in the analysis, I can get everything to pass - This is still with DDR3_CLK_WR being assigned to DDR3_CLK.

No setup violations, no hold violations. What I don't really understand is how it can claim a max-frequency so high in the summary view, when trying to actually push it that far will give all sorts of violations...

Something else cropped up when I launched the constraint editor to play with the environment constraints - it popped up a dialogue saying it couldn't find the DQ/DQS ports - that's the first time I've seen that warning, there's nothing in the log. So it may not be as constrained as I thought it was...

[edit: looks like

Code: [Select]

set_input_delay -clock clk_ddr3_rd -max -add_delay             0.500 [get_ports {ddr_dq[0]}] is acceptable, but

Code: [Select]

set_input_delay -clock clk_ddr3_rd -max -add_delay             0.500 [get_ports {ddr_dq*[*]}] is not. Even the single asterisk inside the [] isn't acceptable. Looks like I have a lot of copy/paste to do...

[SpacedCowboy]

Ok, I've tried a variety of options here now, and with the clock above ~256MHz, I'm always getting setup and/or hold violations.

The best seems to be with:
  - DDR_CLK_WR assigned to DDR_CLK
  - No set_output_delay parameters in the SDC file
  - syntax-acceptable port definitions in the SDC file for all the ddr_dq and ddr_dqs lines in the set_input_delay parameters
  - parameter int        WDQ_SYNC_CHAIN          = 2 + CMD_ADD_DLY - WDQ_CLK_270

With that, I'm seeing:
  - worst slack violation of -1.000ns
  - worst hold violation of -0.35ns

It's almost midnight, so it's checked in and I'm off to bed ...

[BrianHG]

I'm curious, what happens if you disable the set_input_delay ?  IE: we made that spec sooooo tight that it is preventing any proper routing of the output paths.

I'm wondering if our problem stems from our clock 'switch' technique.

Also, we are currently feeding the OE through the ODDR, try feeding the OE directly to the bidir buffer.

Another trick would be to use an output enable buffer, then, tie the IDDR directly to the pin.
May using a bidir buffer forces the input to be placed directly at the pin while the output logic cell is buried or visa-versa.

One thing I do know, it's got to be possible to shift data to the ODDR at a good rate unless you can find a spec somewhere that the ODDR cannot achieve 300MHz, ie 600mtps.  It appears the IDDR works unless you need to go down the list of timing violations and it s actually there at the bottom.


Believe when I say, my write data module is nothing more than:

command_in -> df1 -> df2 -> df3 -> df4 -> df5 -> ODDR...

And your compiler is complaining about the df5 -> ODDR point, after it has had enough DFF to strategically place them right at the IO pin's DFF.


Does Gowin have any data on the maximum throughput of the ODDR primitive?
Maybe they need us using the 4:1 serializer to achieve 600mtps.

[BrianHG]

@SpacedCowboy, I think I know what's going on.
Perform my above tests.  The hadr-wire the DQ clock to the DDR_CLK_WDQ only, then hard-wire it only to the DDR_CLK only.

If a hard-wire to exclusive DDR_CLK cleans up the timing violations, then we will need to move the tunable DDR_CLK_RD to PLL#2 and wire the DDR_CLK_WDQ to PLL#1 setting everything back to 270 deg phase.  This might fix the timing violations.

Also, DDR3 needs to run at a minimum of 300MHz though I tested proper functionality at 250 in the past.  I once had a limiter in my code set to 300, but you are lucky I lowered it to 250 for these tests.

[SpacedCowboy]

@SpacedCowboy, I think I know what's going on.

Well, I'm glad *someone* does

"The tests" being to do both of:

- disable all the set_input_delays as well as set_output_delays
- also feed OE directly to the bi-dir (though this will be 3 clocks out of phase with the data because the ODDR has 3FFs in a chain, see attached)

... then (inside the DQ path) instead of

Code: [Select]

assign dq_clk[x] = (PIN_OE_WDQ_wide[x]) ? DDR_CLK_WDQ : DDR_CLK_RDQ;
we try:

Code: [Select]

assign dq_clk[x] = DDR_CLK_WDQ;
and:

Code: [Select]

assign dq_clk[x] = DDR_CLK_RDQ;
separately ? (I'll probably just wire them directly, but the assign shows the change better)

This is for CLK_WDQ being 270 degrees out of phase with CLK_DDR3, and not hardwired and with WDQ_SYNC_CHAIN=3 as originally set, right ?


Just checking

[BrianHG]

@SpacedCowboy, I think I know what's going on.

Well, I'm glad *someone* does

"The tests" being to do both of:

- disable all the set_input_delays as well as set_output_delays
- also feed OE directly to the bi-dir (though this will be 3 clocks out of phase with the data because the ODDR has 3FFs in a chain, see attached)

... then (inside the DQ path) instead of

Code: [Select]

assign dq_clk[x] = (PIN_OE_WDQ_wide[x]) ? DDR_CLK_WDQ : DDR_CLK_RDQ;
we try:

Code: [Select]

assign dq_clk[x] = DDR_CLK_WDQ;
and:

Code: [Select]

assign dq_clk[x] = DDR_CLK_RDQ;
separately ? (I'll probably just wire them directly, but the assign shows the change better)

This is for CLK_WDQ being 270 degrees out of phase with CLK_DDR3, and not hardwired and with WDQ_SYNC_CHAIN=3 as originally set, right ?


Just checking

Yes, 1 clock alone, just the write clock.
Test compile.
Then in the PLL module, hard-wire the write clock to the DDR_CLK out.
Test compile.
Then in the IODDR, re-assign the combo read/write clock switch.
Test compile.


We are snooping around where that slack error is coming from.
Even if you set the write clock to 0deg, because it is one on a different PLL, it may not be able to meet the timing we want due to FPGA fabric routing.
For all we know, the error may be because the dumb ODDR clock is actually a negative clock input, but I doubt that one as the simulations should have been messed up.

Note that if the in and out IOBUFFER 'delays' can be real-time software controlled, then it might be possible to get everything down to 1 PLL and tie my generated tuning steps into a module to set the read delay and make my DDR3 run all off of 1 PLL with just a DDR_CLK, and the .50 & .25 outputs.

Note that the IObuffer delay also needs to be visible in modelsim to prove test the code changes.

[SpacedCowboy]

Ok, so test #1 conditions:

Code: [Select]

In BrianHG_DDR3_PLL:
===================
assign   DDR3_CLK      = PLL1_clk_out[0];      // DDR3 CK clock running at 1/2 the DQ rate.
assign   DDR3_CLK_WDQ  = PLL1_clk_out[1];      // DDR3 write data clock 90 degree out of phase running at 1/2 the DQ rate.
assign   DDR3_CLK_RDQ  = PLL1_clk_out[2];      // DDR3 phase adjustable read data input clock running at 1/2 the DQ rate.

In top.sv:
=========
DDR3_WDQ_PHASE          = 270,

In ddr_io_port_gowin:
====================
WDQ_SYNC_CHAIN          = 3  + CMD_ADD_DLY - WDQ_CLK_270

route DQ OE directly to IOBUF:
           IOBUF gowin_dq_iobuf_inst
                (
                .O(gowin_dq_in),                    // IOBUF -> IDDR
                .IO(DDR3_DQ[x]),                    // DQ pad
                .I(gowin_dq_out),                   // ODDR -> IOBUF
                .OEN(~PIN_OE_WDQ_wide[x])           // input when 1'b1

Set DQ clock to toggle:
            assign dq_clk[x] = (PIN_OE_WDQ_wide[x]) ? DDR_CLK_WDQ : DDR_CLK_RDQ;


I get an error that the top module can't directly drive the IOBUF ...

ERROR (CK0011) : Instance 'DDR3_PHY/BHG_DDR3_IO_PORT_GOWIN/gowin_DQ_bus[0].gowin_dq_oddr_inst'(ODDR) of module 'top' cannot drive instance 'DDR3_PHY/BHG_DDR3_IO_PORT_GOWIN/gowin_DQ_bus[0].gowin_dq_iobuf_inst'(IOBUF)

The code is below: I replaced the commented out part of the DQ IOBUF with the version that does not send OE through the ODDR.

Code: [Select]

    for (x=0; x<DQ_WIDTH; x = x + 1)
        begin : gowin_DQ_bus

            wire gowin_dq_out;
            wire gowin_dq_in;
            wire gowin_dq_tx_out;

            // See above comment in DQS code for this...
             assign dq_clk[x] = (PIN_OE_WDQ_wide[x]) ? DDR_CLK_WDQ : DDR_CLK_RDQ;

            ODDR gowin_dq_oddr_inst 
                (
                .Q0(gowin_dq_out),                  // ODDR -> IOBUF
                .Q1(gowin_dq_tx_out),               // OE   -> IOBUF, 1'b0 => output
                .D0(PIN_WDATA_PIPE_h[0][x]),        // Input data [SDR]
                .D1(PIN_WDATA_PIPE_l[0][x]),        // Input data [SDR]
                .TX(~PIN_OE_WDQ_wide[x]),           // Input 'output enable' 1'b0=out
                .CLK(dq_clk[x])                     // write clock
                );

            IDDR gowin_dq_iddr_inst 
                (
                .Q0(RDQ_l[x]),                      // SDR to app #0
                .Q1(RDQ_h[x]),                      // SDR to app #1
                .D(gowin_dq_in),                    // DDR input signal
                .CLK(dq_clk[x])                     // read clock
                );

//           IOBUF gowin_dq_iobuf_inst
//               (
//               .O(gowin_dq_in),                    // IOBUF -> IDDR
//               .IO(DDR3_DQ[x]),                    // DQ pad
//               .I(gowin_dq_out),                   // ODDR -> IOBUF
//               .OEN(gowin_dq_tx_out)               // input when 1'b1
//               );
           IOBUF gowin_dq_iobuf_inst
                (
                .O(gowin_dq_in),                    // IOBUF -> IDDR
                .IO(DDR3_DQ[x]),                    // DQ pad
                .I(gowin_dq_out),                   // ODDR -> IOBUF
                .OEN(~PIN_OE_WDQ_wide[x])           // input when 1'b1
                );
        end

Swapping in the commented section will make it compile again, so it's definitely that single line in the IOBUF OE. I can do the tests without the direct wiring, if that will help ?

[BrianHG]

What happened, this is not what I asked.  Or did I miss read what you` are saying.

Change:

Code: [Select]

assign dq_clk[x] = (PIN_OE_WDQ_wide[x]) ? DDR_CLK_WDQ : DDR_CLK_RDQ;to:

Code: [Select]

assign dq_clk[x] = DDR_CLK_WDQ ;
This should work as it used to work.
The only problem should be a potential timing issue with the DDR_CLK_RDQ.

The second test is within the PLL module to assign the output DDR_CLK_WDQ to DDR_CLK.
Compile again.

[SpacedCowboy]

Assuming the test is still useful,

Test #1 conditions:

Code: [Select]

In BrianHG_DDR3_PLL:
===================
assign   DDR3_CLK      = PLL1_clk_out[0];      // DDR3 CK clock running at 1/2 the DQ rate.
assign   DDR3_CLK_WDQ  = PLL1_clk_out[1];      // DDR3 write data clock 90 degree out of phase running at 1/2 the DQ rate.
assign   DDR3_CLK_RDQ  = PLL1_clk_out[2];      // DDR3 phase adjustable read data input clock running at 1/2 the DQ rate.

In top.sv:
=========
DDR3_WDQ_PHASE          = 270,

In ddr_io_port_gowin:
====================
WDQ_SYNC_CHAIN          = 3  + CMD_ADD_DLY - WDQ_CLK_270

route DQ OE directly to IOBUF:
           IOBUF gowin_dq_iobuf_inst
                (
                .O(gowin_dq_in),                    // IOBUF -> IDDR
                .IO(DDR3_DQ[x]),                    // DQ pad
                .I(gowin_dq_out),                   // ODDR -> IOBUF
                .OEN(~PIN_OE_WDQ_wide[x])           // input when 1'b1

Set DQ clock to toggle:
            assign dq_clk[x] = (PIN_OE_WDQ_wide[x]) ? DDR_CLK_WDQ : DDR_CLK_RDQ;

Timing analysis results:
  - all clk_ddr3 setup & hold results were fine, minimum setup slack is 1.413ns
  - clk_ddr3_wr setup results miss by worst slack of -1.658ns


Test #2 conditions: as #1, except

Code: [Select]

In BrianHG_DDR3_PLL:
===================
assign   DDR3_CLK_WDQ  = DDR3_CLK;      // DDR3 write data clock 90 degree out of phase running at 1/2 the DQ rate.


Timing analysis results:
  - clk_ddr3 setup results miss by worst slack of -0.813ns



Test #3 conditions: as #2, except

Code: [Select]

In ddr_io_port_gowin:
====================
assign dq_clk[x] = DDR_CLK_RDQ;

Timing analysis results:

Ok, wow.

  - all clk_ddr3 setup & hold results were fine, minimum setup slack is 1.413ns
  - all clk_ddr3_wr setup & hold results were fine, minimum setup slack is 1.413ns
  - rather than > 100 'read' clock issues, I see 4, as attached:


Re:your last question, I think we crossed over in time there. The above ought to be what you were asking for, if I'm understanding you correctly.

[BrianHG]

Your test condition 3 was not to be done.  It will always read wrong because of the 'false_path' and it is not what I asked.


Go back to test condition #2 and show me the negative slack report.

[SpacedCowboy]

Ah, mistook DDR_CLK for DDR_CLK_RD...

Ok, here's the results from test #2 (ignoring the clk_rd setup violations), attached

[BrianHG]

Ok.
1) Are the set_output_delay constraints active?

2) Are the et_input_delay constraints active?

3) What happens when in the PLL, you now also assign the DDR_CLK_RDQ to the DDR_CLK?

[SpacedCowboy]

I was just about to say mea culpa

I just double-checked the SDC file, and the inputs delays are still active. I mustn't have saved the file

I have to run - the kid has soccer, but I'll come back to it ASAP and re-run without the input delays, then assign DDR_CLK_RDQ to DDR_CLK and re-run again.

[BrianHG]

Also, are your DQ pins assigned to the correct FPGA's dedicated IO DQ groups?  If not, this can be a problem.

[SpacedCowboy]

I took the DQ pin locations from their example DDR code, so that ought to be ok.

[BrianHG]

I ask because looking at the setup report, you see 6 paths at >=0.2ns, then there is a gigantic drop to 0.08ns.

Remember, different FPGA sizes may have different dedicated or optimized locations.

[SpacedCowboy]

On my phone atm, it’s the example for this particular board, but I’ll double-check when I get back

[BrianHG]

I hope Gowin isn't a copy of Lattice:

Read this .pdf:
https://www.latticesemi.com/view_document?document_id=50461
Page 67, you will see 2 different types of DDR.

GDDRX1 with it's data rate of 250/500, and GDDRX2 with it's data rate of 400/800.

The difference is an annoying crap fest.
Does Gowin have a similar IDDR/ODDR function set?

[SpacedCowboy]

First, I verified the location constraints against the schematic, and they all match.

Ok, test #1 as above

  - all clk_ddr3 setup & hold results were fine, minimum setup slack is 1.413ns
  - attached test1-setup-clk_ddr3_wr violations
  - no hold violations

test #2, as above: DDR3_CLK_WDQ  = DDR3_CLK
  - attached test2-setup-clk_ddr3 violations
  - nothing to report for clk_ddr3_wr (obviously enough)
  - attached test2-hold violations

test #3, also have DDR3_CLK_RDQ  = DDR3_CLK
  - no setup or hold violations at all
  - attached test3-path-slacks table


I'm not aware of any difference in the DDR's - everything seems to be uniform on a pad-by-pad basis, and there's only one "ODDR" primitive that I can see in their list.

[BrianHG]

Is you clock 300MHz or 400MHz?

If everything checks, then we need to investigate the IO delay options as path #3 seems to give us the best results.
You will need to set the output delay based on a parameter and adjust the input delay in real-time based on our 16 possible stepped clock.

If we can make this work, then at lease there is a chance of getting the design down to 1 PLL is you can make the .50 and .25 outputs, all with 0deg phase.

[SpacedCowboy]

So the clock above was at mult=22, div=2  --> Freq =297. I bumped it up a few times, and got as far as mult=30, div=2 ---> 405 MHz.
 
It very nearly managed mult=32, there are only 2 failing paths.

This is all with the 'fast, cold' model, not the 'slow, hot' model, though. I doubt it'd make it that far under the 'slow, hot' model.