DDR3 initialization sequence issue

[promach]

What is the technical rationale for the need of SIX FF synchronizers ?

[BrianHG]

For 90 degrees: Only needed 3 for a 300MHz DDR3 ram controller on a -6 speed grade Cyclone.  Needed 4 for a slower -8 speed grade Cyclone.  5 was required to get a -6 speed grade to run at 400MHz, but, sometimes when I loaded up other code a lot of logic in the FPGA, some nets wouldn't make the cut.  I couldn't go any further since I exceeded tWL, so, I also had to add a pipe to my entire command, bank and address outputs to keep everything in sync giving me that additional extra 1 I wanted to achieve a 400MHz timing with everything in the black.

For 270 degrees: Subtract 1 from everything above.  Going up to 6 allowed 500MHz to run fine for the first time.  However, at 450MHz, tWL is also a clock longer, so, I did not need to add any other coding changes as my code allows dynamically configuring this pipe size with the parameter 'WDQ_SYNC_CHAIN'.  In fact, my IO port section allows dynamically configuring the register depth of read and write and command pipes all individually with parameters.

Different FPGA, different manufacturers & compilers will require more or less steps to shift the clock with different frequencies.  You just need to test and play.  There is an advantage here to using FPGAs which have delays on the DDR IO buffers, but they do have their drawbacks of compatibility across different types.


IE: I just increased the number until FMAX was steady and fast every compile.

[promach]

I am curious as in how adding more level of FF synchronizers to DQS-related signal would actually lead to tWL violation ?

[BrianHG]

Not DQS, it's the DQ path.  DQS is only on the ck_0 clock exclusively.

Note that everything else in my DDR controller path has been adapted to have the same delay.  (well, with the recognition of the CWL delay of course)  Otherwise the ram wouldn't work.  The data would be in the wrong place compared to where the command was sent and where the ram was expecting it.

DQS only has an output enable, no data.  It is generating a fixed clock.

I generate the DQS' OE at lines 477-482 in my code and it runs on ck_0.

And why are you showing me ^tDQSS(min).  That is the worst possible alignment.  You want the DQSnom (nominal), Otherwise, some ram chips or modules along the path will be outside the DDR_CK -> DQS alignment.  The 'min' gives you 0 room for error and routing and FPGA IO jitter and pin-pin tolerances.

[promach]

sometimes when I loaded up other code a lot of logic in the FPGA, some nets wouldn't make the cut.  I couldn't go any further since I exceeded tWL, so, I also had to add a pipe to my entire command, bank and address outputs to keep everything in sync giving me that additional extra 1 I wanted to achieve a 400MHz timing with everything in the black.

I guess I need to simulate my code with Micron simulation model inside Modelsim for all the setup timing coding patches I have applied.
This will involved some emulation of Xilinx primitive using some home-made version just to prove that all the other DDR3 timings are not affected by those coding patches.

Note: Micron simulation model does not work within Xilinx ISIM simulator.

ck_270 (3T/4) has more data capture cycle margin compared to ck_90 (T/4) , but what I do not understand is how adding more level of FF synchronizers (in your case, SIX) is able to increase the data capture cycle margin ?

[BrianHG]

ck_270 (3T/4) has more data capture cycle margin compared to ck_90 (T/4) , but what I do not understand is how adding more level of FF synchronizers (in your case, SIX) is able to increase the data capture cycle margin ?

The fitter can decisively place the logic latches at any point on the fabric to optimize the maximum required hold timing.  It can also delay the latching clock time or delay the output at each logic DFF step to meet those required setup and hold timings.  1 -DFF is enough if the logic is fast enough to meet the timing 3T/4.  In the case of altera's maximum 400-500MHz core DFF-to-adjacent-DFF, no logic inbetween, a few steps are needed to ease the transfer and get it ready synced to the DDRIOBUF input clock.  If your FPGA can internally run DFF-DFF at 800MHz, to do that 3T/4 at 300MHz, you will only need 1-DFF, or none at all depending on fabric routing.  For a 300MHz controller, I bet you need at least 1 even with Xilinx where as the minimum with Altera Cyclones, I could get it to work with was originally 2.  But if I fill my core fabric with a ton of other functional logic, I would be either forced to buy the full Quartus and manually place the location of my ram controller on the fabric, or add that large sync chain and use the free version of Quartus and allow it to randomly build the fabric anyway it likes, and then the timing to the IO buffers will still make it no matter the fabric routing.  The later means anyone can build my design and still achieve FMAX without fancy manual core placement.

Also, what would happen if you have 4x16bit DDR3 ram chips with a 484pin FPGA?  2 on the left and 2 on the right?  How will the IO timing to the core which now needs to siphon data from IOs on opposite sides of the FPGA?  I know from experience that my design has the necessary length of pipe to coalesce the data coming from and going to opposite sides of the FPGA will achieve the desired FMAX.

Remember, just getting a DDR3 controller to work with 1 ram chip all wired to 1 corner of the FPGA, no multiport manager, just wired to a JTAG test port or my RS232 debugger would require almost no such extravagant effort at all.

[promach]

In the following simulation waveform, using 50MHz crystal clock (clk signal) is not suitable since 350MHz PLL clock (ck signal) will feed from the OSERDES data at 7 times the speed of the 50MHz clock.

Note: I have two separate home-made OSERDES as suggested by @NorthGuy

So, this means the data output to the DDR3 RAM is not following the proper ordering.  Each set of 8-pieces of data ({5, 6, 7, 8, 9, 10, 11, 12}) should only be sent to DDR3 RAM ONCE
Note: 8 pieces of data is due to SERIALIZATION_FACTOR = 8

I tried the following code modification but I am stucked with CDC from fast clock (350MHz ck_dynamic signal) to slow clock (87.5MHz clk_serdes signal)
Note: 350MHz divided by (SERDES_RATIO >> 1) equals 87.5MHz

Conventional FF synchronizer only works for CDC from slow clock to fast clock

Code: [Select]

[phung@archlinux DDR]$ git status --short
 M ddr3_memory_controller.v
 M test_ddr3_memory_controller.v
[phung@archlinux DDR]$ git diff ddr3_memory_controller.v
diff --git a/ddr3_memory_controller.v b/ddr3_memory_controller.v
index 55f2823..78ec1e2 100644
--- a/ddr3_memory_controller.v
+++ b/ddr3_memory_controller.v
@@ -181,6 +181,8 @@ module ddr3_memory_controller
                output reg data_read_is_ongoing,
        `endif
       
+       output clk_serdes,  // 87.5MHz
+       
        output reg ck_en, // CKE
        output reg cs_n, // chip select signal
        output reg odt, // on-die termination
@@ -632,6 +634,7 @@ reg MPR_ENABLE, MPR_Read_had_finished;  // for use within MR3 finite state machi
 `else
 
        wire clk_pll;
+       wire clk_serdes;
        wire ck, ck_out;
        wire ck_90;
        wire ck_180, ck_180_out;
@@ -647,6 +650,11 @@ reg MPR_ENABLE, MPR_Read_had_finished;  // for use within MR3 finite state machi
                       
                        // Clock out ports
                        .clk_pll(clk_pll),  // OUT 50MHz, 0 phase shift, for solving CDC issues
+                       
+                       // SERDES_RATIO = 8, but 2 separate serdes are used due to double-data-rate restriction
+                       // So, 350MHz divided by (SERDES_RATIO >> 1) equals 87.5MHz
+                       .clk_serdes(clk_serdes),  // OUT 87.5MHz, 0 phase shift, for SERDES use
+                       
                        .ck(ck),  // OUT 350MHz, 0 phase shift
                        .ck_90(ck_90),  // OUT 350MHz, 90 phase shift, for dq phase shifting purpose
                        .ck_180(ck_180),  // OUT 350MHz, 180 phase shift
[phung@archlinux DDR]$
[phung@archlinux DDR]$ git diff test_ddr3_memory_controller.v
diff --git a/test_ddr3_memory_controller.v b/test_ddr3_memory_controller.v
index 4c8acbf..e87225f 100644
--- a/test_ddr3_memory_controller.v
+++ b/test_ddr3_memory_controller.v
@@ -305,6 +305,8 @@ wire ldqs_iobuf_enable;
 wire data_read_is_ongoing;
 `endif
 
+wire clk_serdes;  // 87.5MHz
+
 reg [BANK_ADDRESS_BITWIDTH+ADDRESS_BITWIDTH-1:0] i_user_data_address;  // the DDR memory address for which the user wants to write/read the data
 
 `ifdef HIGH_SPEED
@@ -338,11 +340,7 @@ reg done_writing, done_reading;
                        data_write_index = data_write_index + 1)
                begin: data_write_loop
        `endif
-               `ifdef TESTBENCH                       
-                       always @(posedge clk_sim)
-               `else
-                       always @(posedge clk)
-               `endif
+                       always @(posedge clk_serdes)
                        begin
                                if(reset)
                                begin
@@ -387,7 +385,7 @@ reg done_writing, done_reading;
                                                `endif
                                        `endif
                                       
-                                       test_data <= test_data + 1;
+                                       test_data <= test_data + SERDES_RATIO;
                                        write_enable <= (test_data < (STARTING_VALUE_OF_TEST_DATA+NUM_OF_TEST_DATA-1));  // writes up to 'NUM_OF_TEST_DATA' pieces of data
                                        read_enable <= (test_data >= (STARTING_VALUE_OF_TEST_DATA+NUM_OF_TEST_DATA-1));  // starts the readback operation
                                        done_writing <= (test_data >= (STARTING_VALUE_OF_TEST_DATA+NUM_OF_TEST_DATA-1));  // stops writing since readback operation starts
@@ -575,6 +573,8 @@ ddr3_control
                .data_read_is_ongoing(data_read_is_ongoing),
        `endif
       
+       .clk_serdes(clk_serdes),  // 87.5MHz
+       
        .ck_en(ck_en), // CKE
        .cs_n(cs_n), // chip select signal
        .odt(odt), // on-die termination
[phung@archlinux DDR]$


[NorthGuy]

In the following simulation waveform, using 50MHz crystal clock (clk signal) is not suitable since 350MHz PLL clock (ck signal) will feed from the OSERDES data at 7 times the speed of the 50MHz clock.

1:7 is very inconvenient. Create clocks which are easy to work with. Start from the max speed you can get from BUFG (250 to 400 MHz depending on your speed grade). For example -2 grade allows 375 MHz. This is the frequency you can feed to ODDR. Divide it by 4. It'll be 93.75 MHz. Create a 93.75 MHz clock and use it for your main clock throughout FPGA. This way you'll need two 4:1 serializers and ODDR. Or use 187.5 MHz. This way you'll need two 2:1 serializers and ODDR.

If all your clocks come from the same PLL, they're all automatically aligned.

[promach]

but this method still could not avoid the CDC from from fast clock (350MHz ck_dynamic signal) to slow clock (87.5MHz clk_serdes signal)

[NorthGuy]

but this method still could not avoid the CDC from from fast clock (350MHz ck_dynamic signal) to slow clock (87.5MHz clk_serdes signal)

When clocks are produced by the same PLL, they are aligned (that is the edge of the slower clocks nominally coinsides with an edge of the faster clock), so you don't need any special CDC. Your serializers/deserializer will be clocked with the slower clock on the fabric side and with the faster clock on the IO side. That's the only place where these clocks need to meet.

[promach]

I believe there is some underlying hardware limitation that prohibits PLL (with dynamic phase shift option enabled) from generating frequencies larger than 374.5318MHz as shown in screenshot below.

[NorthGuy]

I believe there is some underlying hardware limitation that prohibits PLL (with dynamic phase shift option enabled) from generating frequencies larger than 374.5318MHz as shown in screenshot below.

Look at the datasheet (ds162). BUFG can only be 375 MHz for -2 grade despite the PLL itself may go faster.

[promach]

I am using xc6slx16-3ftg256

PLL (without dynamic phase shift option enabled) could generate output frequencies above 400MHz.

I do not understand what you meant by Or use 187.5 MHz. This way you'll need two 2:1 serializers and ODDR.

[NorthGuy]

I am using xc6slx16-3ftg256

PLL (without dynamic phase shift option enabled) could generate output frequencies above 400MHz.

Then you can go to 400 MHz. Did you tell ISE that you have -3 grade.

I do not understand what you meant by Or use 187.5 MHz. This way you'll need two 2:1 serializers and ODDR.

If you use 400 MHz fast clock then using 2:1 serializer will require 200 MHz clock on the other side.
If you use 4:1 serializer, it will require 100 MHz slow clock.

Either of these is acceptable for you.

[promach]

wait, it is the turning on of dynamic phase shift option of the PLL that actually caused the maximum frequency limit to drop from 400MHz to 374.5MHz.

And since dynamic phase shift option is needed, so I could not have 8:1 (no 400MHz for me),  this is why I am only having 7:1 (350MHz)

again, I am confused as in why use 4:1 serializer, it will require 100 MHz slow clock ?

[NorthGuy]

wait, it is the turning on of dynamic phase shift option of the PLL that actually caused the maximum frequency limit to drop from 400MHz to 374.5MHz.

Then you can only do 375 MHz.

And since dynamic phase shift option is needed, so I could not have 8:1 (no 400MHz for me),  this is why I am only having 7:1 (350MHz)

Any frequency can be divided by 8 (or by 4, or by any other number). 375/8 = 46.875 MHz, or 375/4 = 93.75 MHz, or 375/2 = 187.5 MHz. Take your pick and create the corresponding clock. Abandon your oscillator clock for anything except for PLL feed.

again, I am confused as in why use 4:1 serializer, it will require 100 MHz slow clock ?

The X:1 serializer will require 375/X frequency. On the IO side you have 375 Mb/s. On the fabric side you have X wires - the data gets spread over these wires, so each wire carriers 375/X Mb/s.

When you connect two of such serializers to an ODDR or IDDR, you get double data rate outside of FPGA - 375*2 = 750 Mb/s. Between IDDR/ODDR and serializers, the signal is carried by two wires (D0 and D1), each carrying 375 Mb/s. On the fabric side you get 2*X wires (each of the two serializers has X wires). Each wire carries 750/(2*X) = 375/X Mb/s = 375/X MHz.

So, if you want two 4:1 serializers (X = 4), your slow clock must be 375/X = 375/4 = 93.75 MHz

If you want two 2:1 serializers (X = 2), your slow clock must be 375/X = 375/2 = 187.5 MHz

If you abandon serializers, and only use ODDR/IDDR (X = 1), your slow clock must be 375/X = 375/1 = 375 MHz, which is the same as ODDR/IDDR clock.

[promach]

The PLL maximum output frequency limit is 374.5318MHz , not 375MHz

So, clk_serdes will have frequency of 350MHz / 4 = 87.5MHz for the case of 4:1 serializers

In this case, CDC from fast clock (350MHz ck_dynamic) to slow clock (87.5MHz clk_serdes) still could not be avoided.

If I eliminate serializer and uses only 350MHz ck, then I will have pack error as follows:

Code: [Select]

ERROR:Pack:1107 - Pack was unable to combine the symbols listed below into a
   single IOB component because the site type selected is not compatible.

   Further explanation:
   The pad symbol ck is connected to a symbol that is outside of I/O comp. There
   is no routing resource between them.

   Symbols involved:
    BUF symbol "ddr3_control/OBUF_ck" (Output Signal = ck)
    PAD symbol "ck" (Pad Signal = ck)

[NorthGuy]

In this case, CDC from fast clock (350MHz ck_dynamic) to slow clock (87.5MHz clk_serdes) still could not be avoided.

Since you decided to shift clock phase, you probably need a shifted slow clock for your serializers as well. Otherwise, serializers won't work. Then, you need to find a way to pass the data from the shifted slow clock to your main clock.

[promach]

What do you exactly mean by shift clock phase ?

[NorthGuy]

What do you exactly mean by shift clock phase ?

There are two ways to align data with the clock. Either you shift data (with IODELAY), or you shift the clock (with PLL). You decided to do the second. Now your read clock is shifted relative to the write clock.

[BrianHG]

In this case, CDC from fast clock (350MHz ck_dynamic) to slow clock (87.5MHz clk_serdes) still could not be avoided.

Since you decided to shift clock phase, you probably need a shifted slow clock for your serializers as well. Otherwise, serializers won't work. Then, you need to find a way to pass the data from the shifted slow clock to your main clock.

My serializers operate on the full 400MHz read clock itself, so I don't need to care about this problem as it doesn't exist in my design.  My final 128bit data chunk going to the slower 100Mhz clock has a special function to adapt for any phase shift on the 400MHz read clock, though I do first transfer that 128bit chunk to the 400MHz ck_0 first to aid in metastability with the same phase corrective tech, then pass that ck_0 128bit chunk to my slower 200MHz/100Mhz core clock.  To save on logic cells, if your dual-port, dual clock ram blocks can operate at 400MHz, you may use it here instead.

My write data serializers also operate on the full 400MHz ck_0 clock, and only the smaller 16 bit phase shifted pipe runs on the ck_90 clock.  Once again, if your dual-port, dual-clock ram ports can operate at 400MHz, it could be used in place of the logic based serial shifters and serializers.

[promach]

My final 128bit data chunk going to the slower 100Mhz clock has a special function to adapt for any phase shift on the 400MHz read clock

@BrianHG In your case, why would a slower 100MHz clock be needed to help with metastability arising from the use of PLL dynamic phase shift on read clock ?

[BrianHG]

My final 128bit data chunk going to the slower 100Mhz clock has a special function to adapt for any phase shift on the 400MHz read clock

@BrianHG In your case, why would a slower 100MHz clock be needed to help with metastability arising from the use of PLL dynamic phase shift on read clock ?

My 100/200MHz clock is like your 50MHz clock since my controller operates at 200MHz.

The enhanced metastability comes from first transferring the randomly phase shifted read clock aligned 128 bit chunk to the ck_0 clock, still 400MHz, (note that my data ready toggle is delayed by 1 clock going to the ck_0 to ensure that the entire 128bit bus have all first reached their destination regardless of FPGA route timing), then, the data now on that guaranteed ck_0 400MHz 128 bit latch can be sent to any other PLL clock frequency as long as that destination clock also has a fixed 0 degree phase without timing violations or errors incurred due to a random phase clock way at the beginning.

[promach]

then, the data now on that guaranteed ck_0 400MHz 128 bit latch can be sent to any other PLL clock frequency as long as that destination clock also has a fixed 0 degree phase without timing violations or errors incurred due to a random phase clock way at the beginning.

In my case, I could not generate "other PLL clock frequency" that has the same fixed 0 degree phase due to some internal hardware limitation of the Xilinx PLL IP core.

Could you advise ?

[NorthGuy]

then, the data now on that guaranteed ck_0 400MHz 128 bit latch can be sent to any other PLL clock frequency as long as that destination clock also has a fixed 0 degree phase without timing violations or errors incurred due to a random phase clock way at the beginning.

In my case, I could not generate "other PLL clock frequency" that has the same fixed 0 degree phase due to some internal hardware limitation of the Xilinx PLL IP core.

Could you advise ?

I am not aware of such limitations. The PLL primitive has 6 clock outputs, each of which lets you specify the divider (CLKOUTx_DIVIDE) and the phase (CLKCOUTx_PHASE). All you need to do is to find appropriate VCO frequency (between 400 and 1080 MHz for your speed grade) and specify the dividers for your clock. If a divider for your fast clock is X, then the divider for your slow clock will be (2*X) or (4*X), depending on your needs.

Say, if you want 350 MHz fast clock, create 700 MHz VCO (your 50 MHz crystal multipled by 14), then divide by 2 to get 350 MHz clock, by 4 to get 175 MHz clock, or by 8 to get 87.5 MHz clock.