Kintex-7 FPGA board with DDR3 memory and pci-e

[NorthGuy]

I think up to one channel (64 bits) is routable on a 4 layer board, and two channels (128 bit total) might be doable on 6 layers.

One "channel" takes over 100 pins. If you try hard, you may be able to attach 2 channels with 5 banks, most likely 6 - it's 300 IO pins. Any you'll have to route most of the pins in these 6 banks. If you look at pinouts, they're about 10-row deep. I don't think 6-layers is easy, if possible at all.

Also, there's a bank mismatch - you have 3 HP banks and 5 HR banks.

[asmi]

One "channel" takes over 100 pins. If you try hard, you may be able to attach 2 channels with 5 banks, most likely 6 - it's 300 IO pins. Any you'll have to route most of the pins in these 6 banks. If you look at pinouts, they're about 10-row deep. I don't think 6-layers is easy, if possible at all.

Yea, I have my doubts about this as well.

[OwO]

I've already routed the Zynq 32-bit DDR3 on a 4 layer board, and that one was extra hard because of the fixed pin assignments and you can't swap bits in the address bus. There was also the issue of routing each address bit to two chips. I used two signal layers, a ground plane, and a VCCDDR plane. The design passes all memory tests at the maximum supported speed which is 1066 MT/s. With DIMMs and soft controllers there is none of that to deal with, you only need to route each bit to a supported IO and each address bit only need to be routed to one pin. I'm not going to be doing multi-rank memory. If you keep the routing short and use 2T timing you can get away with no termination.

The DDR pins on the Zynq are also up to 5 rows deep and I managed to get it all routed. The 1.0mm pitch of the Kintex also means more channels between vias (every 3 vias can have a large channel passing two signals and a small channel passing one signal).

I had a quick look at the pinout of the Kintex and indeed you can not go to 2 channels because there aren't enough HP bank pins. 1 channel looks very doable on 4 layers.

Edit:
Meanwhile I'm kekking hard: https://forums.xilinx.com/t5/Embedded-Processor-System-Design/Zynq-PCB-routing/td-p/369839

Ideally I would like to use only 8 layers but not sure if this is possible. The DDR3 interface looks like it may need more. Has anyone else used this part and could comment on the feasibility?

I used 10 layers for  XC7Z020-CLG484.  8 layers is hardly possible, unless you use only the lower 16 bits of DDR data bus and one DDR chip.

Yes I think 10 is probably the minimum without using buried/blind vias.

For ddr3 routing probably 4 signal layers except top and bottom are needed. If you add another 4 ground reference and two powers you get to 12 which is a comfortable design. 10 is doable if you are careful with either top / bottom routing of ddr3

[asmi]

I've already routed the Zynq 32-bit DDR3 on a 4 layer board, and that one was extra hard because of the fixed pin assignments and you can't swap bits in the address bus.

Do you mind publishing gerbers for that routing? I would like to see that.

[OwO]

Here are the design files for the Zynq SoM. Rename .pcb.txt to .pcb and .sch.txt to .sch before opening with gEDA.

Top layer overlaid on in1:

in2 overlaid on bottom layer:


You can see the bottom is split into two regions, a VCCDDR plane region and a GND plane region. The DDR signals on in2 travel referenced to the VCCDDR plane always.

The bottom view showing all the stitching caps:


The stackup is JLC 0.1mm prepreg, 1.2mm total thickness.

EDIT: added gschem symbol library. Copy the symbols to your geda symbols directory before opening schematic.

[scatha]

I have a feeling that for your specific uses going with high end graphics cards can be cheaper than FPGA, and it is less export controlled. If Taobao pricing is to be trusted, you can buy two or three nVidia RTX 2080 Ti cards or half a dozen AMD Radeon Vega 64 cards for the price of a single high gate count Kinetix-7 development board with PCIe x8 interface. Also for nVidia they provide pre-tuned, manually optimized implementations of some common algorithms like FFT as concrete CUDA libraries for those high end cards.

The FPGA board used in a system that has a high end GPU only handles interfacing the PC to high speed ADC and DAC, allowing you to use a lower gate count, slower and cheaper part with either less internal SerDes or an external PCIe interface chip.

This is a good point, with modern GPUs a 64M point FFT is considered to be pretty small. Unless you need the low latency it is invariably better to do this sort of signal processing on the GPU if you can, it is just so much easier (and often faster) than on an FPGA. Getting the samples *to* the GPU is the time-consuming part, so if you develop a cheap sampler capable of DMA-ing to the GPU that would be useful to a lot of folks.

[OwO]

FPGA dev board prices don't really mean much, the pci-e ones are all way overpriced because of low volumes. Last I checked a decent GPU (used!) will cost as much as the development cost of a Kintex-7 board (several prototypes + chips). Don't forget power consumption as well, as the applications I have in mind will be doing DSP 24/7, and the >100W power draw of a GPU will give you a surprise on your electricity bill real fast.

[technix]

FPGA dev board prices don't really mean much, the pci-e ones are all way overpriced because of low volumes. Last I checked a decent GPU (used!) will cost as much as the development cost of a Kintex-7 board (several prototypes + chips). Don't forget power consumption as well, as the applications I have in mind will be doing DSP 24/7, and the >100W power draw of a GPU will give you a surprise on your electricity bill real fast.

When did you check it last time? GPU prices took a nose dive recently as part of the cryptocurrency crash. Also the point I was trying to make is that unless it is latency sensitive you can get away with a decent GPU and a very cheap FPGA (something on the caliber of XC6SLX9-2TQG144C) board that uses external PCIe interface chip, allowing you to pull off designs that does not even use any external memory. This means instead of a 6-layer board with DDR3, you can pull it off with a 2-layer board and does not even use any BGA packages.

[asmi]

If you decide to try, let us know what you get.

I've received the chip today. Don't have time to solder it yet, however I checked device's serial number through Xilinx "Device Lookup" service - and it turned out to be speed grade 1I instead of 2I as promised. Date code is 1841 (which I assume means 41st week of 2018). Other than wrong speed grade, it seems OK (see attached photos).

[ali_asadzadeh]

I've received the chip today. Don't have time to solder it yet, however I checked device's serial number through Xilinx "Device Lookup" service - and it turned out to be speed grade 1I instead of 2I as promised. Date code is 1841 (which I assume means 41st week of 2018). Other than wrong speed grade, it seems OK (see attached photos).

Where did you get them?

[Scrts]

I've received the chip today. Don't have time to solder it yet, however I checked device's serial number through Xilinx "Device Lookup" service - and it turned out to be speed grade 1I instead of 2I as promised. Date code is 1841 (which I assume means 41st week of 2018). Other than wrong speed grade, it seems OK (see attached photos).

Where did you get them?

Ordered from Iran, I guess.

It's written in the previous posts...

[NorthGuy]

I've received the chip today. Don't have time to solder it yet, however I checked device's serial number through Xilinx "Device Lookup" service - and it turned out to be speed grade 1I instead of 2I as promised. Date code is 1841 (which I assume means 41st week of 2018). Other than wrong speed grade, it seems OK (see attached photos).

Quit fast. It may be hard to distinguish -1 from -2 grade, may be only by special tests. Of course, it only matters if it works after you solder it

[asmi]

Quit fast. It may be hard to distinguish -1 from -2 grade, may be only by special tests. Of course, it only matters if it works after you solder it

I always order at Aliexpress with delivery via ePacket as it's trackable and quite fast (normally 2-3 weeks to Ontario, might be few days faster for more westward locations).
For speed grade - since 2017 Xilinx stopped laser-etching speed grade onto the package,  right now the only way to find it out without soldering chip onto the board is to use their "Device Lookup" service by reading 2D barcode that is etched on the package (bottom-right corner of package). So it's kind of complicated now, especially since you need Xilinx account and their approval to use this service.
Performance-wise there is a pretty significant difference between -1 and -2, for example MGT at -1 can only reach 3Gbps vs 6Gpbs for -2, DDR3L max speed of 667 vs 800, etc.
I've asked the seller about the wrong speed grade, so will see if it is an honest mistake, or intentional misinformation in the listing.

[OwO]

I scanned my Zynqs but none show speed grade info on either the Xilinx Go app or the web app. It was a speed grade 1 though so I'm not that concerned.

[OwO]

OK so this is where it gets sketchy. I tried to look up the device DNA (read using JTAG) with https://xapps9.xilinx.com/2dbarcode/ but it says "no data found" for all 3 devices I tested.

Seller 1: (date codes 2018)
38120A1E0325343F
2812087254282E1F

Seller 2: (date codes 2014)
2A0D8D1261843C4B
(I have one more from seller 2 that I haven't soldered yet)

The register dump of the last device:

BSCAN_SWITCH_USER_MASK   0001
CLASS   hw_device
DID   jsn-JTAG-SMT2-210251A08870-13722093-0
FULL_PROBES.FILE   /persist/vhdl/project_3/project_3.runs/impl_1/top.ltx
IDCODE   00010011011100100010000010010011
IDCODE_HEX   13722093
INDEX   1
IR_LENGTH   6
IS_SYSMON_SUPPORTED   true
MASK   00001111111111111111111111111111
MASK_HEX   0FFFFFFF
NAME   xc7z010_1
PART   xc7z010
PARTIAL_PROBES.FILES   
PROBES.FILE   /persist/vhdl/project_3/project_3.runs/impl_1/top.ltx
PROGRAM.DPA_COUNT   0
PROGRAM.DPA_MODE   
PROGRAM.DPA_PROTECT   false
PROGRAM.FILE   /persist/vhdl/project_3/project_3.runs/impl_1/top.bit
PROGRAM.HW_CFGMEM   
PROGRAM.HW_CFGMEM_BITFILE   
PROGRAM.HW_CFGMEM_TYPE   
PROGRAM.IS_AES_PROGRAMMED   false
PROGRAM.IS_RSA_PROGRAMMED   false
PROGRAM.IS_SUPPORTED   true
PROGRAM.OPTIONS   
PROGRAM.READBACK_FILE   
REGISTER.BOOT_STATUS   00000000000000000000000000000000
REGISTER.BOOT_STATUS.BIT00_0_STATUS_VALID   0
REGISTER.BOOT_STATUS.BIT01_0_FALLBACK   0
REGISTER.BOOT_STATUS.BIT02_0_INTERNAL_PROG   0
REGISTER.BOOT_STATUS.BIT03_0_WATCHDOG_TIMEOUT_ERROR   0
REGISTER.BOOT_STATUS.BIT04_0_ID_ERROR   0
REGISTER.BOOT_STATUS.BIT05_0_CRC_ERROR   0
REGISTER.BOOT_STATUS.BIT06_0_WRAP_ERROR   0
REGISTER.BOOT_STATUS.BIT07_0_SECURITY_ERROR   0
REGISTER.BOOT_STATUS.BIT08_1_STATUS_VALID   0
REGISTER.BOOT_STATUS.BIT09_1_FALLBACK   0
REGISTER.BOOT_STATUS.BIT10_1_INTERNAL_PROG   0
REGISTER.BOOT_STATUS.BIT11_1_WATCHDOG_TIMEOUT_ERROR   0
REGISTER.BOOT_STATUS.BIT12_1_ID_ERROR   0
REGISTER.BOOT_STATUS.BIT13_1_CRC_ERROR   0
REGISTER.BOOT_STATUS.BIT14_1_WRAP_ERROR   0
REGISTER.BOOT_STATUS.BIT15_1_SECURITY_ERROR   0
REGISTER.BOOT_STATUS.BIT16_RESERVED   0000000000000000
REGISTER.CONFIG_STATUS   01000110000000000001111100001100
REGISTER.CONFIG_STATUS.BIT00_CRC_ERROR   0
REGISTER.CONFIG_STATUS.BIT01_DECRYPTOR_ENABLE   0
REGISTER.CONFIG_STATUS.BIT02_PLL_LOCK_STATUS   1
REGISTER.CONFIG_STATUS.BIT03_DCI_MATCH_STATUS   1
REGISTER.CONFIG_STATUS.BIT04_END_OF_STARTUP_(EOS)_STATUS   0
REGISTER.CONFIG_STATUS.BIT05_GTS_CFG_B_STATUS   0
REGISTER.CONFIG_STATUS.BIT06_GWE_STATUS   0
REGISTER.CONFIG_STATUS.BIT07_GHIGH_STATUS   0
REGISTER.CONFIG_STATUS.BIT08_MODE_PIN_M[0]   1
REGISTER.CONFIG_STATUS.BIT09_MODE_PIN_M[1]   1
REGISTER.CONFIG_STATUS.BIT10_MODE_PIN_M[2]   1
REGISTER.CONFIG_STATUS.BIT11_INIT_B_INTERNAL_SIGNAL_STATUS   1
REGISTER.CONFIG_STATUS.BIT12_INIT_B_PIN   1
REGISTER.CONFIG_STATUS.BIT13_DONE_INTERNAL_SIGNAL_STATUS   0
REGISTER.CONFIG_STATUS.BIT14_DONE_PIN   0
REGISTER.CONFIG_STATUS.BIT15_IDCODE_ERROR   0
REGISTER.CONFIG_STATUS.BIT16_SECURITY_ERROR   0
REGISTER.CONFIG_STATUS.BIT17_SYSTEM_MONITOR_OVER-TEMP_ALARM_STATUS   0
REGISTER.CONFIG_STATUS.BIT18_CFG_STARTUP_STATE_MACHINE_PHASE   000
REGISTER.CONFIG_STATUS.BIT21_RESERVED   0000
REGISTER.CONFIG_STATUS.BIT25_CFG_BUS_WIDTH_DETECTION   11
REGISTER.CONFIG_STATUS.BIT27_HMAC_ERROR   0
REGISTER.CONFIG_STATUS.BIT28_PUDC_B_PIN   0
REGISTER.CONFIG_STATUS.BIT29_BAD_PACKET_ERROR   0
REGISTER.CONFIG_STATUS.BIT30_CFGBVS_PIN   1
REGISTER.CONFIG_STATUS.BIT31_RESERVED   0
REGISTER.COR0.BIT00_GWE_CYCLE   100
REGISTER.COR0.BIT03_GTS_CYCLE   101
REGISTER.COR0.BIT06_LOCK_CYCLE   111
REGISTER.COR0.BIT09_MATCH_CYCLE   111
REGISTER.COR0.BIT12_DONE_CYCLE   011
REGISTER.COR0.BIT15_SSCLKSRC   00
REGISTER.COR0.BIT17_OSCFSEL   000000
REGISTER.COR0.BIT23_SINGLE   0
REGISTER.COR0.BIT24_DRIVE_DONE   0
REGISTER.COR0   00003fec
REGISTER.COR0.BIT25_DONE_PIPE   0
REGISTER.COR0.BIT26_RESERVED   0
REGISTER.COR0.BIT27_PWRDWN_STAT   0
REGISTER.COR0.BIT28_RESERVED   0000
REGISTER.COR1.BIT00_BPI_PAGE_SIZE   00
REGISTER.COR1.BIT02_BPI_1ST_READ_CYCLE   00
REGISTER.COR1.BIT04_RESERVED   0000
REGISTER.COR1.BIT08_RBCRC_EN   0
REGISTER.COR1.BIT09_RBCRC_NO_PIN   0
REGISTER.COR1   00000000
REGISTER.COR1.BIT10_RESERVED   00000
REGISTER.COR1.BIT15_RBCRC_ACTION   00
REGISTER.COR1.BIT17_PERSIST_DEASSERT_AT_DESYNC   0
REGISTER.COR1.BIT18_RESERVED   00000000000000
REGISTER.EFUSE.DNA_PORT   03C218648B1B054
REGISTER.EFUSE.FUSE_CNTL   00C0
REGISTER.EFUSE.FUSE_DNA   2A0D8D1261843C4B
REGISTER.EFUSE.FUSE_KEY   0000000000000000000000000000000000000000000000000000000000000000
REGISTER.EFUSE.FUSE_USER   00000000
REGISTER.IR   010001
REGISTER.IR.BIT0_ALWAYS_ONE   1
REGISTER.IR.BIT1_ALWAYS_ZERO   0
REGISTER.IR.BIT2_ISC_DONE   0
REGISTER.IR.BIT3_ISC_ENABLED   0
REGISTER.IR.BIT4_INIT_COMPLETE   1
REGISTER.IR.BIT5_DONE   0
REGISTER.TIMER   00000000
REGISTER.TIMER.BIT00_TIMER_VALUE   000000000000000000000000000000
REGISTER.TIMER.BIT30_TIMER_CFG_MON   0
REGISTER.TIMER.BIT31_TIMER_USR_MON   0
REGISTER.USERCODE   ffffffff
REGISTER.USR_ACCESS   00000000
REGISTER.WBSTAR   00000000
REGISTER.WBSTAR.BIT00_START_ADDR   00000000000000000000000000000
REGISTER.WBSTAR.BIT29_RS_TS_B   0
REGISTER.WBSTAR.BIT30_RS   00
UNKNOWN_DEVICE   false
USER_CHAIN_COUNT   4
VARIANT_NAME   
XSDB_USER_BSCAN   1,3

[Axk]

Sorry for an off-topic question guys, but how do you control DDR3 with the FPGAs?
Do they have built-in hard core controllers?
To use the controllers do you have to pay?
Are there trial versions for prototyping?

[NorthGuy]

Sorry for an off-topic question guys, but how do you control DDR3 with the FPGAs?
Do they have built-in hard core controllers?
To use the controllers do you have to pay?
Are there trial versions for prototyping?

Some of the FPGAs have built-in DDR controllers. Some have "helper" blocks. Xilinx has free DDR3 component called MIG (Memory Interface Generator). Or you can write your own.