Planning/design/review for a 6-layer Xilinx Artix-7 board for DIY computer.

[nockieboy]

Hi everyone,

This thread is a branch off from another thread here, which is itself a branch from a (large!) thread here.

TLDR for the large original thread; I was almost brand-new to electronics and had built a DIY 8-bit computer on a breadboard and had transitioned to 2-layer PCB design, making the computer as a stack of cards - each card with its own function (CPU, memory, IDE, power/serial etc.), and I needed a video card for the system.  That long thread culminates in the production of some serious SystemVerilog HDL to produce an HDMI-compatible graphics card for my computer, with 2D graphics functions, full SD-card read/write, a massive amount of work by BrianHG to produce an excellent DDR3 memory controller, programmable sound generators, etc., all whilst teaching me about FPGAs and HDL.

The FPGA line used were from Intel/Altera - starting with the Cyclone IV and moving to the MAX 10M50 on the Arrow DECA development board.

I'm now wanting to move across to Xilinx, mostly to test the design on a different platform, but also because I can't seem to find a reasonable upgrade path from the Intel 10M50 to something around the 75K LE mark on Mouser that won't take 4 years to be made and delivered and a mortgage to afford.

Following is part of the previous thread to help kick this one off:

As for your project, I would actually recommend going for a FGG484 package - it will allow implementing a 32 bit DDR3@400 MHz interface to give you additional room to grow in terms of memory bandwidth, it also has 4 multi-gigabit transceivers in case you want to implement some kind of serial interface - like SATA, PCI Express (say for NVMe storage, or some extension slot for future upgrades), DisplayPort, high-bandwidth HDMI (above 1080p@60), or something else. Speed grade 2 devices and higher can go as high as 6.6 Gbps per transceiver.

The bigger packages are more enticing for the very reasons you've mentioned, but again I've never soldered a BGA before and was a little concerned I was making too big a leap from 2-layer boards and QFP/QFNs to 6-layers with 484-BGAs.  Interestingly, every step 'forward/smaller' in the SMD world (with the exception of discretes - 0402 is a little uncomfortable for me without more practice) has gotten easier.  Those E-QFP144 Cyclone IVs were a lot harder than QFNs to solder, for example - I'm hoping BGA will be the same (with the liberal application of heat).

Finally, these Artix parts get pretty toasty, so you will need to provision for some sort of heatsink, or maybe even a small fan to help keep things from overheating. Remember, unlike CPUs, FPGAs don't have built-in overheating protection (unless you implement one yourself, that is), and will happily fry themselves if you are not careful.

Good point, hadn't considered that.  That will have a direct affect on the PCB design.  Instead of going for an intermediate 'Beaglebone-compatible' design which I can plug into the DECA interface (and thus sits at the bottom of the stack with no room for cooling), I might just design a specific board for my DIY computer and include the 3.3V-5V translation on it so it will just go onto the computer stack without an interface board.  That way it can sit at the top of the stack and I can mount a passive heatsink, ducted fan or nitrogen-cooled reservoir as necessary.

But it's kind of offtopic in this thread, so if you want to discuss this further, let's find a more suitable venue.

...and here's that thread.

So I'm looking at using the XC7A100T-2FGG484C as suggested by asmi.  I'm assuming Vivado will support creating bitstreams for this FPGA without the need to pay for a licence?  As far as I could tell from their website, it does.  Having to buy a licence is a deal-breaker for me as this is a learning journey, I won't be making anything for profit.

The PCB for this board will either be custom to fit my computer's stack headers, or potentially could be more of a generic development card design with Beaglebone headers - I am not fully set on either yet as each has its pros and cons.  If I go the custom-board route, it will be powered from the computer's power supply in the stack headers - so power consumption may become a concern as the power supply was never designed with FPGAs in mind (I didn't even know they existed when I slapped a through-hole 7805 and AMS1117-3.3 on the power card), it'll look neat and I can fit a heatsink if required to the FPGA.  The generic development card route will give me more room to fit peripherals on the board and its own, separate, power supply so that will negate concerns about power consumption, it'll be a board design that others may be interested in and make use of, but it won't fit the ergonomics of the computer stack (which, to be fair, is hardly a problem at all as I currently have a DECA sticking out at right-angles to the stack on the bottom at the moment anyway).

EDIT:  As the design progressed, it has become a fully standalone board, with no intention to plug into the old uCOM stack I was using.  Instead, it has become a board designed to run soft-core CPUs as full computer systems.

[asmi]

Maybe for long term it would be best to design a carrier board of sorts which would host both FPGA module, and your existing stack side-by-side? See a picture in attachment. This way you can upgrade/iterate over both parts separately from each other.

1. FPGA module would be the most "high-tech" part of a design, it would use high-speed connectors on the bottom side to connect to a carrier.  The module itself would host the bare minimum required for FPGA to work - FPGA itself, QSPI flash for the bitstream, DDR memory devices and a power delivery system.

2. Carrier will provide power for the whole system, it would also host high-speed interface connectors (PCIE, DisplayPort, HDMI, 1G Ethernet, USB, whatever), and would also connect FPGA module to extension stack.

You can use any Artix-7 part with a free license of Vivado/Vitis.

I think this thread belongs in a FPGA section as it's more about FPGA than it is about CAD. Maybe ask moderators to move it?

[nockieboy]

Maybe for long term it would be best to design a carrier board of sorts which would host both FPGA module, and your existing stack side-by-side? See a picture in attachment. This way you can upgrade/iterate over both parts separately from each other.

1. FPGA module would be the most "high-tech" part of a design, it would use high-speed connectors on the bottom side to connect to a carrier.  The module itself would host the bare minimum required for FPGA to work - FPGA itself, QSPI flash for the bitstream, DDR memory devices and a power delivery system.

2. Carrier will provide power for the whole system, it would also host high-speed interface connectors (PCIE, DisplayPort, HDMI, 1G Ethernet, USB, whatever), and would also connect FPGA module to extension stack.

Ah, I like this idea.  You mean something like this?





This is a Lattice FPGA available on AliExpress.  The base/carrier board has what appears to be a SODIMM connector (?) which the FPGA and its DDR3 chips slot into.  Looks like power generation is done on the FPGA card too.  This would allow me to develop the FPGA section and carrier separately - I could do two carriers, one for a general development board to make public and another that fits my DIY computer's layout.  Food for thought - thanks for that.

You can use any Artix-7 part with a free license of Vivado/Vitis.

That's good to know.

I think this thread belongs in a FPGA section as it's more about FPGA than it is about CAD. Maybe ask moderators to move it?

Will do.

Whilst that's being done, I have a question about power.  Have you built a board for an Artix-7 before?  I'm wondering about power chip selection.  I've got the schematics for the Arty A7 dev board, but the older version uses power management chips that cost in excess of £15 together, or the latest Arty-A7 board (E.2 revision) uses a newer (DA9062) chip that seems to need to be programmed, is only available non-programmed to particular companies or as a bulk purchase (with signed disclaimer by the client) in excess of £12,000 in total... not really an option for me!

I'm following the Arty 7 schematic and parts as closely as I can, but would like to minimise the cost of the power supply as much as is reasonable.  I'm going to need 5V, 3.3V, 1V, 1.35V and 0.675V rails.  Any suggestions for an alternative to the DA9062 I could use?

[asmi]

Ah, I like this idea.  You mean something like this?

No, something more like this: https://www.myirtech.com/list.asp?id=553

This is a Lattice FPGA available on AliExpress.  The base/carrier board has what appears to be a SODIMM connector (?) which the FPGA and its DDR3 chips slot into.  Looks like power generation is done on the FPGA card too.  This would allow me to develop the FPGA section and carrier separately - I could do two carriers, one for a general development board to make public and another that fits my DIY computer's layout.  Food for thought - thanks for that.

I don't like edge connectors because they usually command an extra charge for PCB manufacturing.

Whilst that's being done, I have a question about power.  Have you built a board for an Artix-7 before?  I'm wondering about power chip selection.  I've got the schematics for the Arty A7 dev board, but the older version uses power management chips that cost in excess of £15 together, or the latest Arty-A7 board (E.2 revision) uses a newer (DA9062) chip that seems to need to be programmed, is only available non-programmed to particular companies or as a bulk purchase (with signed disclaimer by the client) in excess of £12,000 in total... not really an option for me!

I'm following the Arty 7 schematic and parts as closely as I can, but would like to minimise the cost of the power supply as much as is reasonable.  I'm going to need 5V, 3.3V, 1V, 1.35V and 0.675V rails.  Any suggestions for an alternative to the DA9062 I could use?

You don't have to use these exact converters, any one which meets Artix DC specs (±5% voltage regulation, <50 mV noise) will work just fine. Artix-7 100T device can consume up to about 7 A of current on Vccint rail (at 1 V nominal). For example, you can use MP8772GQ - it can output up to 12 A of current, so very decent margin, it's quite affordable (just over 3 USD for qty 1), and - most importantly in our times - it's actually available in stock in decent quantities. For other rails you can use even "classic" TLV62130 (if you can find it in stock), or something like MP2384 (which is available in stock in quantities).

Alternatively you can use modules like this one: https://www.monolithicpower.com/mpm3683-7.html Their advantage is that they already have an integrated inductor, so all you need to use it is 2 resistors to set the voltage, a couple of caps for decoupling and filtering. They take less space than a "discrete" solution would, and are easier to route because one of the most critical traces (FET-to-inductor loop) is already done inside the package, so you literally just slap a cap or two nearby and you are done. The downside is the higher price - although it's not as clear-cut as it seem because you need more discrete parts for "discrete" solution, so sometimes modules actually end up cheaper. I personally prefer using modules when I need to pack things as tight as possible (typical problem for "compute modules" like that FPGA module we're talking about), but if space is not a big problem, you can use discrete converters as well.

[BrianHG]

Note that the 48$-39$ 85KLE fpgas from Lattice are finally available once again.
Though, you need to convert my DDR3 controller as Lattice's DDR3 costs money.

Note that in your current design, Parallel layers in my video controller eats a few K-LE per layer while the Sequential layers are near free in LE usage.  Obvious using the 2 layer types together is what gives you the 16-64 layers you may be using.

The way your current pixel writer is designed is slow and excessive in gate count because it was engineered to run on the old FPGA dual-port ram and support backward compatibility.  This is how things end up when you begin with engineering with a design for a 6KLE FPGA, then just recklessly add layers ontop.

[asmi]

@nockieboy Please check your PM.

[rstofer]

I would go to Digilent and look at the Artix 7 boards.

https://digilent.com/shop/boards-and-components/system-boards/fpga-boards/

I only consider the boards with the 100T chip...

Then I would follow the yellow brick road until I got to the schematic like:

https://digilent.com/reference/_media/programmable-logic/arty-a7/arty-a7-e2-sch.pdf

You will note a blank page in the schematic.  In the past, this was the details of connecting their JTAG programmer chip into the FPGA.  I don't know for sure what or even if something is missing.  Still, it's a start.

I would avoid building more of a PCB than absolutely required.  If a -35T device was big enough, I would plug the CMOD A7-35T device into a daughtercard.  Still, pinout is limited and may not suit the project.

[asmi]

Me and nockieboy just ordered a few 100Ts and 35Ts for this project as I found them in stock for the great price. So something will definitely come out of this

CMOD and Arty series boards are no good because they lack high speed connectors.

[nockieboy]

I would go to Digilent and look at the Artix 7 boards.

https://digilent.com/shop/boards-and-components/system-boards/fpga-boards/

I only consider the boards with the 100T chip...

Then I would follow the yellow brick road until I got to the schematic like:

https://digilent.com/reference/_media/programmable-logic/arty-a7/arty-a7-e2-sch.pdf

You will note a blank page in the schematic.  In the past, this was the details of connecting their JTAG programmer chip into the FPGA.  I don't know for sure what or even if something is missing.  Still, it's a start.

I would avoid building more of a PCB than absolutely required.  If a -35T device was big enough, I would plug the CMOD A7-35T device into a daughtercard.  Still, pinout is limited and may not suit the project.

Yes, the Arty-A7 board is what I'm using as a reference to start my own design off.  That Arty-A7 schematic includes the JTAG connector pinout to connect the Xilinx programmer into (I have a copy of one somewhere on a shelf for an old Spartan board I got years ago but never used).  Hopefully that should be sufficient to get the bitstream loaded onto the board.

You don't have to use these exact converters, any one which meets Artix DC specs (±5% voltage regulation, <50 mV noise) will work just fine. Artix-7 100T device can consume up to about 7 A of current on Vccint rail (at 1 V nominal). For example, you can use MP8772GQ - it can output up to 12 A of current, so very decent margin, it's quite affordable (just over 3 USD for qty 1), and - most importantly in our times - it's actually available in stock in decent quantities. For other rails you can use even "classic" TLV62130 (if you can find it in stock), or something like MP2384 (which is available in stock in quantities).

Alternatively you can use modules like this one: https://www.monolithicpower.com/mpm3683-7.html Their advantage is that they already have an integrated inductor, so all you need to use it is 2 resistors to set the voltage, a couple of caps for decoupling and filtering. They take less space than a "discrete" solution would, and are easier to route because one of the most critical traces (FET-to-inductor loop) is already done inside the package, so you literally just slap a cap or two nearby and you are done. The downside is the higher price - although it's not as clear-cut as it seem because you need more discrete parts for "discrete" solution, so sometimes modules actually end up cheaper. I personally prefer using modules when I need to pack things as tight as possible (typical problem for "compute modules" like that FPGA module we're talking about), but if space is not a big problem, you can use discrete converters as well.

Thanks asmi, that's perfect - I'll look at those suggestions in some detail over the next week or two as I work on the power supplies.  Do you use EasyEDA at all?  If so, I'm happy to share the project with you so you could give me more constructive feedback if you wish?

As far as the mezzanine board idea goes, I understand what you mean now - thanks for the links.  I had a quick look on Mouser last night for those mezzanine/back-to-back connectors, but they don't seem to have a good supply of them (certainly not the 144-pin versions anyway), but I guess the whole layout/form-factor of the board (and whether I'll use a mezzanine for the FPGA itself) is something I can worry about a little later.  Getting a good schematic sorted is my first priority, and now I've committed to the Artix A7-100T I'm keen to get moving on it - it's just a busy time of year with family commitments as I'm sure you'll understand!

It looks like I'm going to need two DDR3 chips in the design too, so that's something I'll need to bear in mind, thanks to the conversation with BrianHG over in the GPU thread:

...We are beginning to enter the realm of a simple 3D accelerator and with an added texture reader prior to this writer, with proper design, and maybe a second DDR3 chip for a 256bit wide bus & 128x speed, we will pass the first Sony Playstation in rendering capability.

One of the elements of the PCB design that has caused me headaches with the (admittedly 0.8mm pitch) LFE5U FPGA was the decoupling capacitors underneath the board.  I can solder 0402 confidently, but I don't feel confident at all going smaller than that.  How did you solder the 0201 caps on your boards?

[asmi]

Thanks asmi, that's perfect - I'll look at those suggestions in some detail over the next week or two as I work on the power supplies.  Do you use EasyEDA at all?  If so, I'm happy to share the project with you so you could give me more constructive feedback if you wish?

One of my commercial customers paid for my Altium Designer license a couple of years back (before the COVID), so that's what I'm using now. But I will be happy to help with your design as well, if not with doing the actual layout, but at least with feedback.

As far as the mezzanine board idea goes, I understand what you mean now - thanks for the links.  I had a quick look on Mouser last night for those mezzanine/back-to-back connectors, but they don't seem to have a good supply of them (certainly not the 144-pin versions anyway), but I guess the whole layout/form-factor of the board (and whether I'll use a mezzanine for the FPGA itself) is something I can worry about a little later.  Getting a good schematic sorted is my first priority, and now I've committed to the Artix A7-100T I'm keen to get moving on it - it's just a busy time of year with family commitments as I'm sure you'll understand!

That's OK, there is no rush, especially since there is going to be a Chinese New Year coming, so enjoy your time with the family. I've lost my mother to a COVID two years ago, so now I know how important it is to spend as much time as you can with the family, because you never know how much of it you have left...

As for connectors, I know a good place where you can find all kinds of connectors always in stock. And you can sometimes even get some for free But we will get there when we will get there, right now it's more important to plan your design thoroughly such that you won't have to do a redesign because of some small stupid thing you just forgot about, or didn't think hard enough.

Also, when you receive your FPGAs, please resist your urge to open them up. These things are moisture- and ESD-sensitive, so don't take unnecessary risks if you can help it. Or - if you can't resist - at least take out the 35T part as it's cheaper in case something bad happens

It looks like I'm going to need two DDR3 chips in the design too, so that's something I'll need to bear in mind, thanks to the conversation with BrianHG over in the GPU thread:

...We are beginning to enter the realm of a simple 3D accelerator and with an added texture reader prior to this writer, with proper design, and maybe a second DDR3 chip for a 256bit wide bus & 128x speed, we will pass the first Sony Playstation in rendering capability.

If you wish, you can implement two 32 bit DDR3 interfaces, or even up to 4 independent 16 bit DDR3 interfaces, or implement 4 x 8 bit interface instead of 2 x 16 bit one for higher capacity. There are a lot of possibilities. You just need to figure out what you need before you embark on a board design.

One of the elements of the PCB design that has caused me headaches with the (admittedly 0.8mm pitch) LFE5U FPGA was the decoupling capacitors underneath the board.  I can solder 0402 confidently, but I don't feel confident at all going smaller than that.  How did you solder the 0201 caps on your boards?

I use a stereo microscope (this one: https://amscope.com/products/se410-xyz but you can find cheaper options) and under 10x magnification 0201's are not that hard to solder unless your hands are really shaky. I typically do a reflow, so I place all parts by hand on a solder paste under microscope, and then stuff it into the oven to reflow it all at once. But if neccessary I can solder them manually one-by-one. That said, I try to avoid them and stick to 0402 whenever I can, because 0402 are super easy to solder under the microscope - infact I even taught my wife to do it and she now does it just as good (if not better) than I do - despite her knowing exactly nothing about electronics

I highly recommend you consider buying such microscope - it makes microsoldering a breeze since your depth perception still works (thanks to stereoscopic image). My wife finds using it so comfortable that she even does her nails now under it Make a good Christmas present for yourself Also if you have small kids, they might also find it interesting looking at various things under magnification. You will also need a pair of good tweezers (search for "titanium alloy tweezers" on Aliexpress - they are quite good, and should last for a long time, just be careful to avoid punctures as the "business end" is super-sharp).

Finally, to make soldering BGAs (or anything really) easier you might want to consider getting some sort of preheater. It doesn't need to be very powerful because realistically you will only be preheating your board to like 100°C or so, so if you have some sort of hot plate which can reach such temperature - it's good enough. Just make sure you can comfortably work with a hot air gun above PCB while it's on a heater - otherwise it's just a matter of time until you accidentally make a wrong move and end up getting a burn. Trust me, it will eventually happen in any case , but there is no reason to hasten that eventuality

[BrianHG]

Thanks asmi, that's perfect - I'll look at those suggestions in some detail over the next week or two as I work on the power supplies.  Do you use EasyEDA at all?  If so, I'm happy to share the project with you so you could give me more constructive feedback if you wish?

One of my commercial customers paid for my Altium Designer license a couple of years back (before the COVID), so that's what I'm using now. But I will be happy to help with your design as well, if not with doing the actual layout, but at least with feedback.

Yes, I am an Altium user as well, however, if you are making a product for public domain release, and will beginning with a new EDA tool, I would put in the effort to learn and use the latest KiCad.  The latest now does support trace length and impedance matching required for the higher speed DDR3 designs and if you make your design available on GitHub, others who don't have the money for an Altium license may still either contribute of use your cad files.

[BrianHG]

It looks like I'm going to need two DDR3 chips in the design too, so that's something I'll need to bear in mind, thanks to the conversation with BrianHG over in the GPU thread:

...We are beginning to enter the realm of a simple 3D accelerator and with an added texture reader prior to this writer, with proper design, and maybe a second DDR3 chip for a 256bit wide bus & 128x speed, we will pass the first Sony Playstation in rendering capability.

If you wish, you can implement two 32 bit DDR3 interfaces, or even up to 4 independent 16 bit DDR3 interfaces, or implement 4 x 8 bit interface instead of 2 x 16 bit one for higher capacity. There are a lot of possibilities. You just need to figure out what you need before you embark on a board design.

Having a separate DDR3 controller for display buffer, and texture memory buffer can accelerate 3D texture filling with avoiding a cache mechanism to recover speed losses due to cross-memory access.  This is how the original 3DFX graphics card operated as back then, we did not have the huge gate densities and block-ram of today's 3D accelerators.

[nockieboy]

But I will be happy to help with your design as well, if not with doing the actual layout, but at least with feedback.

Thanks asmi, feedback is all I need - I'm happy to do the layout myself (that's the biggest learning point) but I don't want to make a silly mistake if another set of eyes is able to pick up on something.

That's OK, there is no rush, especially since there is going to be a Chinese New Year coming, so enjoy your time with the family. I've lost my mother to a COVID two years ago, so now I know how important it is to spend as much time as you can with the family, because you never know how much of it you have left...

I'm sorry to hear that.  Yes, you never know, so make the most of your time!

As for connectors, I know a good place where you can find all kinds of connectors always in stock. And you can sometimes even get some for free But we will get there when we will get there, right now it's more important to plan your design thoroughly such that you won't have to do a redesign because of some small stupid thing you just forgot about, or didn't think hard enough.

Thank you.

If you wish, you can implement two 32 bit DDR3 interfaces, or even up to 4 independent 16 bit DDR3 interfaces, or implement 4 x 8 bit interface instead of 2 x 16 bit one for higher capacity. There are a lot of possibilities. You just need to figure out what you need before you embark on a board design.

I'm sure BrianHG will have an input on this.  Xilinx has its own DDR3 controller IP, doesn't it?  I also seem to recall a long time ago you and BrianHG having a discussion about Xilinx FPGAs and DDR support.  Just wondering what problems and benefits a switch to Xilinx will create...?

I use a stereo microscope (this one: https://amscope.com/products/se410-xyz but you can find cheaper options) and under 10x magnification 0201's are not that hard to solder unless your hands are really shaky. I typically do a reflow, so I place all parts by hand on a solder paste under microscope, and then stuff it into the oven to reflow it all at once. But if neccessary I can solder them manually one-by-one. That said, I try to avoid them and stick to 0402 whenever I can, because 0402 are super easy to solder under the microscope - infact I even taught my wife to do it and she now does it just as good (if not better) than I do - despite her knowing exactly nothing about electronics

I highly recommend you consider buying such microscope - it makes microsoldering a breeze since your depth perception still works (thanks to stereoscopic image). My wife finds using it so comfortable that she even does her nails now under it Make a good Christmas present for yourself Also if you have small kids, they might also find it interesting looking at various things under magnification. You will also need a pair of good tweezers (search for "titanium alloy tweezers" on Aliexpress - they are quite good, and should last for a long time, just be careful to avoid punctures as the "business end" is super-sharp).

Haha - that's good recommendation if the missus uses it too!  Not sure I could convince my wife to do soldering of such small parts, with or without a microscope!   Something like that microscope is ideal for me and is definitely on my wishlist, though it'll be a few months before I can invest that much money into one.

Finally, to make soldering BGAs (or anything really) easier you might want to consider getting some sort of preheater. It doesn't need to be very powerful because realistically you will only be preheating your board to like 100°C or so, so if you have some sort of hot plate which can reach such temperature - it's good enough. Just make sure you can comfortably work with a hot air gun above PCB while it's on a heater - otherwise it's just a matter of time until you accidentally make a wrong move and end up getting a burn. Trust me, it will eventually happen in any case , but there is no reason to hasten that eventuality

Yes, I have a hot plate already for when I start BGA soldering - one of these:

Yes, I am an Altium user as well, however, if you are making a product for public domain release, and will beginning with a new EDA tool, I would put in the effort to learn and use the latest KiCad.  The latest now does support trace length and impedance matching required for the higher speed DDR3 designs and if you make your design available on GitHub, others who don't have the money for an Altium license may still either contribute of use your cad files.

Okay, well it looks like I'm going to have to bite the bullet and learn how to use KiCAD then, despite having serious dislike of it.   Maybe once I've overcome the UI issues and odd UX choices, I'll grow to like it.

Having a separate DDR3 controller for display buffer, and texture memory buffer can accelerate 3D texture filling with avoiding a cache mechanism to recover speed losses due to cross-memory access.  This is how the original 3DFX graphics card operated as back then, we did not have the huge gate densities and block-ram of today's 3D accelerators.

@BrianHG - does the Xilinx have any performance benefits or memory controller IP or hardcore functions that stand out as affecting the GPU's DDR3 controller at all?

[asmi]

I'm sure BrianHG will have an input on this.  Xilinx has its own DDR3 controller IP, doesn't it?  I also seem to recall a long time ago you and BrianHG having a discussion about Xilinx FPGAs and DDR support.  Just wondering what problems and benefits a switch to Xilinx will create...?

Read through the section "Chapter 1: DDR3 and DDR2 SDRAM Memory Interface Solution -> Interfacing to the Core" of this document: https://docs.xilinx.com/v/u/en-US/ug586_7Series_MIS It explains how to talk to controller. Basically there are four possibilities, ordered from higher level to lower level, with each step down giving you progressively more control over memory devices, but also leaving more of housekeeping work to you as well:
1. AXI4 protocol via AXI4 slave interface block (useful if you have some other components which also talk AXI4), this is how I usually talk to it, it's the highest abstraction level
2. user interface, this level allows sending commands like "read address 0x0010_0000 and return 8xdata bus width worth of data", or "write this 8xdata bus width worth of data to address 0x0020_0000). It's kind of similar to AXI4, but uses a custom bus interface and is architecturally below it. You can also take control over refresh and ZQ calibration timings if you wish.
3. native interface, at this level there is no longer an address, but rank, bank, row, column.
4. PHY only. This is basically only gives you a physical interface, leaving actual controller implementation to you.
Xilinx recommends using levels 1 or 2, I've been using mostly level 1 (AXI4 interface). And just to give you some idea of the amount of resources it takes, a 32 bit DDR3 controller uses 6157 LUTs and 5361 register when AXI4 interface is used (so all memory controller layers are in use). These numbers obviously fluctuate a bit, but they are in the same ballpark. An A100T device has 63400 LUTs and 126800 registers.

Haha - that's good recommendation if the missus uses it too!  Not sure I could convince my wife to do soldering of such small parts, with or without a microscope!   Something like that microscope is ideal for me and is definitely on my wishlist, though it'll be a few months before I can invest that much money into one.

That's good. You can also browse through Aliexpress, there should be plenty of choice starting from about 200 USD. Whichever one you pick, make sure it's got a focal length (sometimes called "Working Distance" in the microscope specifications) of at least 10 cm so that you can comfortably manipulate under it without hitting the head all the time.

Yes, I have a hot plate already for when I start BGA soldering - one of these:

That should be more than sufficient.

Okay, well it looks like I'm going to have to bite the bullet and learn how to use KiCAD then, despite having serious dislike of it.   Maybe once I've overcome the UI issues and odd UX choices, I'll grow to like it.

Really? From my (admittedly VERY limited) experience with EasyEDA I found it's interface to be remarkably similar to KiCAD.
But to tell you the truth, at various times I used DipTrace, Eagle CAD, Orcad/Allergo, Altium Designer and KiCAD, and they all suck in some way or the other. So you basically just need to get used to idiosyncrasies of whichever eCAD you choose, and stick to it. For you I would naturally recommend KiCAD because it's free, and it's quite good (see the project in my signature for the proof - and keep in mind that that project was done in a previous major version of KiCAD).

@BrianHG - does the Xilinx have any performance benefits or memory controller IP or hardcore functions that stand out as affecting the GPU's DDR3 controller at all?

The most important hardware difference is Xlinx 7 series has 6-input LUTs as opposed to 4-input LUTs in Cyclone 4-5 and MAX10 series devices. So the same HDL usually consumes less LUTs on 7 series and can be faster due to lower amount of combinatorial logic levels. As for memory controller, take a look at the document I linked above, and it will hopefully answer all your questions.

[BrianHG]

I'm sure BrianHG will have an input on this.  Xilinx has its own DDR3 controller IP, doesn't it?  I also seem to recall a long time ago you and BrianHG having a discussion about Xilinx FPGAs and DDR support.  Just wondering what problems and benefits a switch to Xilinx will create...?

Read through the section "Chapter 1: DDR3 and DDR2 SDRAM Memory Interface Solution -> Interfacing to the Core" of this document: https://docs.xilinx.com/v/u/en-US/ug586_7Series_MIS It explains how to talk to controller. Basically there are four possibilities, ordered from higher level to lower level, with each step down giving you progressively more control over memory devices, but also leaving more of housekeeping work to you as well:
1. AXI4 protocol via AXI4 slave interface block (useful if you have some other components which also talk AXI4), this is how I usually talk to it, it's the highest abstraction level
2. user interface, this level allows sending commands like "read address 0x0010_0000 and return 8xdata bus width worth of data", or "write this 8xdata bus width worth of data to address 0x0020_0000). It's kind of similar to AXI4, but uses a custom bus interface and is architecturally below it. You can also take control over refresh and ZQ calibration timings if you wish.
3. native interface, at this level there is no longer an address, but rank, bank, row, column.
4. PHY only. This is basically only gives you a physical interface, leaving actual controller implementation to you.
Xilinx recommends using levels 1 or 2, I've been using mostly level 1 (AXI4 interface). And just to give you some idea of the amount of resources it takes, a 32 bit DDR3 controller uses 6157 LUTs and 5361 register when AXI4 interface is used (so all memory controller layers are in use). These numbers obviously fluctuate a bit, but they are in the same ballpark. An A100T device has 63400 LUTs and 126800 registers.

Nockieboy quickest route would be to cut out my DDR3 PHY only layer and just tie my Multiport controller to Xilinx's level 2 interface.  When my multiport sends a read command, it also sends at the same time a 4 bit 'ID' I called the 'read vector' which you may need to manually return with Xilinx's DDR3 read result in parallel.  Everything on the user IO side of my multiport should then be backwards compatible with Nockieboy's existing system, including the multiwindow VGA system.

(Note that the multiwindow system will need 2 Intel inferred dualport dual clock ram blocks changed into Xilinx's equivilant as well as PLL clocks.)

[nockieboy]

Okay, well it looks like I'm going to have to bite the bullet and learn how to use KiCAD then, despite having serious dislike of it.   Maybe once I've overcome the UI issues and odd UX choices, I'll grow to like it.

Really? From my (admittedly VERY limited) experience with EasyEDA I found it's interface to be remarkably similar to KiCAD.
But to tell you the truth, at various times I used DipTrace, Eagle CAD, Orcad/Allergo, Altium Designer and KiCAD, and they all suck in some way or the other. So you basically just need to get used to idiosyncrasies of whichever eCAD you choose, and stick to it. For you I would naturally recommend KiCAD because it's free, and it's quite good (see the project in my signature for the proof - and keep in mind that that project was done in a previous major version of KiCAD).

I've downloaded KiCAD 6 and see virtually nothing has changed.  I just don't have time to wade through padded-out video tutorials (all people seem to make these days to attract the YouTube algorithm) on how to do basic things like set up multiple schematics etc. when I could be launching straight into parts selection and wiring them up in a schematic in EasyEDA.  It helps that I've already got schematics for all the IO etc. done already.  All I really need to do is design the power supplies, wire up the FPGA's IOs and configuration and I can move on to PCB design.

I'm going to stick with EasyEDA for these PCB designs as it'll take me long enough to complete them on an EDA I'm familiar with, let alone one I need to learn how to use.  Once it's done, I'll see about transferring the schematics and PCBs to KiCAD, if the interest is out there for it.  Or I might see if I can progress both alongside each other.  Depends on a lot.  Change of job role coming up in February which means the amount of time I have to do any of this is really hard to tell at the moment.

Maybe I'll be forced to switch to it if there's a killer feature I need for this project that EasyEDA doesn't do but KiCAD does - we'll have to wait and see.

Nockieboy quickest route would be to cut out my DDR3 PHY only layer and just tie my Multiport controller to Xilinx's level 2 interface.  When my multiport sends a read command, it also sends at the same time a 4 bit 'ID' I called the 'read vector' which you may need to manually return with Xilinx's DDR3 read result in parallel.  Everything on the user IO side of my multiport should then be backwards compatible with Nockieboy's existing system, including the multiwindow VGA system.

(Note that the multiwindow system will need 2 Intel inferred dualport dual clock ram blocks changed into Xilinx's equivilant as well as PLL clocks.)

I suspect it should be fairly straightforward as you've designed the DDR3 controller with multiple platforms in mind anyway.  Got to get the boards done first though, and this project is going to represent a significant leap forward for me in complexity.

I hope you all had a great Christmas and are resting up for the New Year's celebrations, however you choose to celebrate them (or not!)

[BrianHG]

I suspect it should be fairly straightforward as you've designed the DDR3 controller with multiple platforms in mind anyway.  Got to get the boards done first though, and this project is going to represent a significant leap forward for me in complexity.

I hope you all had a great Christmas and are resting up for the New Year's celebrations, however you choose to celebrate them (or not!)

First step, simulate Xilinx's DDR3 controller replacing my PHY module.
Then run all my other sims related to your project...

If I remember correctly, you should have a Z80 bus bridge sim tied to the DDR3 controller.
Also, I have a native 'BrianHG_DDR3_CONTROLLER_v16_top_tb.sv' testbench which does verify full sequential bursts.

[asmi]

I've downloaded KiCAD 6 and see virtually nothing has changed.  I just don't have time to wade through padded-out video tutorials (all people seem to make these days to attract the YouTube algorithm) on how to do basic things like set up multiple schematics etc. when I could be launching straight into parts selection and wiring them up in a schematic in EasyEDA.  It helps that I've already got schematics for all the IO etc. done already.  All I really need to do is design the power supplies, wire up the FPGA's IOs and configuration and I can move on to PCB design.

I'm going to stick with EasyEDA for these PCB designs as it'll take me long enough to complete them on an EDA I'm familiar with, let alone one I need to learn how to use.  Once it's done, I'll see about transferring the schematics and PCBs to KiCAD, if the interest is out there for it.  Or I might see if I can progress both alongside each other.  Depends on a lot.  Change of job role coming up in February which means the amount of time I have to do any of this is really hard to tell at the moment.

Maybe I'll be forced to switch to it if there's a killer feature I need for this project that EasyEDA doesn't do but KiCAD does - we'll have to wait and see.

It's up to you of course, just want to mention that if you do decide to implement an FPGA module as a separate board than nothing prevents you from designing it in a different eCAD from your other boards. I will try my best to help you out no matter what tool you pick (assuming I can get my hands on it of course).

I suspect it should be fairly straightforward as you've designed the DDR3 controller with multiple platforms in mind anyway.  Got to get the boards done first though, and this project is going to represent a significant leap forward for me in complexity.

Here I would have to disagree - you can do simulations before you design anything. In fact this is the recommended flow to make sure that a pinout you choose will actually work and close timings. And the higher speed your IO interfaces are running at, the more important it is to simulate simulate simulate before you even create an empty project in your eCAD of choice. I don't even think about designing anything until I have at least a skeleton design ready & working in simulation with I/O interface blocks being the most important parts - because while place & route has a lot of flexibility regarding internal parts of design, I/O components are often hard-tied to specific I/O locations through pin choices you made, and so there is much less a router can do to close the timing if it's tight and/or your choice of I/O is sub-optimal. Again, there are ways to deal with cases when for one reason or another router is not able to close timing, but it's best to avoid getting into these situations if it can be helped.
For my 7 series-based board I typically create a simple CPU design with memory controller and some basic I/O stuff like QSPI, UART and GPIO, as this is super-easy and quick in Vivado (you don't need to write a single line of HDL!), and allows to confirm that DDR controller works like it should, and pinout of memory works too.

I hope you all had a great Christmas and are resting up for the New Year's celebrations, however you choose to celebrate them (or not!)

I very much do plan to celebrate. Already "celebrated" Christmas by sleeping through most of those days off as I could never get enough sleep during regular weeks And I'm really looking forward for another 3 straight days off to catch some more zzz'es

[BrianHG]

I suspect it should be fairly straightforward as you've designed the DDR3 controller with multiple platforms in mind anyway.  Got to get the boards done first though, and this project is going to represent a significant leap forward for me in complexity.

Here I would have to disagree - you can do simulations before you design anything. In fact this is the recommended flow to make sure that a pinout you choose will actually work and close timings. And the higher speed your IO interfaces are running at, the more important it is to simulate simulate simulate before you even create an empty project in your eCAD of choice. I don't even think about designing anything until I have at least a skeleton design ready & working in simulation with I/O interface blocks being the most important parts - because while place & route has a lot of flexibility regarding internal parts of design, I/O components are often hard-tied to specific I/O locations through pin choices you made, and so there is much less a router can do to close the timing if it's tight and/or your choice of I/O is sub-optimal. Again, there are ways to deal with cases when for one reason or another router is not able to close timing, but it's best to avoid getting into these situations if it can be helped.
For my 7 series-based board I typically create a simple CPU design with memory controller and some basic I/O stuff like QSPI, UART and GPIO, as this is super-easy and quick in Vivado (you don't need to write a single line of HDL!), and allows to confirm that DDR controller works like it should, and pinout of memory works too.

Yes, simulate everything.
I would have gotten nowhere without simulating everything when engineering my DDR3 controller & video processor as well as the Z80 bridge which has radically improved once we did a full simulation of it.
We also simulated your PSG to prove and get the right sound.

You have every testbench I already created.  I already prepared for you every sub-module simulation as well as complete top level system sims.  I'm sure what I have made can all be adapted to Vivado, or, you can bring Vivado's primitives back into Mosdelsim.  You will end up building a useless board if you do not take everything I've already provided, swap in Vivado's DDR3 controller into the right point in the code you have and prove in advance that everything works before you build a PCB.

[nockieboy]

It's up to you of course, just want to mention that if you do decide to implement an FPGA module as a separate board than nothing prevents you from designing it in a different eCAD from your other boards. I will try my best to help you out no matter what tool you pick (assuming I can get my hands on it of course).

Thanks asmi.   I'll be using EasyEDA initially to make a core FPGA card - probably 6 layers - to house the FPGA itself, clock, DDR3 chips, JTAG and power supplies.  Then I can make a carrier board housing the IO hopefully with no major issues, as the IO will be a lot slower than anything on the core card itself.  I'm hoping I can make the core card small enough to fit within the form-factor of my computer, so I can make a dedicated carrier card with level converters and HDMI/USB I/O that fits the uCOM stack and a development carrier board for general use of the FPGA for anyone and everyone who's interested.

Here I would have to disagree - you can do simulations before you design anything. In fact this is the recommended flow to make sure that a pinout you choose will actually work and close timings. And the higher speed your IO interfaces are running at, the more important it is to simulate simulate simulate before you even create an empty project in your eCAD of choice. I don't even think about designing anything until I have at least a skeleton design ready & working in simulation with I/O interface blocks being the most important parts - because while place & route has a lot of flexibility regarding internal parts of design, I/O components are often hard-tied to specific I/O locations through pin choices you made, and so there is much less a router can do to close the timing if it's tight and/or your choice of I/O is sub-optimal. Again, there are ways to deal with cases when for one reason or another router is not able to close timing, but it's best to avoid getting into these situations if it can be helped.
For my 7 series-based board I typically create a simple CPU design with memory controller and some basic I/O stuff like QSPI, UART and GPIO, as this is super-easy and quick in Vivado (you don't need to write a single line of HDL!), and allows to confirm that DDR controller works like it should, and pinout of memory works too.

Fair point.  Funny how I still don't go straight to simulation as the first part of the process.

If it's any consolation, I'm not designing the schematics and assigning I/O 'blind'.  I'm using reference design(s) wherever possible - I've not found a lot on the XC7A100T unfortunately (no full schematics, anyway), but I've got some reference material I'm using based on the AC7100B board.  It uses two DDR3 chips and includes their pin assignments, so I'm hoping that mirroring those assignments will avoid a lot of the pitfalls that you've mentioned re: pin selection.  There's no detailed PCB drawings unfortunately, so routing two 16-bit datapaths will still be on me.

I'm aiming to create the same 32-bit data bus width with up to 25Gb/s bandwidth - hopefully that'll be sufficient for the project, @BrianHG?

Once the 32-bit DDR3 layout and traces are done, as far as I can see that's the hardest part done; I'll route out the remainder of the IO to the mezzanine connectors and, as these run at much lower speeds, selection and routing will be slightly less of a concern.  The manual I linked to above for the AC7100B includes GTP transceivers and differential signal paths?  I don't know much about these and am not looking to include them (or their power supplies) on the core FPGA card, unless anyone can convince me they're needed?

[BrianHG]

My DDR3 bus isn't 32 bit.  It is the size of a BL8 command (Burst Length 8 ).

So, currently, on the DECA, with a 16bit DDR3 ram chip, 8 X 16bit = 128bit bus.

This is the first thing you should simulate, setting Xilinx's DDR3 controller to 1x 16bit ram chip with a 128bit read and write bus.  The write data should contain a 'Byte Enable' for each 8 bits.

(From here on in, here is an optional path)

Next, rename my DDR3 controller to "BHG_DDR3_Controller_top_using_Xilinx_core.sv' and also rename it's internal 'Altera PHY' source to 'Xilinx DDR3 PHY level 2', and copy my altera's beginning of it's module to your newly renamed 'Xilinx DDR3....'.  Inside that new module, you will need to instantiate Xilinx's controller with it's PLL wired to the existing IO ports.

You will need to also create/move/adapt to a Xilinx PLL.  Since I know nothing about Xilinx, the PLL may also be instantiated by it's own DDR3 controller, so that may just move inside it's PHY.

Don't forget that my multiport still needs to know all the DDR3 parameters, like addressing BANK-ROW order which you set in Xilinx's DDR3 controller so it may properly optimize port selection and burst interleave and sizing.  It's probably just easier to translate my original incoming parameters to fit Xilinx's DDR3 controller parameters.

Try to simulate my controller's existing test-bench with the new Xilinx setup.

If that works, you can try the Z80 bus interface testbench simulation with the new controller.

Then worry about increasing to 2x 16bit DDR3 ram chips, IE: now a 256 bit internal bus.

Good luck.

[BrianHG]

Ok, just too sleepy to realize you were talking about the bus size on the PCB, not the internal bus size in the FPGA.  However, my above comments should still be taken into account when adapting the design.

[dolbeau]

I'll be using EasyEDA initially to make a core FPGA card - probably 6 layers - to house the FPGA itself, clock, DDR3 chips, JTAG and power supplies.
(...)
If it's any consolation, I'm not designing the schematics and assigning I/O 'blind'.  I'm using reference design(s) wherever possible - I've not found a lot on the XC7A100T unfortunately (no full schematics, anyway)

There's a lot of SoM boards out there with full schematics. You can also use I/O designs from non-A100T Artix-7, as they are built on the same 'building blocks'. HR I/O banks from one will work the same way as HR I/O banks from another. Ditto HP.

You can look up e.g. the Trenz TE0712 and others from Trenz, ZTex 2.13 and others from ZTex, etc. You can also look at cheaper QmTech boards with integrated peripherals like the Wukong, etc. All of those have full schematics available, the boards from Trenz and Ztex have multiple variants with a range of Artix-7.

There's not that many designs with an Artix-7 and a full PCB in addition to schematics that I know of, but there's at least one from Antmicro. The schematics include comments on what is needed for various Artix-7 (decoupling, ...).

I've looked into doing the same thing (FPGA + power + flash + DDR3) for my projects that currently use a ZTex 2.13a plugged into my boards, but most examples/reference materials suggest 8 layers (expensive PCB!), and routing the high-speed signals for the DDR3 still seem too complex for my skills at least when I look at Xilinx UG583.

[asmi]

Thanks asmi.   I'll be using EasyEDA initially to make a core FPGA card - probably 6 layers - to house the FPGA itself, clock, DDR3 chips, JTAG and power supplies.  Then I can make a carrier board housing the IO hopefully with no major issues, as the IO will be a lot slower than anything on the core card itself.  I'm hoping I can make the core card small enough to fit within the form-factor of my computer, so I can make a dedicated carrier card with level converters and HDMI/USB I/O that fits the uCOM stack and a development carrier board for general use of the FPGA for anyone and everyone who's interested.

I kind of suspect that Deca board has more than 6 layers judging by the density of components. Perhaps 10 or 12 would be my guess. Most SoMs I came across are 10 or 12 layers.

If it's any consolation, I'm not designing the schematics and assigning I/O 'blind'.  I'm using reference design(s) wherever possible - I've not found a lot on the XC7A100T unfortunately (no full schematics, anyway), but I've got some reference material I'm using based on the AC7100B board.  It uses two DDR3 chips and includes their pin assignments, so I'm hoping that mirroring those assignments will avoid a lot of the pitfalls that you've mentioned re: pin selection.  There's no detailed PCB drawings unfortunately, so routing two 16-bit datapaths will still be on me.

Don't do this, especially using random chinese board as a reference is not the greatest idea. Instead, get Vivado/Vitis set up and come up with your own, and odds are you are going to change it yet again when you will do routing, which will need to be verified again.
But before you do any of that, come up with an exact  list of stuff you want connected to the FPGA to do some planning, and so you will know which I/O interfaces you will need to implement.

I'm aiming to create the same 32-bit data bus width with up to 25Gb/s bandwidth - hopefully that'll be sufficient for the project, @BrianHG?

That is the question I've asked a few posts back, if neccessary/required, it's possible to implement two 32 bit controllers, or some other mix of controllers if it will be beneficial to your project. For what it's worth, real video cards have a bunch of independent memory controllers as it allows video core to access several non-adjacent memory regions on the same clock, but adds it's own level of complexities regarding managing what is stored on each of connected memory devices.

Once the 32-bit DDR3 layout and traces are done, as far as I can see that's the hardest part done; I'll route out the remainder of the IO to the mezzanine connectors and, as these run at much lower speeds, selection and routing will be slightly less of a concern.  The manual I linked to above for the AC7100B includes GTP transceivers and differential signal paths?  I don't know much about these and am not looking to include them (or their power supplies) on the core FPGA card, unless anyone can convince me they're needed?

I understand your desire to rush to PCB design, but trust me, the time spent planning is the time well spent as it might ultimately save you a bunch of money on PCB respins and redesigns.

32 bit DDR3 layout might be the hardest one, but certainly not the fastest one. In your case HDMI will probably be the fastest one (beyond GTPs). As per official specs any regular I/O can go up to 1.25 Gbps per line, but in reality I've never seen an FPGA device that is not capable of doing 1080p@60 at 1.45 Gpbs per differential line. These are certainly faster than 400 MHz DDR3 signals, so don't get too caught up with DDR3 being the biggest obstacle. It might seem that way on a first glance, but it doesn't mean that it actually is. I know DDR3 looks intimidating, trust me I felt the same way back when I was about to embark on my very first one, but it might very well be simply a red herring with real complexity being elsewhere.

GTP are multi-gigabit transceivers and are in my opinion the most exciting feature of these devices, so I would absolutely recommend you to route them to the high-speed connectors and wire up their power supplies. The PDS for GTPs is a bit involed because it requires two ultra-clean power rails (<10 mVpp ripple is no joke!) at 1 and 1.2 V nominal, and the signal routing needs to be very careful as these lines can potentially run at over 6 Gbps, but that is precisely what makes them so exciting. That is my opinion of course and you are free to ignore it, but I think you will miss out a lot if you skip that part.

[asmi]

I've looked into doing the same thing (FPGA + power + flash + DDR3) for my projects that currently use a ZTex 2.13a plugged into my boards, but most examples/reference materials suggest 8 layers (expensive PCB!), and routing the high-speed signals for the DDR3 still seem too complex for my skills at least when I look at Xilinx UG583.

With recent pricing update at JLCPCB it's no longer the case, as five 10x10 cm 8 layer PCBs are now under $100.