Planning/design/review for a 6-layer Xilinx Artix-7 board for DIY computer.

[BrianHG]

Those MPM3683s are around £7 per piece - a touch on the expensive side for what I want, if I'm honest.  The MPM3833s, however, are less than half that at around £3/piece - I'll look a little closer at these as an option once I start worrying about space on the PCB.

When comparing prices, remember that parts I've mentioned are modules and so already include inductor and a bunch of other passives.

That's the one thing I've never done - assigning colours to traces.  That must explain our differing experiences of EasyEDA.

I've attached two views of the same layout (this is a 64 bit DDR3 SO-DIMM interface), one without trace coloring (so trace color just indicates a layer it's on), and another one where different trace groups (address/control group and 8 byte lanes) were assigned different colors. And two more views of a single layer from that layout, again with and without trace coloring. Which one is easier to read? And also imagine that none of those traces are there yet (which is going to be your case), and you need to quickly figure out how to place components such that it will be easier to route.

The cyan DDR3 command lines length seem needlessly long with a few too way too short and all messed up.
It's like you set a length tolerance and some lines went maximum while others went minimum.

The DQ & DQS seems ok.

Is it possible that the tighter DQ/DQS specs just made them much better and for some reason your cyan command lines went min and max instead of optimal length?  Min & Max routing doesn't mean it wont work, but that will lower the tip top possible performance.

[BrianHG]

It was a bus timing and contingency issue.
The answer was change all the 74F245s, 74F574s & 74F04 to 74HC245, 74HC574 and 74HC04.

The higher 2.5v high logic level (vs 1.2v ttl) input delayed output enables long enough to clear everything up and the card ran cool.

What did they say when you came out with that peach of a solution?  Might have been a few embarrassed engineers (or one, perhaps) slinking away into the background. 

I think they themselves had the Fusion 40 card designed by a contracted third party who made it to the Amiga 68k expansion CPU slot spec.  My solution was a glance look observation patch type fix which I never got paid for.  The fix worked and from there on in, anything new they designed was done in-house.

It was nothing more than a relief from them as the fix was quick and cheap for their cards already in the market which needed patching.  Thankfully, the ICs were on sockets.

[asmi]

The cyan DDR3 command lines length seem needlessly long with a few too way too short and all messed up.
It's like you set a length tolerance and some lines went maximum while others went minimum.

The DQ & DQS seems ok.

Is it possible that the tighter DQ/DQS specs just made them much better and for some reason your cyan command lines went min and max instead of optimal length?  Min & Max routing doesn't mean it wont work, but that will lower the tip top possible performance.

Address/control lines in DDR3 needs to be at least as long as the longest DQ group due to flyby routing, and some of the routing is on different layers so it's hard to see it on a screenshot. But that wasn't the point - I wanted to demonstrate how helpful trace coloring is and why I use it everywhere. It's especially useful in placing when you can quickly see where major groups of traces needs to go. Ratlines can help, but when you've got over 600 pin devices (676 balls in that screenshot), it can be really hard to figure out what needs to go where and how to place components to make subsequent routing easier.

[BrianHG]

Here is an old 800mbps DDR2 layout of mine.  Only the crucial differential CLK, and DQS trace lengths were matched within spec.

All the others DQ were within +/- 5mm within their respected domains while the command had a +/-10mm tolerance.

Each DQ/DQS signal went through 1 via from FPGA on top to sodimm module on bottom.
Each command went through 2 vias.
(Green = top, Red = bottom, cyan/magenta = bottom layers.)

If I wanted DDR3, I would need to add some swiggles to length match some of the shorter DQ traces and super balance the DQS traces to achieve 1866mbps spec, or, just program the trace lengths into the FPGA's IO fine delay compensation.

[asmi]

If I wanted DDR3, I would need to add some swiggles to length match some of the shorter DQ traces and super balance the DQS traces to achieve 1866mbps spec, or, just program the trace lengths into the FPGA's IO fine delay compensation.

Nope, that's not how it works. It's actually the other way around - you pull package delays data from your SoC or FPGA and then adjust your layout to take these delays into consideration. 400 MHz DDR2 and 933 MHz DDR3 are very different animals, and layouts for them are going to be very different when it comes to tolerances, impedance and crosstalk. A lot of things you can get away with in slower interface will become a critical flaw at fast one. Your layout as is will probably work for DDR3 (through I'm not sure as some SoCs require flyby routing and will not work with balanced tree of DDR2), but only for slower speeds.

[nockieboy]

When comparing prices, remember that parts I've mentioned are modules and so already include inductor and a bunch of other passives.

Oh I haven't forgotten that.  The MPM3833 is my preferred choice at the moment, only working out slightly more expensive than the TPS option (even factoring in the inductor), but looks more easily solderable (not as small) but overall taking up less space than the TPS part because of not needing the external inductor.

That's the one thing I've never done - assigning colours to traces.  That must explain our differing experiences of EasyEDA.

I've attached two views of the same layout (this is a 64 bit DDR3 SO-DIMM interface), one without trace coloring (so trace color just indicates a layer it's on), and another one where different trace groups (address/control group and 8 byte lanes) were assigned different colors. And two more views of a single layer from that layout, again with and without trace coloring. Which one is easier to read? And also imagine that none of those traces are there yet (which is going to be your case), and you need to quickly figure out how to place components such that it will be easier to route.

I think they all look fantastic.  The colour ones are clearer to see which signals are grouped, but to me at least that's only an informative 'nice-to-have' and something I could maybe live without.  Bearing in mind I haven't tried routing DDR3 or even DDR2 yet.

I think they themselves had the Fusion 40 card designed by a contracted third party who made it to the Amiga 68k expansion CPU slot spec.  My solution was a glance look observation patch type fix which I never got paid for.  The fix worked and from there on in, anything new they designed was done in-house.

It was nothing more than a relief from them as the fix was quick and cheap for their cards already in the market which needed patching.  Thankfully, the ICs were on sockets.

You never got paid for it?   It obviously greased the wheels for future work with them, though?

Nope, that's not how it works. It's actually the other way around - you pull package delays data from your SoC or FPGA and then adjust your layout to take these delays into consideration. 400 MHz DDR2 and 933 MHz DDR3 are very different animals, and layouts for them are going to be very different when it comes to tolerances, impedance and crosstalk. A lot of things you can get away with in slower interface will become a critical flaw at fast one. You layout as is will probably work for DDR3 (through I'm not sure as some SoCs require flyby routing and will not work with balanced tree of DDR2), but only for slower speeds.

Oookay.  I've clearly got a lot to learn about this stuff before I can get anywhere near routing a DDR3 chip to a standard that you guys will be happy with.



By the way, Happy New Year everyone.

[asmi]

I think they all look fantastic.  The colour ones are clearer to see which signals are grouped, but to me at least that's only an informative 'nice-to-have' and something I could maybe live without.  Bearing in mind I haven't tried routing DDR3 or even DDR2 yet.

Well technically everything eCAD has to offer can be done without and is a nice-to-have. But it doesn't mean it should.
Anyway, we will see how it goes when it comes to actually doing it.

Oookay.  I've clearly got a lot to learn about this stuff before I can get anywhere near routing a DDR3 chip to a standard that you guys will be happy with.

Don't you worry as you won't be implementing a 933 MHz DDR3 interface any time soon. The best Artix-7 can do is 533 MHz, and that is only in speed grade 3, while devices we've ordered are speed grade 2, which can only go up to 400 MHz. If you we do out layout and pinout right, we can later perhaps try overcloaking them to speed grade 3, but that is not guaranteed to work so I would only rely on having 400 MHz interface.

By the way, Happy New Year everyone.

We're still waiting for one to come over here

[nockieboy]

By the way, Happy New Year everyone.

We're still waiting for one to come over here

The hangover has cleared at last, the decorations are down and I'm thinking about the power supply for the FPGA now.

The 1.0V rail (VCCINT) is going to need to supply some power - I've been doing the schematics for the other power rails and moved on to design VCCINT's supply and realised that the TPS62823 only supplies a maximum 3A.  Will that be enough for the FPGA?  I know it's hard to guesstimate power consumption with an FPGA as depends entirely on the gate usage, frequency etc, but the Chinese schematic/reference of unspecified origin that I was using quotes a 3A-capable supply for VCCINT.   Is there any safety margin in that or should I design for something a little beefier?

[nockieboy]

Further to the above post, here's the power schematic for the board, first version.  Don't ask why I'm using a TPS62823 for VCCINT and MPM3833s for the rest - for some reason I'd gotten confused and thought the TPS part was able to put out more current, so I kept it for VCCINT as I thought it would supply more current if needed.  That's clearly not the case, so I can (and probably will) replace the TPS part a copy-paste of one of the MPM3833 supplies beneath it, with R15/R16 adjusted accordingly.  That TPS part looks really small and may cause me some headaches trying to solder it, which is a good reason to use an MPM3833 part - that and I won't need the inductor.

As always, feedback appreciated.  Remember I'm trying to balance cost against footprint size and ease of construction.

[asmi]

Further to the above post, here's the power schematic for the board, first version.  Don't ask why I'm using a TPS62823 for VCCINT and MPM3833s for the rest - for some reason I'd gotten confused and thought the TPS part was able to put out more current, so I kept it for VCCINT as I thought it would supply more current if needed.  That's clearly not the case, so I can (and probably will) replace the TPS part a copy-paste of one of the MPM3833 supplies beneath it, with R15/R16 adjusted accordingly.  That TPS part looks really small and may cause me some headaches trying to solder it, which is a good reason to use an MPM3833 part - that and I won't need the inductor.

A100T can consume up to 7 A of current on Vccint rail, so none of those parts will work. That's why I offered to take a look at MPM3683-7 - it can provide up to 8 A so there is some margin, and as per datasheet, it seems to have low enough ripple to power 1 V power line of GTPs. GTPs will also require a clean 1.2 V rail with a current of about 0.4 amps, so we can use a good quality LDO to step it down from 1.8 V rail. This might be a bit tricky as not all LDO can go that low for power, pre-biased LDO will probably be the best.

I would suggest to use DDR3 instead of DDR3L as it will theoretically give us ability to overclock things to 533 MHz interface. So switch 1.35 V rail to 1.5 V nominal.

You will also need a DDR3 termination regulator. Something like MP20075 - it provides both Vref (midrail) for memory and FPGA, and can sink/source up to 3 amps of current, which seems more than enough for our design.

Also do we have a full list of all peripherals which will be connected to FPGA? Some of those might require Vccio other than 3.3 V. You also might want to design-in some prexibility to change Vccio of some IO banks.

[Someone]

As always, feedback appreciated.  Remember I'm trying to balance cost against footprint size and ease of construction.

Dont split Vccint and Vccbram unless you are doing suspend/low power shenanigans.

But as with some of the comments already, STOP. Put a representative design into the FPGA tools before doing any of the hardware thinking/layout/planning. Then you will have a reasonable power estimate, some validated pin assignments, and a clocking tree.

Once the power estimates are in, LDOs will probably do half the rails if you do care about space and cost. The cost in $$ and footprints will likely end up in the capacitors as much as the power supplies themselves so there is pressure to design the distribution and tuning well. If you are trying to squeeze space and cost, eliminate that sequencing chip and either use enables or make the power rails all come on (and off) together (permissible solution that Xilinx supports).

[nockieboy]

A100T can consume up to 7 A of current on Vccint rail, so none of those parts will work. That's why I offered to take a look at MPM3683-7 - it can provide up to 8 A so there is some margin, and as per datasheet, it seems to have low enough ripple to power 1 V power line of GTPs. GTPs will also require a clean 1.2 V rail with a current of about 0.4 amps, so we can use a good quality LDO to step it down from 1.8 V rail. This might be a bit tricky as not all LDO can go that low for power, pre-biased LDO will probably be the best.

Okay, cool - will do a redesign to incorporate the increased power demand, and had forgotten to include the GTP supplies.  That MPM3683 is not a cheap part though, especially considering it only has one output.  Would this be a suitable alternative? SIC402ACD-T1-GE3 - single 10A output.

Are the MPM3833s okay for the other rails though?  If it's just VCCINT that needs more juice, then the price of the MPM3683 isn't an issue.

I would suggest to use DDR3 instead of DDR3L as it will theoretically give us ability to overclock things to 533 MHz interface. So switch 1.35 V rail to 1.5 V nominal.

Okay, I've switched to DDR3 (instead of DDR3L) chips (these ones) and have increased the voltage from 1.35V to 1.5V for VCCDDR.  I've dropped the DDR3 size down to 64MBx32. 

You will also need a DDR3 termination regulator. Something like MP20075 - it provides both Vref (midrail) for memory and FPGA, and can sink/source up to 3 amps of current, which seems more than enough for our design.

Also do we have a full list of all peripherals which will be connected to FPGA? Some of those might require Vccio other than 3.3 V. You also might want to design-in some prexibility to change Vccio of some IO banks.

I'll work on this tomorrow.  For some reason I had some spare time and fell into the power schematic, so did some work on that while it seemed like a good idea.

Dont split Vccint and Vccbram unless you are doing suspend/low power shenanigans.

They're only split in terms of the net names, so I can ensure sufficient decoupling caps for the BRAM connections, but I'm going to get rid of the link/separate net name and just box off the BRAM decoupling caps on the schematic.  I'll still know what to do with them on the PCB design.

But as with some of the comments already, STOP. Put a representative design into the FPGA tools before doing any of the hardware thinking/layout/planning. Then you will have a reasonable power estimate, some validated pin assignments, and a clocking tree.

I guess because this is the bit I find easiest.  You're right though, I need to get more of an overview before rushing headlong into the design.

Once the power estimates are in, LDOs will probably do half the rails if you do care about space and cost. The cost in $$ and footprints will likely end up in the capacitors as much as the power supplies themselves so there is pressure to design the distribution and tuning well. If you are trying to squeeze space and cost, eliminate that sequencing chip and either use enables or make the power rails all come on (and off) together (permissible solution that Xilinx supports).

Ah okay, I didn't know how the Xilinx FPGA would be with just letting all the horses out of the gate at once.  If it's not an issue, I'll scrub the sequencer then - thanks.

[Someone]

Dont split Vccint and Vccbram unless you are doing suspend/low power shenanigans.

They're only split in terms of the net names, so I can ensure sufficient decoupling caps for the BRAM connections, but I'm going to get rid of the link/separate net name and just box off the BRAM decoupling caps on the schematic.  I'll still know what to do with them on the PCB design.

Either you are going for a cost/space constrained design, and they are the same power rail with caps all sharing the work so need to be designed for the task.

or you are just copying the UG483 recommended design because there is no pressure on size/cost

Decoupling methods other than those presented in these tables can be used, but the decoupling network should be designed to meet or exceed the performance of the simple decoupling networks presented here. The impedance of the alternate network must be less than or equal to that of the recommended network across frequencies from 100 KHz to 10 MHz.
Because device capacitance requirements vary with CLB and I/O utilization, PCB decoupling guidelines are provided on a per-device basis based on very high utilization so as to cover a majority of use cases. Resource usage consists (in part) of:

Xilinx provide pretty good documentation, but you have to actually read it. Very few systems are going to push the power distribution up to or beyond their baseline example.

....noting the proposed MPM3833C solution is already sitting outside their worked/reference example, so you're immediately off into needing to do the calculations yourself.

[asmi]

Okay, cool - will do a redesign to incorporate the increased power demand, and had forgotten to include the GTP supplies.  That MPM3683 is not a cheap part though, especially considering it only has one output.  Would this be a suitable alternative? SIC402ACD-T1-GE3 - single 10A output.

That part requires an insane number of external components to get it going. I looked at it in the past and I think I even bought some of them to try out.

Are the MPM3833s okay for the other rails though?  If it's just VCCINT that needs more juice, then the price of the MPM3683 isn't an issue.

I thought I was pretty clear that I offered MPM3683-7 only for Vccint rail. MPM3833s should be fine for other rails.

Okay, I've switched to DDR3 (instead of DDR3L) chips (these ones) and have increased the voltage from 1.35V to 1.5V for VCCDDR.  I've dropped the DDR3 size down to 64MBx32. 

All DDR3L chips can work in DDR3 mode (at 1.5 V). So no need to switch devices. I would plan for 4Gb parts - or even 8Gb ones for the layout, but you can always place smaller part - these devices are fully backwards compatible.

I'll work on this tomorrow.  For some reason I had some spare time and fell into the power schematic, so did some work on that while it seemed like a good idea.

That's a good start, but we will need to make a strategic decision on the connectors used for the module, because those connectors' pin power rating will determine how many pins we will need to allocate for the main power in. High speed connectors typically have a power rating of 2 A per pin assuming no two adjacent pins are used for power. Since the module can consume over 10 W of power (+ we will need to leave a provision for a fan, which can consume up to 1 W), we will need to plan out power budget.
But before we can settle on connectors, we will need to figure out how many pins we will need and what's going to be a pinout. This is why I keep asking about listing everything that is to be connected. We will probably want to route out as many IO as we can, so maybe it will be better to just see which IOs are going to be available.

They're only split in terms of the net names, so I can ensure sufficient decoupling caps for the BRAM connections, but I'm going to get rid of the link/separate net name and just box off the BRAM decoupling caps on the schematic.  I'll still know what to do with them on the PCB design.

Your schematic has a zero Ohm resistor dividing those lines. No need for that crap. Just do a schematic like I did in mine (see a pdf I linked few posts above), and include a table from Xilinx user guide which lists what caps and of what nominal will we need. See UG483, chapter 2, section PCB Decoupling Capacitors -> Recommended PCB Capacitors per Device. I typically use 0201 caps for 0.47 uF (as these need to be places inside of via field), 0402s for 4.7 uF, larger caps size is not very important as they can be placed anywhere, so I usually use whatever cap I happen to have - like 0805 or even 1210.

Ah okay, I didn't know how the Xilinx FPGA would be with just letting all the horses out of the gate at once.  If it's not an issue, I'll scrub the sequencer then - thanks.

Yeah, I never bothered with rail sequencing unless I had a system controller MCU for some other reason which I could just reuse for sequencing, and never had any problems with it.

[asmi]

....noting the proposed MPM3833C solution is already sitting outside their worked/reference example, so you're immediately off into needing to do the calculations yourself.

Not sure I understand what do you mean here?

[nockieboy]

Dont split Vccint and Vccbram unless you are doing suspend/low power shenanigans.

They're only split in terms of the net names, so I can ensure sufficient decoupling caps for the BRAM connections, but I'm going to get rid of the link/separate net name and just box off the BRAM decoupling caps on the schematic.  I'll still know what to do with them on the PCB design.

Either you are going for a cost/space constrained design, and they are the same power rail with caps all sharing the work so need to be designed for the task.

or you are just copying the UG483 recommended design because there is no pressure on size/cost

Initially the VCCBRAM net didn't exist, I just ran VCCINT to the VCCBRAM inputs on the FPGA.  However, whilst copying the UG483 recommended decoupling layout, I realised there are a few caps specifically for the VCCBRAM inputs, so I thought I'd just split the net via a 0R resistor so that I could see more clearly which caps were for VCCBRAM and which weren't by their net names instead of just by their annotation.  There are other ways I could have done this that wouldn't be as clear on the PCB layout, like just 'boxing off' a few VCCINT caps on the decoupling schematic for the VCCBRAM connectors, but I went for this one knowing I'd be likely to change it in the future.  Didn't think it would be that much of an issue to be honest.

I don't know if you read the TL;DR at the start of this thread, but I'm not a professional PCB designer or electronics engineer.  I'm a hobbyist, at best.  When I talk about pressure on size/cost, I'm talking about saving money on big things like chip choice, not worrying about saving a few thousandths of a dollar on a cap here and there.  I'm literally going to be building (hopefully) one, maybe up to four (to use all the FPGAs I've bought) of these things.  I'll be making the designs public, so I suppose there's that, but I'm not designing for mass production and so perhaps I'm not looking at this from the same angle you are with your career/expertise behind you.

Xilinx provide pretty good documentation, but you have to actually read it. Very few systems are going to push the power distribution up to or beyond their baseline example.

I am.  I have UG470, UG471, UG472, UG473, UG474, UG475, UG482, UG483 and UG586 (amongst others) open in tabs on my browser.  I'm trying to read them as I go and use them as references - I've already used their decoupling design, for example.

I thought I was pretty clear that I offered MPM3683-7 only for Vccint rail. MPM3833s should be fine for other rails.

Sorry;  I either misunderstood, misread or outright forgot.

Okay, I've switched to DDR3 (instead of DDR3L) chips (these ones) and have increased the voltage from 1.35V to 1.5V for VCCDDR.  I've dropped the DDR3 size down to 64MBx32. 

All DDR3L chips can work in DDR3 mode (at 1.5 V). So no need to switch devices. I would plan for 4Gb parts - or even 8Gb ones for the layout, but you can always place smaller part - these devices are fully backwards compatible.

Okay, I'll stick with the original choice then.  I've really got to stop doing this late at night.  I saw 'Vdd=Vddq=1.35V (1.283-1.45V)' and missed the bit underneath that said 'backward compatible to 1.5V'.

But before we can settle on connectors, we will need to figure out how many pins we will need and what's going to be a pinout. This is why I keep asking about listing everything that is to be connected. We will probably want to route out as many IO as we can, so maybe it will be better to just see which IOs are going to be available.

This is the downside to designing a core card that can fit onto any carrier board design - you don't really know what's going to be on the carrier board, so I guess we have to consider worst-case scenarios and try to route as many IOs as possible.

That aside, here's what I really need it to do - everything else is a bonus:

Core board:

2xDDR3 via 32-bit bus @ 400MHz.
JTAG programming port.
1 or 2 IO LEDs for testing the core board without a carrier board.
All power supplies except 5V, which will be provided by carrier board or external source if no carrier connected.

Carrier board (specific to my use-case) - nothing too demanding really other than HDMI:

HDMI TX capable of 1080p @ 60Hz or better.
Audio output via headphone/line out.
USB-A host PHY (CH559 or something similar - open to suggestions - serial/SPI/I2C interface to FPGA) for keyboard/mouse.
USB FPGA programmer.
5V power supply via USB or DC power connector.
80-pin interface to 5V host bus via level converters.
Various buttons/LEDs/RGB LEDs.

I'm going to need at least 60 I/Os for the carrier board for the host interface, then there's the HDMI, JTAG, audio codec, USB, generic IO lines.

Your schematic has a zero Ohm resistor dividing those lines. No need for that crap. Just do a schematic like I did in mine (see a pdf I linked few posts above), and include a table from Xilinx user guide which lists what caps and of what nominal will we need. See UG483, chapter 2, section PCB Decoupling Capacitors -> Recommended PCB Capacitors per Device. I typically use 0201 caps for 0.47 uF (as these need to be places inside of via field), 0402s for 4.7 uF, larger caps size is not very important as they can be placed anywhere, so I usually use whatever cap I happen to have - like 0805 or even 1210.

As I mentioned above, I'm using UG483 already.  I've never used 0201 caps and am a little concerned a butterfly will flap in a garden twenty miles away and they'll all just disappear like a cloud of nanites.   Might have to have some discussion around how you solder those things, as 0402 feels like my limit at the moment.  And I need to get a microscope!

Ah okay, I didn't know how the Xilinx FPGA would be with just letting all the horses out of the gate at once.  If it's not an issue, I'll scrub the sequencer then - thanks.

Yeah, I never bothered with rail sequencing unless I had a system controller MCU for some other reason which I could just reuse for sequencing, and never had any problems with it.

Excellent. The sequencer is gone, like all those 0201s as a mosquito flies by.

[asmi]

Core board:

2xDDR3 via 32-bit bus @ 400MHz.
JTAG programming port.
1 or 2 IO LEDs for testing the core board without a carrier board.
All power supplies except 5V, which will be provided by carrier board or external source if no carrier connected.

How are you going to test a module without a carrier if it is to be powered by the carrier? Also having JTAG both on a module and on a carrier is not a great idea because of stubs which may negatively affect performance of JTAG.
You also forgot to mention QSPI flash for a bitstream. For A100T you will need at least 64 Mb, preferably more to leave some space for application data and code storage.

Carrier board (specific to my use-case) - nothing too demanding really other than HDMI:

HDMI TX capable of 1080p @ 60Hz or better.

You need to be more specific if you only want 1080p@60 or "better", because the former can be implemented using regular IO pins, while for the latter we will need to use GTPs and some additional level shifting circuitry. I would recommend sticking to the former because of simplicity - unless you actually want that "better"

Audio output via headphone/line out.

Ummm, these are analog signals, therefore you will need some sort of audio codec or DAC (if only output is desired). Do you have any specific one in mind, or it doesn't matter?

USB-A host PHY (CH559 or something similar - open to suggestions - serial/SPI/I2C interface to FPGA) for keyboard/mouse.

I'm not familiar with that device, traditionally USB ULPI PHY devices are used for USB 2.0, something like this one: https://www.microchip.com/en-us/product/USB3300

USB FPGA programmer.

This of course it up to you, but I'm not really convinced that placing programming circuitry on every board is such a great idea, as opposed to just buying a single programmer and connecting it to all FPGA boards as neccessary. Also see my note above regarding having JTAGs both on a module and a carrier.

5V power supply via USB or DC power connector.
80-pin interface to 5V host bus via level converters.
Various buttons/LEDs/RGB LEDs.

I'm going to need at least 60 I/Os for the carrier board for the host interface, then there's the HDMI, JTAG, audio codec, USB, generic IO lines.

What about Ethernet? Wouldn't you want your computer to be able to talk to outside world (or maybe just to other devices on your local network)?
What about some sort of USB-UART port to talk to PC for debugging purposes?
What about some kind of storage? SD card interface, or eMMC device, or perhaps something else?

As for other stuff, typically to maximize utility of a module, all IO pins of a bank are routed length-matched such that high-speed peripherals can be connected if desired. And for example for 1G Ethernet PHY you will certainly need that kind of routing. I'm not really sure we will be able to length-match entire banks with 4 routing layers, but we need to at least attempt to do so.

As I mentioned above, I'm using UG483 already.  I've never used 0201 caps and am a little concerned a butterfly will flap in a garden twenty miles away and they'll all just disappear like a cloud of nanites.   Might have to have some discussion around how you solder those things, as 0402 feels like my limit at the moment.  And I need to get a microscope!

These caps are so cheap that you might as well buy like a 1000 of them so that even if you only manage to solder each fifth cap with other four teleported into another dimension, you will still have plenty of them remaining after all is said and done. You will need about 30 to 40 caps for a single board. I've bought a full reel of 15k parts a year or so ago, so I don't think I will run out of them any time soon
Now, with the new via-in-a-pad tech available at JLCPCB we can try using 0402s with a plated over pads instead of 0201s, but I've never actually used that tech before so it's a terra incognita for me and I'm not sure if it's going to work or not. But if you want to, you are free to give it a try to see if it's going to work or not. I mean, it should work, but I just don't have personal experience with that.

[nockieboy]

How are you going to test a module without a carrier if it is to be powered by the carrier?

That's the point of the H3 header.  You can supply 5V to the core board directly, without a carrier board.  Trying to be flexible.

Also having JTAG both on a module and on a carrier is not a great idea because of stubs which may negatively affect performance of JTAG.

Ah okay.  Well, blame the Chinese dev board schematics for that one.  Seemed like a good idea to have both - the core board is going to need a JTAG interface for testing.

You also forgot to mention QSPI flash for a bitstream. For A100T you will need at least 64 Mb, preferably more to leave some space for application data and code storage.

Yes, I did - I forgot a couple of things actually which you've picked up on.  The QSPI chip I've chosen is a W25Q128JVPIQ, its a 128Mb part for plenty of storage, as I think I have a few already anyway.

You need to be more specific if you only want 1080p@60 or "better", because the former can be implemented using regular IO pins, while for the latter we will need to use GTPs and some additional level shifting circuitry. I would recommend sticking to the former because of simplicity - unless you actually want that "better"

1080p@60Hz it is then.

Audio output via headphone/line out.

Ummm, these are analog signals, therefore you will need some sort of audio codec or DAC (if only output is desired). Do you have any specific one in mind, or it doesn't matter?

I was planning on something like a PCM5101A, I guess - again, unless there's better suggestions.

USB-A host PHY (CH559 or something similar - open to suggestions - serial/SPI/I2C interface to FPGA) for keyboard/mouse.

I'm not familiar with that device, traditionally USB ULPI PHY devices are used for USB 2.0, something like this one: https://www.microchip.com/en-us/product/USB3300

The CH559 is an odd device, not something I particularly like (there's very little documentation and it's all in Chinese or very poor English), but it's a means to an end.   I need a way of being able to connect a USB keyboard and/or mouse to the carrier board and have keycodes or mouse coordinates sent to the FPGA (and forwarded on to the host) as bytes of data.  The CH559 does this in one chip - it has an E8051 MCU in it which handles the USB stack and can be programmed in C.  I've got one sending out keycode/mouse data via serial, but haven't had the chance to work on it much more.

USB FPGA programmer.

This of course it up to you, but I'm not really convinced that placing programming circuitry on every board is such a great idea, as opposed to just buying a single programmer and connecting it to all FPGA boards as neccessary. Also see my note above regarding having JTAGs both on a module and a carrier.

I think it'll make my life easier in my particular case - there's a lot of reprogramming going on with the GPU still being developed.  I see the core/carrier as more of a development board anyway, so having a programmer built-in would be beneficial instead of having to plug the programmer in all the time.  Plus, I imagine people would want to design their own carriers anyway so it would be up to them if they're happy with on- or off-board programming.

The JTAG stub issue is something I was unaware of though, and I don't know how much of an issue that is.

What about Ethernet? Wouldn't you want your computer to be able to talk to outside world (or maybe just to other devices on your local network)?

Not for my uCOM, no.  That's not to say that I'm against designing for Ethernet on the carrier board, I don't have to populate those parts I guess and it could be useful if I progress to networking later with a more powerful host computer.

What about some sort of USB-UART port to talk to PC for debugging purposes?
What about some kind of storage? SD card interface, or eMMC device, or perhaps something else?

Yes, USB-UART and SD will be needed on the carrier too.  I forgot about them.

As for other stuff, typically to maximize utility of a module, all IO pins of a bank are routed length-matched such that high-speed peripherals can be connected if desired. And for example for 1G Ethernet PHY you will certainly need that kind of routing. I'm not really sure we will be able to length-match entire banks with 4 routing layers, but we need to at least attempt to do so.

I might be coming at the project from the wrong angle then.  In my mind, I'm designing the core board first, then worrying about the carrier once the core board is finalised.  Perhaps that's wrong, but I figure so long as we best-effort length-match the IOs out to the mezzanine connectors, we're doing the best we can?  Presumably we need to route some IOs as differential pairs for the HDMI as well.

These caps are so cheap that you might as well buy like a 1000 of them so that even if you only manage to solder each fifth cap with other four teleported into another dimension, you will still have plenty of them remaining after all is said and done. You will need about 30 to 40 caps for a single board. I've bought a full reel of 15k parts a year or so ago, so I don't think I will run out of them any time soon
Now, with the new via-in-a-pad tech available at JLCPCB we can try using 0402s with a plated over pads instead of 0201s, but I've never actually used that tech before so it's a terra incognita for me and I'm not sure if it's going to work or not. But if you want to, you are free to give it a try to see if it's going to work or not. I mean, it should work, but I just don't have personal experience with that.

I'm more than willing to give the via-in-pad tech a try, especially if that means I can use 0402s instead of nanoparticles.

[asmi]

Ah okay.  Well, blame the Chinese dev board schematics for that one.  Seemed like a good idea to have both - the core board is going to need a JTAG interface for testing.

It's not that it's not going to work at all, but you might be forced to lower JTAG frequency, which will hinder performance for both programming and if you are going to use any debugging cores.

Yes, I did - I forgot a couple of things actually which you've picked up on.  The QSPI chip I've chosen is a W25Q128JVPIQ, its a 128Mb part for plenty of storage, as I think I have a few already anyway.

It looks like that part is not listed as supported for programming via Vivado: https://docs.xilinx.com/r/en-US/ug908-vivado-programming-debugging/Artix-7-Configuration-Memory-Devices

1080p@60Hz it is then.

Ok, we will need a TPD12S521 as ESD protection and level shifter on the carrier board, but that should be it.

I was planning on something like a PCM5101A, I guess - again, unless there's better suggestions.

That's just a DAC, are you sure you don't want a full codec, with both inputs and outputs?

The CH559 is an odd device, not something I particularly like (there's very little documentation and it's all in Chinese or very poor English), but it's a means to an end.   I need a way of being able to connect a USB keyboard and/or mouse to the carrier board and have keycodes or mouse coordinates sent to the FPGA (and forwarded on to the host) as bytes of data.  The CH559 does this in one chip - it has an E8051 MCU in it which handles the USB stack and can be programmed in C.  I've got one sending out keycode/mouse data via serial, but haven't had the chance to work on it much more.

Well that is not really a USB interface then. I was thinking about a full-blown USB 2.0 implementation, which is where ULPI PHY are typically used. What kind of connectivity does it require on FPGA side?

I think it'll make my life easier in my particular case - there's a lot of reprogramming going on with the GPU still being developed.  I see the core/carrier as more of a development board anyway, so having a programmer built-in would be beneficial instead of having to plug the programmer in all the time.  Plus, I imagine people would want to design their own carriers anyway so it would be up to them if they're happy with on- or off-board programming.

It's both, sometimes features of a carrier have implications for the module design. Which is why I want to have a list of stuff on a carrier so that we can confirm that at least what you are going to have on your carrier is going to work.

The JTAG stub issue is something I was unaware of though, and I don't know how much of an issue that is.

See above. Xilinx recommends against messing with JTAG routing, besides if you are going to have a JTAG port on a module, I don't really see a point of adding another one on a carrier, because you can always use the one on a module even if it's connected to a carrier. Maybe I don't understand something here.

Not for my uCOM, no.  That's not to say that I'm against designing for Ethernet on the carrier board, I don't have to populate those parts I guess and it could be useful if I progress to networking later with a more powerful host computer.

Ethernet is very popular on FPGA boards because it's fairly easy to get it going for limited connectivity with outside world, and since you can run Linux on a softcore inside FPGA, you can get a full TCP/TP implementation essentially for free. And TCP/TP stack (and USB host stack) is one of the very few good reasons to run Linux inside FPGA. Of course you don't need any of that right now, but what about the future? There is a reason computer networks appeared almost at the same time as computers.

I might be coming at the project from the wrong angle then.  In my mind, I'm designing the core board first, then worrying about the carrier once the core board is finalised.  Perhaps that's wrong, but I figure so long as we best-effort length-match the IOs out to the mezzanine connectors, we're doing the best we can?  Presumably we need to route some IOs as differential pairs for the HDMI as well.

I'm looking at these connectors: https://www.samtec.com/products/bse for the module and https://www.samtec.com/products/bte for the carrier. They are available in a bunch pin count options, and Samtec offers samples for free so you might get them without paying a dime. I did get some connector samples from them in the past, though I did later buy some commercially for my customers' boards. These connectors have quite low crosstalk between adjacent pins, and with a low mating height they can go as high as 16 Gbps single ended/10 Gbps differential, which is plenty for both regular IO lines and GTP transceiver lines.
Speaking of GTP lines, since you are not going to use them immediately, I'd suggest to route them to a pair DisplayPort connectors (one for transmit, another one for receive) as they require very few external components (basically a bunch of AC-coupling caps and some ESD protection), and you can use them to connect to other things via that connector, to itself to test loopback scenario, or even to another copy of the same board for some high-speed board-to-board interface.

I'm more than willing to give the via-in-pad tech a try, especially if that means I can use 0402s instead of nanoparticles.

Ok, it's your project, so decisions are yours to make. But so are the risks that something goes south

[BrianHG]

1080p@60Hz it is then.

Ok, we will need a TPD12S521 as ESD protection and level shifter on the carrier board, but that should be it.

The NXP IC, PTN3366BSMP is a superior selection.  ESD and multi-level 'amplified' buffered level shifter with 3GHz EQ to help 4K30Hz support in one 90cent IC.  You do not need to worry about the VCCIO on the bank you are using for the HDMI. 

The IC will translate any received differential signal from 0.7v to 3.3v into a HDMI compliant open-drain current-steering differential output signals.

It also has active DDC level shifting.

[asmi]

The NXP IC, PTN3366BSMP is a superior selection.  ESD and multi-level 'amplified' buffered level shifter with 3GHz EQ to help 4K30Hz support in one 90cent IC.  You do not need to worry about the VCCIO on the bank you are using for the HDMI. 

The IC will translate any received differential signal from 0.7v to 3.3v into a HDMI compliant open-drain current-steering differential output signals.

It also has active DDC level shifting.

It's a very poor choice for Artix because LVDS requires Vccio of 2.5 V, which makes the rest of the bank IOs pretty useless as most other standards tend to use either 3.3 V or 1.8 V. The only differential standard supported at Vccio of 3.3 V is TMDS, which is what HDMI uses. That is why most HDMI implementations just use FPGA pins directly without any level shifting or other shenannigans.

[BrianHG]

The NXP IC, PTN3366BSMP is a superior selection.  ESD and multi-level 'amplified' buffered level shifter with 3GHz EQ to help 4K30Hz support in one 90cent IC.  You do not need to worry about the VCCIO on the bank you are using for the HDMI. 

The IC will translate any received differential signal from 0.7v to 3.3v into a HDMI compliant open-drain current-steering differential output signals.

It also has active DDC level shifting.

It's a very poor choice for Artix because LVDS requires Vccio of 2.5 V, which makes the rest of the bank IOs pretty useless as most other standards tend to use either 3.3 V or 1.8 V. The only differential standard supported at Vccio of 3.3 V is TMDS, which is what HDMI uses. That is why most HDMI implementations just use FPGA pins directly without any level shifting or other shenannigans.

The IC is an analog comparator amplifier.  It only requires a 3.3v supply.  Any signal you feed the differential inputs from 0.100v through 3.3v will be converted to HDMI levels.  Any IO bank on any IO voltage will work.  You can use the DDR3 IO banks, a 2.5v IO bank, a 3.3v IO bank, a 1.2v IO bank, a 0.7v IO bank and they will all work the same.  It's inputs are cap (on PCB) AC internally terminated and biased.

[asmi]

The IC is an analog comparator amplifier.  It only requires a 3.3v supply.  Any signal you feed the differential inputs from 0.100v through 3.3v will be converted to HDMI levels.  Any IO bank on any IO voltage will work.  You can use the DDR3 IO banks, a 2.5v IO bank, a 3.3v IO bank, a 1.2v IO bank, a 0.7v IO bank and they will all work the same.  It's inputs are cap (on PCB) AC internally terminated and biased.

What's the point in doing a conversion if you can do it directly instead? Simplicity is a virtue in my book. Especially since I know for fact that direct output works, while I've never seen that device used on any 7 series board I came across.

[nockieboy]

Yes, I did - I forgot a couple of things actually which you've picked up on.  The QSPI chip I've chosen is a W25Q128JVPIQ, its a 128Mb part for plenty of storage, as I think I have a few already anyway.

It looks like that part is not listed as supported for programming via Vivado: https://docs.xilinx.com/r/en-US/ug908-vivado-programming-debugging/Artix-7-Configuration-Memory-Devices

Good spot.  Okay, I've changed the part to this one:  S25FL128LAGNFI013.  Hopefully that will be okay?  What do you use normally?

I was planning on something like a PCM5101A, I guess - again, unless there's better suggestions.

That's just a DAC, are you sure you don't want a full codec, with both inputs and outputs?

Okay, maybe a TLV320AIC3100IRHBR would be a better choice?  I don't know anything about these devices unfortunately, other than the DECA board uses one of these which is quite expensive for something I don't really need (I'll be running audio through the HDMI).  However, all these chips seem to require is an SPI or I2S bus, so as long as we ensure there's provision for at least one in the core board connections, we should be good.

The CH559 is an odd device, not something I particularly like (there's very little documentation and it's all in Chinese or very poor English), but it's a means to an end.   I need a way of being able to connect a USB keyboard and/or mouse to the carrier board and have keycodes or mouse coordinates sent to the FPGA (and forwarded on to the host) as bytes of data.  The CH559 does this in one chip - it has an E8051 MCU in it which handles the USB stack and can be programmed in C.  I've got one sending out keycode/mouse data via serial, but haven't had the chance to work on it much more.

Well that is not really a USB interface then. I was thinking about a full-blown USB 2.0 implementation, which is where ULPI PHY are typically used. What kind of connectivity does it require on FPGA side?

Serial definitely, or possibly an SPI connection (if I can set aside the time to set one up on the device and test it).  A USB3300 PHY would be easy enough to include I guess, we just need to ensure we've got 12 I/Os we can use for a ULPI interface.

See above. Xilinx recommends against messing with JTAG routing, besides if you are going to have a JTAG port on a module, I don't really see a point of adding another one on a carrier, because you can always use the one on a module even if it's connected to a carrier. Maybe I don't understand something here.

Eh.  I guess it'll reduce the BOM cost as well.  So we'll just stick with a JTAG port on the core card.

Ethernet is very popular on FPGA boards because it's fairly easy to get it going for limited connectivity with outside world, and since you can run Linux on a softcore inside FPGA, you can get a full TCP/TP implementation essentially for free. And TCP/TP stack (and USB host stack) is one of the very few good reasons to run Linux inside FPGA. Of course you don't need any of that right now, but what about the future? There is a reason computer networks appeared almost at the same time as computers.

Yeah, that's a fair point.  I know nothing about Ethernet really, so I'll be heavily guided by what I can find in existing dev board schematics.  If you have any experience with a particular Ethernet PHY, let me know which device you'd prefer.

I'm looking at these connectors: https://www.samtec.com/products/bse for the module and https://www.samtec.com/products/bte for the carrier. They are available in a bunch pin count options, and Samtec offers samples for free so you might get them without paying a dime. I did get some connector samples from them in the past, though I did later buy some commercially for my customers' boards. These connectors have quite low crosstalk between adjacent pins, and with a low mating height they can go as high as 16 Gbps single ended/10 Gbps differential, which is plenty for both regular IO lines and GTP transceiver lines.

Those connectors look just fine to my inexperienced eye, but they're not cheap (around £20 for a pair of 80-pin connectors).  How many are we considering for a core board?  I'm thinking 4 x 80-pin connections, but that's a ball-park top-of-my-head estimate.

How do you go about getting free samples?  Are they okay giving out freebies to hobbyists who are unlikely to make a purchase?

Speaking of GTP lines, since you are not going to use them immediately, I'd suggest to route them to a pair DisplayPort connectors (one for transmit, another one for receive) as they require very few external components (basically a bunch of AC-coupling caps and some ESD protection), and you can use them to connect to other things via that connector, to itself to test loopback scenario, or even to another copy of the same board for some high-speed board-to-board interface.

Okay, so two DisplayPort connectors for the GTP lines too.

1080p@60Hz it is then.

Ok, we will need a TPD12S521 as ESD protection and level shifter on the carrier board, but that should be it.

The IC is an analog comparator amplifier.  It only requires a 3.3v supply.  Any signal you feed the differential inputs from 0.100v through 3.3v will be converted to HDMI levels.  Any IO bank on any IO voltage will work.  You can use the DDR3 IO banks, a 2.5v IO bank, a 3.3v IO bank, a 1.2v IO bank, a 0.7v IO bank and they will all work the same.  It's inputs are cap (on PCB) AC internally terminated and biased.

What's the point in doing a conversion if you can do it directly instead? Simplicity is a virtue in my book. Especially since I know for fact that direct output works, while I've never seen that device used on any 7 series board I came across.

I've been using the PTN3366 on the previous GPU card with the Cyclone IV and I can see why BrianHG is recommending its use, but if you're sure we can directly drive HDMI from the Artix-7 then reducing the part count and simplifying the layout has to be a positive?

[asmi]

Good spot.  Okay, I've changed the part to this one:  S25FL128LAGNFI013.  Hopefully that will be okay?  What do you use normally?

I've bought a bunch of S25FL256LAGBHI030 and S25FS128SAGBHV200's for cheap few years back, so that's what I use. They are in 1 mm pitch BGA package (which was a reason I picked them - along with the price) because there are footprint-compatible options all the way up to 1Gb, so I figured it will be convenient for me in case I ever want to have more storage. But any part listed in the list I linked should be fine. Just watch for the voltage level - some of parts listed there are 1.8 V, which would require powering bank 14 with that voltage, with all implications for other pins in the same bank.

Okay, maybe a TLV320AIC3100IRHBR would be a better choice?  I don't know anything about these devices unfortunately, other than the DECA board uses one of these which is quite expensive for something I don't really need (I'll be running audio through the HDMI).  However, all these chips seem to require is an SPI or I2S bus, so as long as we ensure there's provision for at least one in the core board connections, we should be good.

Unfortunately here you are going to lose me, as I'm a digital guy and not very good with analog stuff. That said, I2S is a fairly standard bus (I think Xilinx even provides free IP cores for I2S transmitter and I2S receiver), so any one you can get your hands on should be good to go, provided it's a got a half-decent datasheet/appnote showing how analog part is supposed to be designed which you could reproduce on your board.

Serial definitely, or possibly an SPI connection (if I can set aside the time to set one up on the device and test it).  A USB3300 PHY would be easy enough to include I guess, we just need to ensure we've got 12 I/Os we can use for a ULPI interface.

There is no reason not to include both There is a plenty of free IO pins in this FPGA.

Eh.  I guess it'll reduce the BOM cost as well.  So we'll just stick with a JTAG port on the core card.

I actually bought a tag-connect pogo-pin connector recently, and I want to give it a try along with my Digilent HS3 programming cable to save some more even on a connector (it only requires a footprint). BTW if you want to save a few bucks, you can buy a Xilinx programmer on Aliexpress for a few bucks and they seem to work fairly OK. I prefer using geniune Digilent programming cable, it's about $55 so not a big deal considering it's a one-time investment.

Yeah, that's a fair point.  I know nothing about Ethernet really, so I'll be heavily guided by what I can find in existing dev board schematics.  If you have any experience with a particular Ethernet PHY, let me know which device you'd prefer.

There is a bit of a shortage of PHY devices right now, so pretty much any device that can talk RGMII and you can get your hands on should be good. There are some gotchas with older RGMII v1.x devices which required adding a clock delay loop on a PCB, but with RGMII v2.0 there is now an option to add that delay internally, so we will need to check a datasheet for whichever device you end up choosing if it requires PCB clock delay or not. I've managed to snag a few of 88E1510-A0-NNB2C000's a couple of months ago, these are fairly expensive (though still in stock at Mouser right now), so if you find something at a more reasonable price, it should still be good.

Those connectors look just fine to my inexperienced eye, but they're not cheap (around £20 for a pair of 80-pin connectors).  How many are we considering for a core board?  I'm thinking 4 x 80-pin connections, but that's a ball-park top-of-my-head estimate.

I think a pair of 120 pin connectors (240 pins total) should be plenty for our needs. Each connector is 53 mm long, so something like 5 x 6 cm PCB should be good. I looked up parts, and it looks like those two are in stock and in reserve (so they usually ship quickly): https://www.samtec.com/products/bse-060-01-f-d-a-tr $7 for qty 1, $6.47 for qty 10, mating part is this one: https://www.samtec.com/products/bte-060-01-f-d-a-k-tr $7.3 for qty 1, $6.75 for qty 10

How do you go about getting free samples?  Are they okay giving out freebies to hobbyists who are unlikely to make a purchase?

It never hurts to ask, they sent me a whole bunch of samples over the years, I usually asked for 10 mating pairs, and they never denied, even though I never actually ordered anything commercially from them directly (though I had bought their parts via Digikey and Mouser after I used free samples for prototypes and confirmed their suitability, so I guess their goal of converting me into a paying customer has been achieved).
Besides, you never know - maybe you will end up selling these modules in the future? If they are generic enough (and I'm trying to "steer" design such that they would be) and are actually available, you might get quite a few customers

I've been using the PTN3366 on the previous GPU card with the Cyclone IV and I can see why BrianHG is recommending its use, but if you're sure we can directly drive HDMI from the Artix-7 then reducing the part count and simplifying the layout has to be a positive?

I remember that this solution has been devised because Cyclone-4/5 and MAX10 don't support TMDS standard directly. And since Artix-7 do support that standard directly, it seems like an unneccessary complication to me. Also there are plenty of Artix-7 boards which do implement it directly, so it's not just my personal experience with doing so. I personally had no problems with such implementation driving 3 meters long HDMI cable to a 1080p monitor, so I figured it's robust enough for most practical cases.
-----
BTW Artix-7 devices I've ordered have been shipped this morning, DHL says they should arrive by this Friday.

We will also need a bunch of clock chips and some crystals, but we will see into it once we have better idea of the whole system. At the very least we will need a main system clock on a module, and a 135 MHz LVDS clock for GTP/DisplayPort, and some crystals for Ethernet PHY (typically 25 MHz) and a USB 2.0 ULPI PHY (typically 24 MHz) - all of those are on a carrier.