Planning/design/review for a 6-layer Xilinx Artix-7 board for DIY computer.

[asmi]

Something else just sprang to mind for a peripheral - as well as the 10/100/1000 Ethernet interface, what about wifi?  What would be the best way to implement that?  I guess a wifi module, like the ESP32, would work but is there a way to do it more simply, without the middle-man microcontroller?

ESP32 and other MCU-based solutions are very slow and are only good for IoT devices which need to only send a small packets of data every once in a while. Modern laptops typically use mini-PCIE cards (like Intel ax210), or you can buy a "normal" PCIE WiFi card and plug it into the PCIE slot.

[nockieboy]

Just realised, I'm still going to need a 5V rail for the HDMI.  Won't need to be as beefy as what I had originally planned for, though!

ESP32 and other MCU-based solutions are very slow and are only good for IoT devices which need to only send a small packets of data every once in a while. Modern laptops typically use mini-PCIE cards (like Intel ax210), or you can buy a "normal" PCIE WiFi card and plug it into the PCIE slot.

That's a good point, although I'd thought that the PCIE slot could be used, it'd be a shame to block it with a wifi card if I can build wifi into the board anyway.  Presumably it's not that straightforward, though.

[asmi]

Just realised, I'm still going to need a 5V rail for the HDMI.  Won't need to be as beefy as what I had originally planned for, though!

HDMI only requires to supply 50 mA of current, so you can simply use any LDO which can work off 12V rail.

That's a good point, although I'd thought that the PCIE slot could be used, it'd be a shame to block it with a wifi card if I can build wifi into the board anyway.  Presumably it's not that straightforward, though.

In a different world I would've used a PCIE switch (for example, this one) and place both mini-PCIE and a regular PCIE ports such that you can connect both at the same time. But in this current reality these switches are both incredibly expensive and are super-hard to find in stock, so I think we will have to settle for having just a single PCIE port.

[miken]

Another option for the HDMI 5V is to use a PTN3381B, which contains a boost regulator.

[asmi]

Another option for the HDMI 5V is to use a PTN3381B, which contains a boost regulator.

We already have 12V, so what's the point to drop lower only to go back higher again?

[miken]

Another option for the HDMI 5V is to use a PTN3381B, which contains a boost regulator.

We already have 12V, so what's the point to drop lower only to go back higher again?

If you guys are planning to use a HDMI buffer then it'd be all-in-one. But if not, sure, there are other ways of going about it.

[BrianHG]

78L05

[nockieboy]

HDMI only requires to supply 50 mA of current, so you can simply use any LDO which can work off 12V rail.

78L05

Yeah, wasn't going to over-engineer that as I'd have little use for a 5V line elsewhere.  Size would be my priority in part-selection for an LDO here rather than current output.

In a different world I would've used a PCIE switch (for example, this one) and place both mini-PCIE and a regular PCIE ports such that you can connect both at the same time. But in this current reality these switches are both incredibly expensive and are super-hard to find in stock, so I think we will have to settle for having just a single PCIE port.

  The price of a lot of stuff has gone through the roof whilst its availability has gone through the floor.  Good old supply 'n' demand I guess.  Damn annoying though, especially with FPGA's not being exactly cheap to begin with.  That was an amazing find, asmi, when we got those 100T's for that price.  I wish I'd re-mortgaged the wife and got a few more!

I had a look for PCIE switches - I'm probably looking at the wrong ones, admittedly, but they don't seem to be too expensive - a couple of bucks each?  If I've found the right chip, it doesn't sound like it would break the bank if it gives me another PCIE socket on the board?

I'm looking at this: PI3PCIE3412AZHEX - £1.87 per IC.  Am I missing something or misunderstanding the issue?

If you guys are planning to use a HDMI buffer then it'd be all-in-one. But if not, sure, there are other ways of going about it.

I've got a TPD12S521 on the BOM currently, although I haven't looked at the HDMI schematic yet.  I need to rip out the PTN3366 that I used for the Cyclone board's HDMI output as the Xilinx can run HDMI directly (though I'm putting some protection in place with the TPD12S521).

I've just taken another look at Mouser (I was unaware that there would be HDMI interface chips that supply the 5V for you!) and found this one - the TPD12S016.  Looks like it does the ESD suppression of the one above and also produces a 5V rail current-limited to 50mA.  Seems like a more suitable solution to a straight LDO?  And it's a single part replacement as opposed to having to add an LDO to existing parts.  Anyone have any thoughts on this?

[asmi]

Yeah, wasn't going to over-engineer that as I'd have little use for a 5V line elsewhere.  Size would be my priority in part-selection for an LDO here rather than current output.

I used AZ1117-5.0 fixed voltage version in my Sparta board, and it seemed to work OK. However for this design I'm thinking about turning board on and off using "enable" pins of all regulators, so the LDO will also need to have an enable pin. Still haven't decided on a part yet, will do some searching on a Mouser once I have some time.

I had a look for PCIE switches - I'm probably looking at the wrong ones, admittedly, but they don't seem to be too expensive - a couple of bucks each?  If I've found the right chip, it doesn't sound like it would break the bank if it gives me another PCIE socket on the board?

I'm looking at this: PI3PCIE3412AZHEX - £1.87 per IC.  Am I missing something or misunderstanding the issue?

What you are looking at is an analog switch, meaning it only physically switches lanes. What I'm talking about is a PCIE protocol switch, which works like a network switch and routes PCIE packets between it's channels, allowing to use both devices.

I've got a TPD12S521 on the BOM currently, although I haven't looked at the HDMI schematic yet.  I need to rip out the PTN3366 that I used for the Cyclone board's HDMI output as the Xilinx can run HDMI directly (though I'm putting some protection in place with the TPD12S521).

I've just taken another look at Mouser (I was unaware that there would be HDMI interface chips that supply the 5V for you!) and found this one - the TPD12S016.  Looks like it does the ESD suppression of the one above and also produces a 5V rail current-limited to 50mA.  Seems like a more suitable solution to a straight LDO?  And it's a single part replacement as opposed to having to add an LDO to existing parts.  Anyone have any thoughts on this?

TPD12S016 still requires LDO, it only does current limiting. TPD12S521 also contains a current limiter, please see the last page of my schematics for reference: https://github.com/asmi84/kicad-projects/blob/master/S7_Min/S7_Min.pdf

[dolbeau]

I've just taken another look at Mouser (I was unaware that there would be HDMI interface chips that supply the 5V for you!) and found this one - the TPD12S016 (...) Anyone have any thoughts on this?

That's what I use in the NuBusFPGA, 'inspired' by the QMTech Wukong FPGA board. Routing is not too difficult despite the 'single-pad' design (the 4 differential pairs only hit one set of pads, so those have to be connected both to the FPGA and the connector). Works like a charm, both in the Wukong (using Litex's standard HDMI stuff for 1920x1080@60Hz, 32 bits) from a xc7a100tfgg676 -1 and in the NuBusFPGA (ditto, also tested with another HDMI PHY to get audio support) from a xc7a35tcsg324-1.

It does have a regulated 5V output for HDMI, but it needs a 5V input as well (in addition to the 3V3 core voltage).

[nockieboy]

Okay, it's been a while due to work and life, but here's a quick update.  Features and schematic are almost pinned down now, I just need a little feedback on the USB hub that I've included below.  I'm unsure about USB_HUB_5V and USB_HUB_ID signals from the USB PHY - I think I'll just strap USB_HUB_5V to 5V and leave USB_HUB_ID floating, but if anyone has any thoughts or can spot any errors, let me know!

The idea is to provide four USB 2.0 ports to the FPGA, so a soft-core CPU running a capable operating system can access a USB keyboard, mouse, storage etc.  A more direct method of connecting a USB keyboard to a non-USB-capable operating system (my 8-bit Z80 computer, for example) will be provided via a peripheral board (connected via a PMOD), which I might design later if I ever get round to it.

Schematic for the USB hub is attached.

I haven't had time to mess with Vivado much yet or start working on the AXI/BrianHG_memory_controller interface.

[dolbeau]

The idea is to provide four USB 2.0 ports to the FPGA, so a soft-core CPU running a capable operating system can access a USB keyboard, mouse, storage etc.  A more direct method of connecting a USB keyboard to a non-USB-capable operating system (my 8-bit Z80 computer, for example) will be provided via a peripheral board (connected via a PMOD), which I might design later if I ever get round to it.

Do you need the 480mbps of USB 2.0?

For keyboard/mouse and (slow...) storage, USB 1.1 could be enough. There's (at least) a OHCI-compliant soft-host available, which can do 1 to 4 ports with just the ESD/TVS stuff externally, plus the VBus regulation if you don't want to force the use of a self-powered USB hub. You only need a single differential pair of I/O per port, and it will work with any OS that has an OHCI driver. I've used it for keyboard/mouse/basic storage under NetBSD/sparc on a '94 SPARCstation 20.

[asmi]

Okay, it's been a while due to work and life, but here's a quick update.  Features and schematic are almost pinned down now, I just need a little feedback on the USB hub that I've included below.  I'm unsure about USB_HUB_5V and USB_HUB_ID signals from the USB PHY - I think I'll just strap USB_HUB_5V to 5V and leave USB_HUB_ID floating, but if anyone has any thoughts or can spot any errors, let me know!

ID pin of the PHY is only required for USB OTG support, you can leave it unconnected since we only want to support USB HOST configuration. Also check the "Hardware Checklist" documents for your USB hub and USB PHY parts on Microchip website - they have a lot of useful information for applications.

[asmi]

Well, it took quite a bit longer than I anticipated due to me not having much of a spare time, but I've completed a board design I envisioned. It turned out to be quite a monster - 10 layers (as opposed to 6), via in a pad, and 150 x 120 mm in size, it's got onboard USB-JTAG as well as JTAG TagConnect footprint, also 1G Ethernet and an HDMI out, everything else is routed out to 3 high speed connectors - one with 4 MGTs and some GPIO, another one with almost a full bank of GPIO, and a third one with a full IO bank (50 balls) routed as length-matched differential pairs (and 2 single ended signals). 3D renders and schematics (updated schematics are posted in a later post) are in attachment for those curious what it looks like. I'm still doing some final checks in an attempt to prevent "a curse of the first revision" to strike me again (which usually are futile) before sending design out to manufacturing, so I'm hopeful, but we'll see.
Some stats:
Board size: 150 x 120 mm
Components on board: 557
Layer count: 10
Smallest drill size: 0.2 mm
Smallest trace width/spacing: 0.1/0.1 mm
Total number of connections: 1871
Total number of non-plated holes: 57
Total number of plated holes: 2180

[asmi]

Good thing I didn't rush it to manufacturing Found and fixed a bunch of things. If anyone has some spare time and inclination, please review updated schematics (in attachment) to see if something jumps out on you. UPDATE - see later post in this thread for latest schematics.

[nctnico]

Why the choice for a Marvell PHY with MII interface? Documentation is hard to get for Marvell chips. I'd use a phy from TI for robustness and good documentation. I'm also missing TVS diodes on the ethernet signals.

[asmi]

Why the choice for a Marvell PHY with MII interface? Documentation is hard to get for Marvell chips. I'd use a phy from TI for robustness and good documentation. I'm also missing TVS diodes on the ethernet signals.

This was one of chips I've managed to stock up on during chipageddon, so I'm using what I have on hand. That said, this specific device has a very good public datasheet, the only thing I don't like about it is that it only works at Vccio of 2.5/3.3 V, while I would rather prefer it to work across 1.8/2.5/3.3 V like many other 1G Ethernet PHYs do.

[nctnico]

Fair enough. IIRC I have come across this phy in one of my customers' designs and I did ran into needing info that was not in the datasheet. But it could be a different Marvell phy though.

Also not sure whether it is a good idea to route the ethernet signals on the top layer if the shield of the ethernet connector isn't well defined. If it reaches to the board, it could short the traces.

As a general rule, I like to place al capacitors with a polarity in the same direction. Makes it easy to spot mistakes and thus increase production reliability. Especially for tantalum this is important. I had one that was mounted in reverse explode in my face at some point.

[asmi]

Fair enough. IIRC I have come across this phy in one of my customers' designs and I did ran into needing info that was not in the datasheet. But it could be a different Marvell phy though.

Yeah, they don't have public datasheet for some of their devices, but this isn't one of them. The datasheet actually covers 4 different devices 88E1510/88E1518/88E1512/88E1514, which are quite similar but offering different features (like some of them provide MDI only to copper, others also to optic).

Also not sure whether it is a good idea to route the ethernet signals on the top layer if the shield of the ethernet connector isn't well defined. If it reaches to the board, it could short the traces.

That's a good point, though in my case the part of connector where pins come out is made of plastic, so I should be safe.

As a general rule, I like to place al capacitors with a polarity in the same direction. Makes it easy to spot mistakes and thus increase production reliability. Especially for tantalum this is important. I had one that was mounted in reverse explode in my face at some point.

It's not always possible to place all parts the way you want and keep layout somewhat compact. And since space on multilayer boards is expensive and forces me to prioritize compactness, I've learnt to check and double- and triple check everything before AND after reflow before powering up.

Thanks for taking some time to review the schematics!

[Gribo]

The capacitors for the ethernet XTAL are a bit off. 20pF CL crystal requires ~27pF for each the capacitors - they are in series and 0402 has ~1pF parasitic capacitance per pad.

[asmi]

The capacitors for the ethernet XTAL are a bit off. 20pF CL crystal requires ~27pF for each the capacitors - they are in series and 0402 has ~1pF parasitic capacitance per pad.

Thank you, I've updated schematics (also changed load caps for FT2232 as well). There were also some other changes as well.

[Gribo]

Can you post new pictures? You have changed some reference designators and it is a bit confusing (J9). I would make all the expansion connectors safely interchangeable, that is, if something is connected to one, it can be safely connected to the others. It might not work, but it shouldn't short out or cause damage.

[asmi]

Can you post new pictures? You have changed some reference designators and it is a bit confusing (J9).

Attached. I've added some more connectors and jumper-selectors (for example for M[2] ball so that FPGA can be set up to boot from QSPI, or not boot at all expecting to be configured via JTAG). Also I forgot to add package pin delays during initial layout, now I've added it and fixed delay matching so that those delays are taken into account.

I would make all the expansion connectors safely interchangeable, that is, if something is connected to one, it can be safely connected to the others. It might not work, but it shouldn't short out or cause damage.

I use the same connector for MGT expansion connector and for a one with differential pairs, pinout is designed to be "compatible" in a sense that nothing should blow up if you connect it to the wrong position - power and ground pins are in the same positions (or not connect), but of course depending on what VADJ1/VADJ2 are set to, circuitly on a daughterboard might not like it if it's set to anything other than what it expects, so I do expect some level of diligence on the part of a user.
Third expansion port (Port 2) is using a different connector, so it's not possible to connect it anywhere else.

[asmi]

I've finally managed to get enough spare time to assemble and thoroughly test the board, and as usual for me, I got quite a few things wrong So here it is, the good, the bad and the ugly:
1. I had the wrong pinout for LVDS clock generator. Managed to fix it by shifting the generator one pad to it's side and adding a couple of bodge wires. Will be fixed for rev B1.
2. I messes up the pin numbering for a voltage translator (numbering begins in the middle of one of sides, and not from one side as usual). As a result, this revision is unable to use QSPI flash memory at all. Thought about bodging, but decided that it's going to be too much trouble. Footprint will be fixed for B1, but I still can't know for sure if it's going to be functional. Will see.
3. Had lots of troubles getting MPM3632C's to work. Turned out to be bad soldering, once I added some solder to the sides, they started working. Very finicky stuff. I've extended the pads somewhat to allow for more solder, will give it another go in B1, if it will still prove to be troublesome, will consider replacing them with regular DC-DC converters, which worked perfectly right from the get-go.
4. I soldered the crystal for Ethernet PHY turned at 90° to what it should be (assembly error), once fixed, it started working with no issues. Confirmed full functionality via hardware evaluation version of Xilinx TriMAC IP (and some mild tweaking of the lwip code to get PHY configured properly). Also tried it with some homemade MAC, and it seems to work just fine.
5. DDR3 SODIMM worked right from the get go with 8GByte module from Micron. I want to buy some more DDR3 SODIMMs from random vendors just to see what it takes to get MIG to work with various off-the-shelf modules found for pennies in computer stores, as "brand" Micron stuff is very hard to find in stock anywhere. That was the biggest design risk, so I'm super happy that it worked right off the bat 
6. HDMI out gave me some grief, but the problem turned out to be a bad HDMI cable Once replaced, everything started working smoothly, but I still wasted almost entire day before I decided to test with a different cable. I also forgot to include pullup resistors on the FPGA side of things (after redriver IC) for DDC, and mixed up SDA/SCL lines, fixed that for B1.
7. I was unable to test fan as dummy me bought the wrong fan (5 V instead of 12 V) . But I can't really see how it can be screwed up.
8. I wired thermal alarm signals to be on-high while the actual signals are on-low So I have two more LEDs permanently on than I intended to. Fixed for B1.
9. This is just a nitpick, but my user switch goes to high when moved down, which bothers my OCD. Turned it around for B1.
10. All other user IOs (RGB LEDs, regular LEDs, pushbuttons) work like sharm. No real suprises here.
11. FT2232-based JTAG programmed also just worked right after programming. But I was fairly sure it will after I had prototyped this subcircuit on a dedicated PCB. One minor nitpick is that TX/RX directions in FT2232 datasheet are from the USB's point of view, so I had my silkscreen wrong. Fixed for B1.
12. I realized that a cooling fan would be really useful as FPGA die temp reached 75°C in some cases once you really load it up. I have an "I"-rated FPGA, so it's good to 100°C, but I don't like that it's so hot.

And two more things I learnt during assembling of this board: 1 - QFNs on the bottom side are just fine and they stay put after reflow of the top side I always suspected that would be the case, but it's good to have practical confirmation. and 2 - via in a pad technology rocks as it allowed me to avoid using 0201 caps and use 0402's instead, which helped massively during assembly.

Attached is a high-ish resolution photo of the top side as it looks right now, you can see some bodges, resistors attached to vias, and a lot of residue from soldering. I'm frankly afraid to subject the board to an ultrasonic wash as I'm afraid it's going to tear some of my bodges away.

I'm currently putting some finishing touches on a rev B1, as I made some more minor changes and additions, will post it here once it's ready.

[asmi]

Second revision is assembled and fully tested. There is a minor mistake in schematics (which appeared in this revision as it didn't exist in previous one), but nothing that can't be fixed with a couple of bodges. I've added a Microchip's EEPROM with pre-programmed MAC address (PN: 24AA025E48T-I/SN) to use when implementing Ethernet, since I didn't have any free IO pins, I've connected it to two lines which lead to SMA connectors with a pair of zero Ohm resistors so that they can be removed if neccessary, I've also connected SODIMM's SPD bus and thermal sensor's I2C bus to these lines as well (again with jumper resistors).

Attached are full schematics in case anyone is curious, as well as a photo of the assembled board. Unfortunately I couldn't find any heatsink with fan which is small enough for 23x23 mm package, so I had to improvise This heatsink is actually 27x27 mm, and I have an idea to buy smaller version of it and make a couple of holes with M3 thread, so that I can mount a fan that way. I will need to do some experiments later.

As you can see on the photo, regular COTS SODIMM module works just fine, so no need to hunt after rare Micron modules, which are the only ones supported officially, but since any SODIMM DDR3 is required to work at JEDEC timings, there shouldn't be any problems using any random module you happen to have on hand. I'm planning to buy a couple more of COTS SODIMMs in the near future to confirm this theory (since they are so cheap, it shouldn't really be a problem).