Nintendo new 3ds XL Capture Card: Making one from scratch

[akp_eev]

I want to stream the video (any maybe audio) from my New Nintendo 3DS XL console, to a PC. I want to try creating a board myself, as a fun learning experience.

I've looked into other solutions, but they're lacking in what I need. There are two other board manufacturers who make 3DS capture boards as well:
   
   * Optimize board, self-install (comes from Japan so extra shipping cost, also install is difficult)
   * Loopy, send my DS off to be modded (expensive, and need to wait for a limited opening in his schedule, not likely to get a spot)

So I've decided to try making my own board instead!

What have I done so far?

I've posted on the GBATemp forums, and I've been pouring over Nintendo DS specs. Here is what I currently understand about what I'll need for my board:

•    Overall: must take in video/audio data from the 3DS and transfer it via USB to a PC. Framerate is 60Hz.
     Top screen is 800x240 pixels, bottom screen is 320x240 pixels.
•    Must be able to take in roughly 40 input lines via ribbon cable, from test points on the console's board. This includes the following:
• 24 bit color, lower screen (24 lines)
• Dotclk + Hsync + Vsync, lower screen (3 lines)
• 4 video data bits + 2 clock lines, upper screen (6 lines)
• For audio, someone on the GBATemp forum told me:
           Audio is 32 + 47 kHz over 2 I2S interfaces. Most capture cards only bother capturing the 32 kHz one
   

•    Need a way to convert the data from parallel to serial. I apparently need either some very fast PISO shift registers, or an FPGA to convert this data.
•    Since the card needs to capture 2 video frames (top/bottom screen) and possibly audio, serial output via USB 3.0 seems like the way to go.
•    I'd like to use a single USB-C out, if possible (NOTE: I do NOT want to do HDMI out. I want to be able to process the video/audio data on PC).
•    I'd like the board to fit inside the console's casing as much as possible (the Optimize boards manage this, if you can cut some of the plastic inside the case to make room for the new board)
•    I'm a software engineer, so I shouldn't have trouble once I can get the raw data coming from the USB. Seems easy enough to use Windows HID api to handle it, or something similar. And I can whip up something to display the images.

   
Questions:

• How do I determine what my board actually needs? Clearly I need something to do the parallel to serial conversion, and somehow take that serial data to the USB out... but what else might I need? And how do I find out?
• It looks like a 1.8v supply is used for the ribbon cable used in the optimize boards, by soldering a wire from the ribbon cable to TP5 on the DS board. Say I used a board like the Alchitry Cu, as a contrived example (https://alchitry.com/boards/cu). Wouldn't I need to solder the ribbon cable to a 5v supply on the DS board instead? Is it possible to power the board via the same USB that carries the data out?
• How do I start looking for FPGAs (or a microcontroller) that fit my requirements?
• How would I BUILD the board? Am I likely to need a number of prototype boards that I use to iterate closer to my final goal? Or would I be able to print the correct board on my first try, if I know enough about how the data is coming out of the DS test points? Seems like the easiest way to approach this is to grab something mostly premade (like the Alchitry Cu, for example). But that's a pretty large board in terms of size, and one of my requirements is to try fitting the board inside the DS casing.

   
Useful links:

   Specs for the lower screen video capture
   Layout of pins/ test points
   Instructions for installing the N-SPA3 Optimize board (this is for a different 3DS model, but similar enough to be useful)
   Instructions (Japanese) for installing the New-SPA3 board (for New 3DS XL)
   Shop for Optimize Boards
   A gallery of images for Loopy's previous boards
   An image of the board Loopy is currently working on for New 3DS XL

Thanks,
-akp

[akp_eev]

Been looking through boards... here's a couple I found that might be up to the task?

Fire Ant
Tang Nano 1K

This would maybe be good if it had more input pins... as is, I think I need at least 30 so this is out

The LCD connector on the Tang Nano looks interesting, though I guess I wouldn't be able to use that since I'll need flex ribbon cable coming from different test points on the DS board. And I don't know that you can read input from that connector? Maybe you can only write to it?

[zzattack]

As a rule of thumb, doing high speed video stuff as a first digital design project is bound to disappoint.
When considering your wish list which includes USB3, multiple screens and audio interfaces, then the question in the same post about providing supply power to your part suggests you should preface your quest with some less ambitious projects first.

Unless the screen is driven using a known protocol, you will also have to reverse engineer that.

[Berni]

As others have said you might be biting off more than you can chew.

Handling live video data needs a lot of digital bandwith and so there is no room for thrown together solutions where software is making up for flaws in cheep crappy hardware.

Handling video from USB by yourself in software is also going to be an adventure when you want it to work smoothly and reliably. So you really should use an existing video capture solution that shows up on a PC as simply a standard video source. The easy way for this is to just use the simplest HDMI video capture device and then feed DVI data into it (simpler with less encoding while being still compatible with HDMI). Alternatively you can get a purpose built USB video interface chip that takes in RGB data and sends it over USB. Then again this is essentially the same kind of chip you would find in a USB HDMI video capture dongle.

This then leaves the FPGA to only be doing video conversion tasks. It needs to take the two screens and merge the video data together into a single RGB or DVI video signal that then can be fed into the video capture device. Doing this can already be quite a task since a lot of FPGAs might not have enough memory to hold a large enough framebuffer, so you might have to use external memory for that, or use a very very expensive big foot FPGA

[ebastler]

A bit more background about your electronics experience would be helpful: Have you worked with FPGAs and microcontrollers before (using off-the-shelf development boards)? Have you done smaller circuit design and PCB layout projects? Can you solder well, and are confortable with SMD parts? What equipment do you have, or have access to -- specifically, do you have an oscilloscope, and what are its specs?

As others have said before: Unless you answer is a resounding "yes" to all the above, you should probably think about some easier projects to get you started. What you are proposing is a rather challenging project which requires the use of somewhat advanced tools amd techniques: FPGAs, oscilloscope and/or logic analyzer, PCB layout, soldering small-pitch SMD parts, flex PCB design and soldering, ...

EDIT: This is definitely a job for an FPGA, not a microcontroller. Trying some simpler projects with a small FPGA development board is a good first step. The two boards you mention in your second post are compact and affordable, but both use FPGAs from "new kid on the block" manufacturers which I can't say anything about. Gowin has certainly been mentioned on the forum, so maybe others can comment.

You will find that the development software is what really defines whether an FPGA is useful to you: Is it available for free at all (for non-commercial use)? Is decent documentation available? Have third-party tutorials and sample projects been published on the web? You may fare better with the etablished FPGA brands: Altera/Intel, Xilinx/AMD, Lattice.

All these software packages are complex beasts, and getting the hang of VHDL or Verilog programming (which looks like C, but is not the same at all) is a challenge per se. Hence, settling on an FPGA family, getting a cheap development board and the manufacturers' software, and starting to play around with something simple are my recommended first steps.

[akp_eev]

First, thank you all for your responses, I greatly appreciate your help. It's a lot of fun to be talking on a real forum again, everything is on discord/reddit nowadays

Berni: Alternatively you can get a purpose built USB video interface chip that takes in RGB data and sends it over USB... This then leaves the FPGA to only be doing video conversion tasks. It needs to take the two screens and merge the video data together into a single RGB or DVI video signal that then can be fed into the video capture device

I'm confused here. The FPGA is ostensibly taking in a flex cable from the 3DS board. There's the bottom screen transmitting 24 bits in parallel (plus hsync and vsync), and the top screen transmitting in serial. The FPGA is merging everything into a single video signal, stored away in some large buffer. Then you'd have your ordered RGB data:

Code: [Select]

Both screens, multiplied by RGB (3 * ((800x240) + (320x240))) = 806400 bits per frame
If the FPGA has converted the incoming data into a 806400 bit-sized frame, is the "USB video interface chip" just... taking the data and shoving it out over USB? The FPGA can't do that task as well?

Berni: Handling video from USB by yourself in software is also going to be an adventure when you want it to work smoothly and reliably...

Can you elaborate on the "smoothly and reliably" part? I have a project or two that uses OpenGL to draw textures quickly, and I've played around with Windows HID api for game controller input. If I'm just getting raw RGB data from the USB, I could shove that into an OpenGL texture and render it in my application. Seems straightforward enough, unless I'm missing something.

ebastler: As others have said before: Unless your answer is a resounding "yes" to all the above...

NOPE! I have not, I'm new to most of these things. I prefer diving into the deep end because I get bored otherwise. But I understand this is going to be a pretty big undertaking, so I'll try fiddling around with a starter FPGA board before ripping apart the 3DS. I've also been looking for oscilloscopes, so I could simply look at the output from the 3DS test points, or maybe some other electronics lying around (I have a spare original Gameboy console that surely has simpler components than the 3DS).

ebastler:You will find that the development software is what really defines whether an FPGA is useful to you

Yeah that started to become more clear as I was seeing gallery shots of the different IDEs used for VHDL/Verilog. Makes sense to grab a more established board and test it out. Any recommendations for good starter boards? I can look in the newbies section of the forum for that too I suppose.

[zzattack]

There's an HDMI-out project for the original Gameboy that is compatible with those small and cheap GoWin development boards. Maybe that's an interesting project to look into first.

[Bicurico]

WHY?

This project seems unnecessary. There are perfect 3DS emulators that will happyly stream the video to HDD.

Why do it the hard way, with chances to destroy your 3DS in the process?

If you want to build som electronics from scratch, look for a project that will actually be useful: you can then share or even sell your development. That seems much more attractive to me than doing something that is IMHO pointless.

Sorry if I overlooked some advantage of doing what you want to do.

Kind regards,
Vitor

[Ranayna]

@Bicurico: If they want to stream on Youtube or Twitch, they legally cannot use an emulator.
And Nintendo is *very* litigous. They *will* go after streamers that even mention emulation.

[Berni]

I'm confused here. The FPGA is ostensibly taking in a flex cable from the 3DS board. There's the bottom screen transmitting 24 bits in parallel (plus hsync and vsync), and the top screen transmitting in serial. The FPGA is merging everything into a single video signal, stored away in some large buffer. Then you'd have your ordered RGB data:

Code: [Select]

Both screens, multiplied by RGB (3 * ((800x240) + (320x240))) = 806400 bits per frame
If the FPGA has converted the incoming data into a 806400 bit-sized frame, is the "USB video interface chip" just... taking the data and shoving it out over USB? The FPGA can't do that task as well?

Yep the FPGA would just be doing video preperation to turn whatever interfaces the 3DS is talking into what the video capture device wants to work with (it is possible the capture device might not like some weird resolutions)

FPGAs are generally pretty dumb pieces of silicon, a fair few rungs down the ladder below CPUs. Generally a FPGA is just a giant pile of logic gates (tho they are more like tiny lookup tables) that get connected together with a pile of muxes to form digital circuitry. By itself the FPGA doesn't even know how to add two numbers together. It has to have an adder circuit assembled inside it for it to be able to sum numbers together. This is where the speed and flexibility of a FPGA comes in as you can build custom circuitry to do any kind of computation. That adder can be 4bits 8bits 32bits 33bits 34bits 199bits, whatever you want and it will still run at roughly the same speed. But this means that in order for the FPGA do so any more complex thinking, such as talking a serial protocol you need to build it the brains to understand the protocol (usually a big state machine), so once you want a FPGA to do fairly complex stuff people will usually just build a CPU inside a FPGA and have it run regular firmware. At that point you have a glorified MCU with flexible customized peripherals.

Yes a FPGA could do USB, and it could even do a custom form of USB that has every 2nd data bit inverted for whatever reason, but it is easier to just take a chip that is designed for doing USB rather than implementing all of this yourself. The chip that is designed to USB will also generally also be smaller, cheaper and lower power.

When writing FPGA code you essentially become a chip designer, it has nothing in common with writing software(even if it does look like code).

Can you elaborate on the "smoothly and reliably" part? I have a project or two that uses OpenGL to draw textures quickly, and I've played around with Windows HID api for game controller input. If I'm just getting raw RGB data from the USB, I could shove that into an OpenGL texture and render it in my application. Seems straightforward enough, unless I'm missing something.

Yes OpenGL is easy for the very reason that you have drivers under your feet that are designed to tightly integrate with dedicated graphics hardware to get the job done. The OS is doing the heavy lifting for you. A HID game controller is dead easy to use trough the APIs (It is all handled by Windows behind the scenes) and needs next to 0 bandwidth to work.

USB on the other hand was never designed to carry video and as such has no such dedicated hardware or driver infrastructure. In fact USB is one of the worst ports for maintaining consistent high throughput because the USB controllers require constant babysitting by the driver to pump data trough it in small chunks. The OS kernel is not agile enough to do this in a timely enough manner so real world transfer speeds suffer. It is actually so bad that it is impossible to reach max utilization on USB 2.0 even when transferring files from a USB drive. You can sometimes get higher speeds from USB on Linux because it has a more performant multitasking kernel in some cases, but still not consistent. Now on top of all this USB shenanigans, you then stick your application that runs in user land and has to deal with other applications jumping in and stealing CPU time. Then to not hog CPU time from other applications you do have to occasionally yield and let the OS do other stuff, how quickly the OS will return CPU time to you is not guaranteed.

This is why it is so much easier to simply introduce yourself to Windows as a  standard video capture device UVC (this is what capture cards, webcams..etc are) and so windows will load you up the appropriate driver that does all the time critical things inside the driver itself (that runs at the kernel level). This driver also works closely with the OS and the USB controller drivers to do things in an efficient and timely manner. But perhaps the most important thing is that this makes the video source automatically show up in all the relevant Windows APIs for video capture (VfW, AVAtream, DirectShow..etc). So any Windows application that captures video will see the capture device and be able to use it (So that you can for example record or steam it to youtube/twitch using OBS). As a nice bonus is means that plugging this into any modern machine running >Windows 7 or Linux will just work out of the box without having to install any drivers or software. It just magically shows up like a webcam would.

[Bicurico]

@Bicurico: If they want to stream on Youtube or Twitch, they legally cannot use an emulator.
And Nintendo is *very* litigous. They *will* go after streamers that even mention emulation.

Thank you.

It now makes sense.

[akp_eev]

This project seems unnecessary. There are perfect 3DS emulators that will happyly stream the video to HDD.

This is a great question! I was going to include this in my original post, but didn't want to overload with unnecessary info.
My other options are:

• Use an emulator (doesn't work well with all games, and I can't use my physical games/data)
• Point a physical camera at the console (janky video quality, and just plain annoying)
• Use debug hardware (VERY expensive and pretty rare, and can't play 3DS carts without custom firmware)
• Use custom firmware and stream over wifi (Framerate isn't great, and can't play original DS carts while streaming)
• Buy an existing Optimize/Loopy capture card (Loopy has bad availability, Optimize boards ship from Japan and are self-install)

Only the last option really meets my needs, but I figured it'd be a fun adventure to make a board myself instead of just ordering one. So here I am!

Yep the FPGA would just be doing video preperation to turn whatever interfaces the 3DS is talking into what the video capture device wants to work with (it is possible the capture device might not like some weird resolutions)

FPGAs are generally pretty dumb pieces of silicon...

There's an HDMI-out project for the original Gameboy that is compatible with those small and cheap GoWin development boards. Maybe that's an interesting project to look into first.

Thank you both so much for this, this is exactly the kind of information I was looking for!
OK, seems a reasonable progression might look like:

• Do an HDMI-out project for original Gameboy, to get my feet wet
• Do an HDMI-out project for some DS console, building on knowledge from previous project
• Do a USB-out project for New 3DS XL

This is why it is so much easier to simply introduce yourself to Windows as a  standard video capture device UVC (this is what capture cards, webcams..etc are) and so windows will load you up the appropriate driver that does all the time critical things inside the driver itself (that runs at the kernel level). This driver also works closely with the OS and the USB controller drivers to do things in an efficient and timely manner. But perhaps the most important thing is that this makes the video source automatically show up in all the relevant Windows APIs for video capture (VfW, AVAtream, DirectShow..etc).

This is so freaking rad! Thanks for this explanation. Two questions here, just out of curiosity:

• This card seems to use USB 2.0. What's going on here? Is this just a janky product? And if it is driverless, how is it presenting as a UVC and able to transfer fast enough?
• If I was a stubborn ass and reaaaally wanted to go the long way around and still commit to using USB 3.0 out from my custom capture board, I could write my own Windows driver to play nice with my output, and then access it from my application. Not saying that would be easy, but that IS a possibility, right?

[Berni]

This is so freaking rad! Thanks for this explanation. Two questions here, just out of curiosity:

• This card seems to use USB 2.0. What's going on here? Is this just a janky product? And if it is driverless, how is it presenting as a UVC and able to transfer fast enough?
• If I was a stubborn ass and reaaaally wanted to go the long way around and still commit to using USB 3.0 out from my custom capture board, I could write my own Windows driver to play nice with my output, and then access it from my application. Not saying that would be easy, but that IS a possibility, right?

Well that USB 2.0 capture card for one it can't do 1080p 60Hz only 30Hz and it is likely going to be using JPEG compression to make that happen. But since you don't need that high of a resolution you can likely get 60Hz out of it. So it might be worth cannibalizing the chipset out of one of these. This is why historically FireWire was so popular for video rather than USB, it had high bandwidth with little CPU overhead.

The UVC drivers do support a few different video formats. That way the data can be transferred in a format that best matches the input stream. Like you can chose between RGB or YUV at various bit depths, it can do chroma subsampling, it can do JPEG compressed etc..

For good results USB 3 is the way to go with UVC because it provides it the required bandwidth to do large resolutions and non compressed formats. The USB 3 controllers also solve a lot of the polling bottleneck issues of USB 2.0 where the CPU doesn't need to intervene as much. This is why all high quality webcams use USB 3.0. But doing USB 3.0 USB 3.2 etc.. on a FPGA is a whole nother can of worms. You need special serdes hardware in a FPGA to make them go 5Gbit, for the 20Gbit USB 3.2 you even need special super fast serdes in a FPGA to go at such a speed. This goes into territory where a single FPGA chip costs $5000 and you need a $10000 per year license for the software to compile code for it. On top of that there is a lot of complexity because USB 3.0 devices first negotiate communication over USB 2.0 so you then need to implement both 2.0 and 3.0.

The way to do USB 3.0 like this is using for example the Cypress CYUSB3017 chip that is basicaly a ARM MCU with specialized hardware inside that can DMA data in and out of a USB3 port at high speeds. The MCU in there is to handle all the USB nitty gritty in enumerating and negotiating with the host and handling control information like setting up resolutions, framerates, firmware updates etc... These sort of chips are found in high quality webcams where they take the image sensor data and pipe it out into USB. You can replace the said image sensor with any other video source. If the chip doesn't take the exact video format you have then you would put a FPGA in front of it to do the conversion (the chip even supports loading firmware into a FPGA over USB).

[akp_eev]

Well that USB 2.0 capture card for one it can't do 1080p 60Hz only 30Hz and it is likely going to be using JPEG compression to make that happen. But since you don't need that high of a resolution you can likely get 60Hz out of it. So it might be worth cannibalizing the chipset out of one of these.

Ah this makes sense. Yeah the resolution on the 3DS won't hit that high. Bought a couple of those cards so I can ogle at the circuits, and just tinker with them.

But doing USB 3.0 USB 3.2 etc.. on a FPGA is a whole nother can of worms. You need special serdes hardware in a FPGA to make them go 5Gbit, for the 20Gbit USB 3.2 you even need special super fast serdes in a FPGA to go at such a speed. This goes into territory where a single FPGA chip costs $5000 and you need a $10000 per year license for the software to compile code for it. On top of that there is a lot of complexity because USB 3.0 devices first negotiate communication over USB 2.0 so you then need to implement both 2.0 and 3.0.

YIKES, that's pricey. I can't imagine Loopy/Optimize are using FPGAs that cost them so much, since they're generally selling their capture kits in the < $200 range. Heck the Optimize cards won't break $100 if you're not shipping outside Japan!

Also, I'm surprised the term SerDes didn't come up when I was searching earlier. I kept looking for PISO shift registers, but I never saw anything about SerDes. Thank you for the heads up here!

The way to do USB 3.0 like this is using for example the Cypress CYUSB3017 chip that is basicaly a ARM MCU with specialized hardware inside that can DMA data in and out of a USB3 port at high speeds. The MCU in there is to handle all the USB nitty gritty in enumerating and negotiating with the host and handling control information like setting up resolutions, framerates, firmware updates etc... These sort of chips are found in high quality webcams where they take the image sensor data and pipe it out into USB. You can replace the said image sensor with any other video source. If the chip doesn't take the exact video format you have then you would put a FPGA in front of it to do the conversion (the chip even supports loading firmware into a FPGA over USB).

YES, this is awesome! I'll start poking around for a simple FPGA I can tinker with for a Gameboy HDMI converter... but while I'm at it I can certainly look into the Cypress stuff. I started looking at the specs for the controller, here! If I have questions about it I may start a new thread.

My one final question is... how would you even begin to handle 2 different screens if you were going to transfer data over USB3.0? One of the cool things about the Loopy/Optimize cards and their custom software is that you can create separate application windows for each screen, sizing/positioning them as you like. So, the software has to differentiate between the top/bottom screen data somehow, right? I would imagine they either insert some sort of control code or something when one screen starts/ends, or maybe they've got a flag on each pixel that tags which screen it goes to (that seems costly, but a dumb bruteforce naive way that could maybe work).

[Berni]

YIKES, that's pricey. I can't imagine Loopy/Optimize are using FPGAs that cost them so much, since they're generally selling their capture kits in the < $200 range. Heck the Optimize cards won't break $100 if you're not shipping outside Japan!

Also, I'm surprised the term SerDes didn't come up when I was searching earlier. I kept looking for PISO shift registers, but I never saw anything about SerDes. Thank you for the heads up here!

The way you keep costs down with FPGAs is by only sticking your custom stuff inside the FPGA and nothing else. Anything that can be handled by external dedicated chips should be done using them(they often cost just a few bucks). So that way you can use a small cheap FPGA without fancy features.

Typically the regular FPGA pins with serialization functionality only run at up to about 400 to 1000 Mbit. Going higher than that requires the special serdes hardware that is specifically designed to go at huge gigabit speeds and uses special dedicated pins for it. It is also one of the common ways FPGA vendors price gauge (these chips usually cost way more). Even then using SerDes takes a good deal of complexity since you have to feed it in parallel from the rest of the FPGA logic fabric that can only run at about 100 to 400 MHz

YES, this is awesome! I'll start poking around for a simple FPGA I can tinker with for a Gameboy HDMI converter... but while I'm at it I can certainly look into the Cypress stuff. I started looking at the specs for the controller, here! If I have questions about it I may start a new thread.

My one final question is... how would you even begin to handle 2 different screens if you were going to transfer data over USB3.0? One of the cool things about the Loopy/Optimize cards and their custom software is that you can create separate application windows for each screen, sizing/positioning them as you like. So, the software has to differentiate between the top/bottom screen data somehow, right? I would imagine they either insert some sort of control code or something when one screen starts/ends, or maybe they've got a flag on each pixel that tags which screen it goes to (that seems costly, but a dumb bruteforce naive way that could maybe work).

For a lot of these USB chips you probably just want to mostly download the applications notes and run the example software they provide. They will often have an application example that creates the correct USB device you need. And you usually get source code so you can tweak and tune it to your liking.

You can send some control data in UVC, or you could even encode data in the pixels of the image. But if the two screens are at a fixed location any popular steaming/compositing software like OBS can crop it down to the part you want (manually set it from pixel X to pixel Y). But if you often need them to be separate it might make sense to simply make it be two separate UVC video sources(maybe the chip can do that, or just use 2 chips and a USB hub chip), each one showing its own screen. Then use OBS to merge the two together when needed.