So this is what I've been doing and what I want to get away from! Once you start asking the microcontroller to do a lot of stuff, it can potentially be difficult to maintain performance (or should I say, more hassle than it's necessarily worth), and if you find yourself adding more and more functionality as project requirements change, you might be doing a lot of code refactoring, and it gets tedious. If you just pipe the data to and from the PC, you can do all the heavy lifting there where there's plenty of performance to spare. I've never done it that way, but I'm thinking it'd end up being a lot more convenient. After all, the code isn't the project, it's just the means for running and measuring/testing it.
Then there's the whole issue of finding the right microcontroller. Usually what happens to me is that I find a part that has almost every feature that I want except one or two things, which ends up making the project a lot more difficult. And if you can find a part that does everything you want, it often doesn't have a demo board available (or not useful enough if it does) and it's in a package that's miserable to work with for a hand assembled one-off.
So get a bigger one! SoM, rPi, etc. Or any kind of PC, from the same sorts of things to mini motherboards to full boxes.
Obviously as things get bigger, you'll need to manage the higher bandwidth buses and OS stuff. Which is why PCIe looks attractive, even if just used to route between hubs (more PCIe lanes) and bridges or peripherals (e.g. USB ports, and more hubs from there?).
It's a logistical problem, no matter how you slice it. I'm not sure what kind of scale you're talking -- how many, and how many kinds of, things you want to use at once, but it starts to sound like coordinating all of them together is your challenge, and the platform and the ratio of hardware to software work required, is just a symptom of that, not the core problem.
Also if you're working with a lot of mechanical stuff, that's a very much solved problem, in everything from industrial automation to hobby CNC. PLCs are plenty smart these days! (Again, the amount of solution being in relative terms...)
And there's always the go-to lab options, like National Instruments' DAQ systems. They've solved it top down bottom up, from GUI interface to hardware. The price is a tiny fraction of the labor required to recreate it -- very much worthwhile, if you're doing as much work as it sounds like you are.
For one, isolation (to avoid ground loops, or worse) becomes tricky and/or expensive. Might have to do it all on the uC peripheral side. Ugh. The other thing is that the USB stack on a lot of these microcontrollers eat up quite a bit of processing cycles. I suppose this isn't a huge deal, considering that each module probably won't have that much variety in functionality to be a cause for concern, but it's worth at least thinking about.
Meh, isolation doesn't bother me anymore. Avoiding loops is a system level problem, you can only mask it with local solutions.
That said, isolation does happen to be a local solution with unusually high efficacy... But indeed, it's not the simplest thing to do, in terms of parts required, or cost of modules.
Taking differential measurements is the easiest partial solution, and just avoiding ground loops in the system in general.
Something a PC can be a PITA with, an experience I've already had on my bench.
Most recent example: PC is plugged in and grounded. USB goes to a hub, to various peripherals, including a serial port and programmer. These connect to the EUT, a DSP project. The audio output from which also goes to the PC (sound card). It's powered by the bench supply (floating), and probed by the scope (grounded). Well, something about these three paths (USB, audio, scope) leaves a lot of noise, and it's not actually RF interference being detected, it's low frequency (including DC) offsets. Noise goes away when any of the three is disconnected. (Best I can tell, it's actually coming from the PC's USB port, but it's weird.)
I solved the noise issue, for testing purposes, by building a differential amplifier on the breadboard. This reads the audio input's ground reference, and translates the DSP's output level to the new ground reference. Voila, noise gone. It happens to be a local solution, but driven all the same by system-level understanding.
And laptops can be even worse, sometimes a buzz or whine comes through the charger so they're quiet when unplugged, but you only get some hours of runtime before you have to plug it back in again.
You mention a USB to SPI bridge, which is also something I've been thinking about. Some of the dedicated parts that I've seen after a brief search I didn't find to be suitable, but I think there's some microcontrollers out there that would have the performance and flexibility I'd want. The benefit with SPI is that the majority of microcontrollers support it in hardware, and if the USB<->SPI bridge just transmits the data extracted from USB over SPI, there's very little overhead in terms of data transmission required on part of the individual module's uC. Also, SPI - and especially Quad SPI - @ 50MHz has substantially more bandwidth than USB Full Speed which is the extent of what a lot of microcontrollers support. I suspect isolation would be easier, too. The biggest potential downside is that the module's uC might often need two hardware SPI peripherals since it might use that to interface with ADCs etc.
You could easily make (or get) a little dev board that's an MCU and a few ports pinned out, and send e.g. serial commands to it, via RS-232, USB, whatever; more generally, a character-mode interface with internal state. You have commands to set and read register states, and in turn affect the state of whatever's attached to it.
Then it's just a networking and data collection problem, managing a plurality of such interfaces. ("Just", he says
)
Wouldn't be conducive to 50MHz QSPI, no, but... do you really need that? If you're going to blast a bunch of SD cards or something, just plug 'em into the poor computer in the first place..?!!
I did this exercise in GPIB, adapting a popular codebase to the platform/interface I opted for; the generalization of that is SCPI, as I understand it?
This is generally how I approach one-offs, demo new peripherals, etc. I have a serial command line interface I drop into the project, then patch in whatever specific commands I need to play with the device in question. Then build up more elaborate commands to automate e.g. initialization, read/write cycles, drawing graphics for displays, etc. Most recently I did this with an emulated (via GPIOs) parallel bus, but also SPI and etc. Or just twiddling I/O registers directly (or doing some light debugging of the program itself, by viewing/twiddling RAM).
Oh hey I've even got a released example:
https://github.com/T3sl4co1l/ReverbMaybe you'd have to do this anyway, just for starters. I don't know that you'd be able to avoid this, aside from dropping in someone else's drivers or whatever for some particular device -- in any case, that of course still leaves the next layer up, coordinating everything.
I just think that for most devices, a PCIe interface is excessive and is more difficult to develop for than everything else discussed. I also want to be able to use a computer that doesn't necessarily have PCIe available (laptop, SBC, etc), and I absolutely don't want a computer to have to sit on the bus. For what I want, I think it's the wrong way to go.
Right, a laptop might not have it available externally, although if you can hack into a docking station, that might provide some access. SBCs vary, maybe? If it's got a slot, you can always put an extension cord in one. Some SoMs definitely have it available.
SATA and USB3 would also be options? I'm not sure how general-purpose SATA is, it's almost always used for drives. But USB is definitely attractive, and offered by most everything, yeah.
Also, IEEE 1394? Not even sure offhand if that's an enumerated bus or point-to-point only... I never have to use it.
Tim