Realized the digital output driving solution is dumb just because I can get away with using MOSFETs. Oh, and I'm at the 500 pin limit of my DipTrace Lite license, that's another reason using 74AHC125s is dumb, and yes, I was too lazy to come up with something else. I guess I just can't get away from DipTrace because I've been using it for a long time and I didn't need any tutorials to figure it out. Eagle would be another option, but the free version has a 80x100mm board limit, useless. KiCAD is a pain in the ass to use, installed it for the third time and then uninstalled it after 10 minutes, Circuit Maker has the annoying thing of having to be always online... anyway, rant over.
Mounting holes might cause me some problems, mainly because everything has to be stackable. The UI board is quite crowded, as you can see in one of the previous posts, but the 80(H)x100(W)mm size is pretty much what I'd like to stick with, or if it's not possible, something with nice round numbers. The main problem with the size comes from the fact that the Waveform Generator board has to be at least 70mm wide because of the locations of the mounting holes on the UI board and has to extend about 20mm below it because of the BNC connectors. So far it's 100(H)x80(W)mm.
On top of that there are going to be 2 Aux Power Supply boards, one with the Analog supply rails, the other one with the Digital supply rails. They'll have to be supported mechanically by the Waveform Generator board, that means 4 extra holes. No room for 4 mounting holes on the UI board but I won't need them there anyway.
Another plug-in board would be the digital isolation module for the Lab Power Supply, that will require only +5V which will be provided by the same board that powers the digital part of the Waveform Generator. I'll have to keep that in mind then I design the Aux Power Supply modules.
An alternative to the boards directly plugging into each other is to use angled pin headers and a pieces of stripboard (soldered or with connectrors) linking them together. Not very elegant but it would do the job. In any case spacers will still be required to keep everything from falling apart.
Finally, I've decided to make this Open Source, and I'll probably do some (rather amateur-ish) YouTube videos if I have enough time after I assemble and test the first boards. GitHub stuff coming after that. Stay tuned for more.
Oh, one more thing, does anyone know any free 3D modelling program that accepts .step or .wrl models and it's to use so I can check if everything fits together perfectly?