Beware of being over-ambitious, it is the graveyard of many projects... Remember, KISS. (No, not being over-affectionate to Dave)
Sure it can be made efficient, with lots of compute nodes, good airflow, good communications and looking sexy, but I suspect getting all these would hammer the "on time" and "on budget" part of the engineers' trinity. Here are some thoughts to throw against the wall and see what sticks:
A mahoosive monolithic motherboard is going to be expensive and a pain to re-spin, even if only double sided. It would need to be thicker than usual to be mechanically rigid enough to support risers and all the modules. How about mounting some form of substrate (e.g. an aluminium plate) on the stand-offs, and have (say) four or 8 smaller PCBs mount on there. This would allow easier development and hardware debugging than messing around with a huge motherboard. It would allow for much freer placement of your own PCB mount screws etc. Also very useful for hiding messy hardware such as Ethernet patch cables. Is there any access via the other side of the case?
That power supply will be quite crusty now after so many years of hard work, I'd like to see inside it. Teardown! Teardown!
How are you going to support the riser cards on the end farthest from the motherboard? The risers will be quite heavy, especially if populated with Ethernet patches, heatsinks etc. This is one problem when mounting horizontally. PCs solve it by bolting it to the rear of the case. Mounting the cards vertically promotes convection cooling, in a commercial unit, air would be drawn in from the bottom of the case and vented at the top/rear.
The information about the new RPi 3 Compute Module is out. Basically, it is very very similar to the old one, just faster with more memory:
https://www.raspberrypi.org/documentation/hardware/computemodule/RPI-CM-DATASHEET-V1_0.pdf "Raspberry Pi guarantee availability of CM1, CM3 and CM3 Lite until at least January 2023" Which is reassuring.
Compute Modules can be populated more densely, incorporate their own flash, can boot over USB, don't have the horrendous power overhead of the RPi's standard Ethernet chip, and already fit nice standard 200 pin memory module sockets. They do, however, need 3.3V and 1.8V, and the comms problem is still there, although it supports USB OTG, meaning each node can be a device, and could be controlled by a master Pi, which is known to be capable of boot programming CMs.
Which ever node type you go with, I would keep it as simple as possible, at least in the first instance, just to get a prototype up and running and capable of being hacked around with to see what it possible. Sexy can wait until it is working. For example, mount 8 RPi3s flat on a daughterboard / riser PCB on stand-offs, use a short patch cables to connect power and ethernet to the riser. The riser just having 1 buck reg per Pi node, and purely passive traces to bring the Ethernet to the motherboard via 0.1in headers, PCI or whatever. For development purposes, a simple breakout board could be made with PC Molex power connector and Ethernet sockets. I wouldn't mess around integrating a switch, in the final design, the motherboard can break out standard (non-magnetics) sockets to the rear of the unit, through the rigid aluminium substrate, via standard patch cables to a network switch mounted at the top of the unit.
Final thought (for now). If you are going to see the insides of the sexy compute monster, then the Apple panel will have to go, presumably being replaced by a laser cut Perspex (acrylic) panel with an Apple and a RPi logo. This will mean cutting a hole in the old panel I'm afraid!
I'm going to have to draw some ideas for myself, to make sense of all the possibilities!