Hi, Dave.
I know your focus tends to be hardware, but have you considered a more 'software' alternative to this project? That is, something different than Ethernet to connect your boards?
Thinking being...
* You don't need high performance, you just need connectivity
* You've got a lot of header pins, SPI, etc to create a shared comms bus
* One Pi on the bus could act as a gateway to the rest of the world via its Ethernet (using NAT)
* The missing link would seem to be a network driver implemented on SPI (i.e., SPI between Pis, not to an Ethernet controller)
[Edit 1/7: I've learned that no SPI slave driver has been developed for the RPi, so you'd have to bit-bang your own bus+network driver using I/O pins.]
I'm on a similar quest, but I haven't yet gone hunting for such a driver, so I don't have a turnkey solution for you. But boy, would it simplify the design - I've got to think someone has coded this by now. You could practically use an IDC ribbon cable for the bus and some standoffs to stack the Pis. (Or, for a PCB design, a right-angle female 2x20 header pairs nicely with an IDC shrouded pin header - no PCB slot required. Cost for headers from Adafruit is cheaper than Digikey, at least for the US.)
The above would have a bonus of being a low-cost solution for Pi Zero, which might make it $/GHz more attractive. If you could circumvent the need for an SD card (which would be a whole separate project), that practically cuts the Pi Zero cost in half.
Speaking of Zero, the new Orange Pi Zero looks pretty darned attractive at USD$7 with quad core 1.2GHz, 256MB RAM, and wireless. All it'd need is a couple pins for power. OS support seems sketchy out the gate, but I expect that'll resolve itself in due time. If it works well, it seems like game-over for this kind of 'clustering' project.
And Pi Compute Module... has previously offered small form-factor, but higher cost. Since the CM3 is claimed to be backward-compatible at the connector with CM1, I expect it to similarly be a subset of Pi3. But it does have MMC instead of SD, which is nice. It's handier for integrating into a commercial product, but if you can make a Pi3 work it'll probably be a cheaper 'Raspberry' option (i.e., if you ignore Orange).
Cheers,
Richard