They make OR-ing controller ICs that will switch the MOSFETs. This will address the problem of the voltage drop. You might search through Digikey's product index (Product Index > Integrated Circuits (ICs) > PMIC - OR Controllers, Ideal Diodes) and see if you can find something to fit the bill.
Switching the 0V terminals of the sources with a controller IC might be a trick. If you think about it, since the IC is an active component, you need a 0V return for the controller IC to work (it can't power-up to switch the sources otherwise.) So you'll likely have to rely on the body diode of N-channel MOSFETs to provide a return path. Both the high-side P-Channel MOSFETs and low-side N-Channel MOSFETs would be 'backwards' from how they would normally be connected so that the body diodes can conduct to allow the controller to power-up. Once powered-up the controller can switch on the appropriate power source and turn the others off.
The problem with this is that the 0V terminals of the sources won't be able to wander more than about -0.7V from 'circuit ground.'