In single supply circuits I'm used to put a switch on the positive rail and capacitors between the virtual ground and the negative and between the positive and the negative of the battery rails (close to the supply terminal of the opamps).
I realize your circuit replicates the usual schematic of real ground circuits. So I wonder if I'm wrong or if that makes any difference.
The battery and the switch are in series, the order of components in a series circuit does not matter, only the orientation (of some components) needs to be maintained. Thus it makes no difference where the switch is, if you like it better on the positive battery terminal you can put it there and it will function the same.
I think batteries have low noise, but with a noisy single supply voltage, wouldn't capacitors from the positive rail to the shunt regulator increase the virtual ground noise?
The capacitors are there to provide a low impedance return path at high frequencies for the signals flowing onto the voltage reference node through R3 & R5 since the impedance of the shunt bandgap reference U3 increases with frequency do to the roll of of it's internal feedback. The battery starts as a low impedance, but that increases as it discharges.
I did not really pay that much attention to the decoupling capacitors. I just put them in and only paid attention to the 10µf limit of capacitance in parallel with U3. I suspect the circuit would work fine C1 through C4 removed. If I was going to mass produce this I might look into removing them to decrease the parts cost.
The simple answer to your noise question is yes, but the real answer is more subtle. You would need to look at the impedance of U3 vs frequency and factor that with capacitors, the sensitivity of the resistivity of Q1 to V
CE changes, the impedance of the battery vs frequency and discharge state, and the power supply rejection and balance that against the high frequency current return paths. I did not do that analysis since I did not think it was important in this case. However it might be in another case.
Both of your questions are good questions, thank you for asking.