I will remove D3 and D6, I can see how they are redundant.
Don't get rid of them. It's more reliable with them.
D3/D6 protect against reverse polarity. This is an unlikely or potentially impossible scenario (only the OP knows for sure what’s connected, but it’s described as an audio amplifier). In that case it would technically be less reliable if fitted.
As for the size of the input capacitors, this isn’t so straightforward. With the circuit as is, the datasheet reckons you’ll achieve something like 86dB of rejection (quite a lot, but datasheets can fib), so any ripple you get will be attenuated by 86dB (this will drop hugely as the frequency goes up). Then you’ve got the CMRR of the amplifier, which might be perhaps 40-70dB, which will mean that you’ll be reducing 50/60Hz ripple by perhaps 130+dB, far in excess of what you’ll hear or be able to measure.
So going mad with the input capacitors will likely not help. Reducing the HF component of the noise (from the diode bridge, transformer noise, mains borne interference) will be as (or more) important - so adding an LC filter after the each rectifier might be a better use of space/BOM cost. Look up the rejection graphs at higher frequencies.
As for the topology and ‘ground loops’ - nothing wrong with the design, you effectively have two separate floating supplies, commoned at a single point. In some senses it’s a better design as you’ll get the same performance out of each (not that a LM317/LM337 complementary supply is has wildly different +ve / -ve performance).