If the signal is positive-definite, such as a voltage reference, then the non-inverting configuration is much easier to use with a single supply.
For the inverting configuration, you need a voltage divider to establish a middle voltage on the non-inverting input, reference the input signal to that voltage level, and the output voltage can then go negative with respect to that terminal. In that case, I find it better to use a true split supply.
AC errors, such as phase shift, depend on the "noise gain", which can be lower for the non-inverting amplifier. Also, lower resistors can be used for the feedback network, giving lower voltage noise from the non-inverting amplifier. In the inverting amplifier, there is always a reasonably-large resistor in series with the signal, increasing the voltage noise.
The red capacitor is almost mandatory for the non-inverting circuit shown: if the feedback network gives high closed-loop gain, with a small resistor from the inverting input to ground, the output voltage will go very high (quiescent conditions). Even for reasonable closed-loop gain > 1, the capacitor will reduce the DC output drift. Since there is also an input capacitor, due to the DC bias at the non-inverting input, your signal gain will fall to zero at DC, anyway.