Let's put a capacitor at the gate for no reason?
Remove C2 from Q3!
Also C1 from Q1 makes little sense.
Are you testing the circuits with some load?
The voltage you're measuring might be caused by leakage currents.
Try adding a 1-10K resistor to the outputs!
In the circuit on the left there is text that supposes C1 on the left circuit is for reducing inrush current (which it does not do), but C2 in the right circuit makes use of the Miller Effect to slow down the switching, and thus reduce inrush currents.
To make use of the Miller Effect C1 should be between the base and the emitter of the BJT.
And inrush currents such sub circuits can be severe enough to starve the main circuit of their power supply voltage.
----------------
As for the original question. It is quite possible that the "turned off" circuit is being powered by a logic line. Normally IC's have ESD diodes between (nearly) all pins and the GND and Power pins. This means that if an input voltage is higher then the power supply, then the ESD diode starts conducting and this can be enough to power the whole (low power) circuit. The 2V4 you mention is also a diode drop lower then your 3V (Not 3V3?) and this also points in the direction of a conducting diode.