It's not a good idea because the output transistor would be subject to reverse voltage, although most transistors can normally handle 5V with no problems.
To make sure nothing bad happens, connect a diode between the input to output with the cathode on the input.