You may still have some power issues with this depending on what's connected to the MCU and how. Many ICs have protection diodes connected between their IO pins and their internal power rails. If you've got MCU pins connected to other ICs on the board, and the MCU drives any of those lines high while powered by the programmer, current can flow from the MCU pin, through those internal protection diodes, and into the 5V rail. What happens next depends on what's connected and how. With a light enough load, the rest of the circuit may power up and run more or less normally this way. With a larger load, you may brownout your power supply, or the MCU may not be able to source enough current, and the circuit may not operate correctly--or worse, may go into a self destructive state, either because it's not operating at the correct voltage, or because current is flowing in ways it isn't intended to.
You can provide protection against this in the MCU firmware, by having it check how it's being powered before configuring any IO. You could do that by connecting a spare GPIO to the power pin on your programming connector with a pulldown and read the digital level to see if the programmer is conencted. Or since you seem to have an external reference voltage, you could check if that's present, either by reading the digital level, or by measuring the voltage with ADC--typically you can set the ADC to use the MCU's internal voltage reference and then measure the external reference voltage against that.