The problem is the programmer resets the MCU (there's a magic SWD sequence for this), which after resent immediately starts executing code. The programmer then attempts to halt the processor, but there is a delay in the time it takes the halt to be issued, issued and so the MCU may already have executed your init code that configures SWD out of existence. The result is you can't gain control over it. If you hold RST# this means the programmer is sitting trying to halt by the time you release it and eventually you'll luck out and succeed.
Needless to say, leave the debug pins until you're 100% confident everything else works reliably first, then by all means make minor changes to use them for something trivial that can easily be disabled if you need to debug some problem. But more specifically, I never do this in dev builds; in those if there's a problem and the firmware panics it will sit blinking a LED waiting for me to attach a debugger. The panic message will captured in a buffer, and the traceback will give me the location and stack context. This isn't possible if the pins have been reassigned (either that, or they need to be fiddled with in the panic handler, which I would recommend against also).