Whatever I tried the moment I disconnect the debugger the same code on the same hardware seems to work differently. It has to do with the stop mode and the LSI clock which seems to cause some synchronization problem. Then I found out that you can enable the debugger in software without it being connected. This solved my problem as I got the same behavior with or without debugger connected. Here the 2 lines of code which made it work:
// simulate as if debugger is ON by applying clock and enable stop mode
RCC->APBENR1 |=0x08000000; // apply clock to debugger
LL_DBGMCU_EnableDBGStopMode();