You are right, it is oscillating a lot. This is how it looks like when I touch the output:
But it is a problem of my measurement setup, I guess because of the cables, because it only oscillates when I'm at the same time near the input with my other hand. I can even adjust the frequency and amplitude when I move my other hand towards the output, very sensitive, like a theremin
I added a 100 nF capacitor to the input and the problem is gone. Now the only remaining artifact is when I touch the input, then I can see mains:
But I guess this is to be expected, because enough current flows somehow from my body into it and back with parasitic capacitors. When I touch the output, absolutely no oscillations anymore, problem solved
And even no need to solder anything, you just need to screw in the capacitor in the banana socket (usually I use banana plugs, so I don't need the holes), just in case someone has the same problem: