I am/was aware I lack understanding of this circuit, and am blind to it's inner workings. But the reason for that is... it's very hard to simulate.
The stupid changes I made, were made to make it work in LTspice. Then, when not getting further there, I decided to build one to see. Obviously build it "as simulated"...
I didn't have a suitable PWM generator at the time, so it got shelved and forgotten. Now 3 year later, better generator & simple FPGA capability at hand,
previous problems and details happily forgotten and inspired by your post; I did some updates to the surrounding circuit, but just left the not understood "core mystery part" alone.
I now tried simulating it in Qspice as well, and it needing different "simulation help" circuit elements (and turning off "fast math" in the settings as well), it's still quite temperamental.
But at least I could see a bit how what I added wasn't helping. Now removing the parts originally "added for simulation", output HF noise and switching blip did decrease tremendously
I've attached the updated schematic and layout and a picture of the Rdson of the switches used, when running +-Vref instead of Gnd/+Vref the resistance symmetry should be good enough?
Edit: Surprising to me, the NL shape didn't change much from that though!