After a bit more probing around, it's not the USB/UART, that was just from bad probing
It appears to be when I use the external clock, here with internal clock (all measured directly on the AD5933, on input between SGND and feedback using very short ground connection, the output is clean)
Reasonably nice and clean.
But when I use external clock, it looks horrible with small signals:
And zooming, it's surely the external clock:
Edit: added details on measurement