I have quite some confidence in the DMM: it shows a low battery relatively early and did work well the last 30 years or so. The same meter shows normal 5.08 V at the USB supply when nothing bad connected. I did not consider the DCDC converter to produce very much EMI - at least the voltmeter part it powers works fine, with little interference (maybe a little noise in the 7 th digit). So I don't think there was enough interference for the DMM to read wrong. The input capacitance of 12 µF is a bit low for the DCDC converter, and slightly above the 10 µF that are supposed to be at the USB max. I still don't think it should be so sensitive.
The PC is more like an office computer (lower profile desktop with a more custom power supply). So it is not so easy to exchange, but I also don't expect it to be build super cheap.
It was not just the DMM reading high, but also the chip on the USB-UART bridge (just used for power through) was getting hot (could smell it and it still felt hot after unplugging it).
Yes I consider myself lucky that nothing got really damaged. I only had the higher voltage for maybe 5 seconds or so.
Anyway I have no intention to repeat.
So no more USB power to DIY circuits from this computer.
Just a fuse and maybe a FET to turn off the power is also what I expect on the main board.
It seems the USB uses kind of standby power, as some devices still get power when in standby mode.