Hi again,
I have solved it. It was a faulty op-amp.
Long story short: I tired many different approaches to find the error. But it is not easy when there is no schematic. In pure desperation I set the SMU to generate 21 V to my Keysight 34465 DMM. The DMM showed 21.000 V and the SMU showed 22.8 V-ish (very unstable). Then I took my hot air gun and started blow hot air on the opened SMU. At first from a distance, to see if anything happened, and slowly moved closer and closer. I was particularly interested to see if anything happened when I warmed up components around the A/D conversion. But nothing happened. Not even when I blow hot air to the voltage reference generation or the Altera FPGA.
There are three active components before the A/D-conversion and those are U229 (DG4080 – analog switch), U228 (LT1007 – Op-amp) and U226 (LT1097 – Op-amp).
When blowing hot air on them the SMU immediately showed 21.000 V very stable. After cooling down a bit, the unstable reading of 22.8 V came back. I then took a smaller hot air pen and were able to locate the fault to LT1007.
The LT1007 is buffering the signal from the analog switch. When looking at the output from it, it is very clear that the measurements is done by switching between the reference and the signal that shall be measured. Due to the faulty Op-Amp both these were corrupted, but still similar enough to give a measured value that is almost correct.
After replacing U228 to a new LT1007 my SMU is working again.
After calibration, I now have a fully functional Keithley 2400 SMU.
Have used it for some time now and it works perfect.