Problem solved - I am no Keithley repair newbie anymore.
I started to follow the signal path with a 80mVAC input signal (followed 3roomlab's tipps), measuring the RMS-AC voltage at pin ACV with an additional control meter (Fluke 87V).
When I measured with the probe at DG111/U103 pin 1 (expected a low), Suddenly the correct value was shown on my control meter.
Reason: a bad soldered pin, loosen over the time. I re-solder U103 and now all works fine. For security a did a check of the other ICs in that area, if they are potential bad also, but none found.
Tool runs again perfect, Self diag runs through, measured values in the low AC volt ranges are correct now. I checked also the AC current ranges - ok.
As the pin was the MUX control pin and not at a part of the signal path, I have the strong hope the calibration is not risked.
But I will verify in the next days.
@ 3roomlab: thanks for the experianced tips.
PS: I found Keithley maintanance description very good and sufficiant, especially the manual stepping through the diagnostic is very helpful.
BR
PeLuLe