@DeepLink: It has been while since I had time to work on the HP. Today, I worked through the service manual and used SPCL function as per your suggestion.
Checked the values of Table 8-12 (Amplitude Limits A2TP5): All correct, within spec. However, values of Table 8-13 for 1.17 and 1.19 were much too low.
According to the manual, this hints in the direction of an error in the "Differential-to-single ended amplifier".
I found out that depending on the selected voltage range (via SPCL function) the problem occurs. "1.17SPCL" (equals voltage range 0.189V) and up (= lower voltage ranges) causes the display to show erratic values while 1.16 and below (= higher voltage ranges) show consistent correct values.
While in "1.16 SPCL" I can reduce input voltage below 0.189V while still getting correct readings. However, as soon as I hit "Automatic Operation" at an input voltage below 0.189V, the display returns to show erratic values.
In mode 1.16 SPCL, connecting a scope to A2TP6 and A2TP7 shows stable voltage amplitudes change proportional to the input voltage. If below 0.189V and switching to "Automatic Operation" the voltage amplitude at the TPs starts to jitter which I believe is the reason for the false display reading. Perhaps a bad capacitor? Is there a commonly known culprit?
I have yet to find out what difference the signal flow between "SPCL" operation and "Automatic Operation" is.
Any recommendation for next steps is highly appreciated.