Well, not much progress. Couldn't enable any of the options with TiN's keygen. The original key (with no options) validates ok, but the Tek software won't accept any new key created. Ill look into that some other time.
Channel 3 is now within 5% of the other channels reading. Been switching manually (with a 5V PSU) and comparing channels, and the resistance of the relay near the ADC's was 85 instead of 50ohms. Soldered in some parallel resistance and now it's close to 50 ohm. Strange thing is, it has to be a few ohms lower than other channels to be within spec. Guess it won't do 1,5GHz no more, but at least it should be usable in lower frequency ranges.
Channel 1 is still a showstopper. It has a 280mV offset on the ranges of 50mV and lower. Offset is the same on all ranges. Input will not produce any decent measurement. With an 1kHz 1Vpp signal I can see something superposed on the 280mV offset on the lowest ranges although roughly 200times lower than expected.
This messes up the SPC, so it wont remember its settings. It just says fail, and all channels on every range seems to have zero offset, except for the lower ranges of channel one (the 280mV offset).
I guess I could replace the relays with standard RF170 relays and place my own (quality) resistors in parrallel, but I cant seem to get these relays anywhere. Tried desoldering one of them, but the 12 layer board will take some more work than a standard desoldering gun.