Well this was an interesting experiment. I've spent the whole day at the bench, measuring, modelling and modifying this prototype. In the end I did manage dampen the resonances in the attenuator section to an acceptable level, but only after limiting the bandwidth to 250MHz and eliminating the relays from the signal path. I assessed the performance of the attenuator by comparing the transient response to that of a second OPA653 channel assembled without an input attenuator - ideally, the only difference between the two should be in the signal amplitude, not the transient overshoot or ringing. After that I solder wicked all of the remaining input network components from the board and assembled a passive 10:1 (no relays!) attenuator dead bug style as compact as practical right at the input of the OPA653, with the input BNC relocated to the same location. That worked just as well as the unattenuated reference channel in the full 500MHz bandwidth.
In a nutshell, my initial prototype design would have made an excellent front-end for a 100MHz oscilloscope where all of the =>500MHz infelicities would be masked, but as a front-end for my Tek 7904A, it turned out to be a design failure. So now I find myself ruminating over which alternative direction to take. I still want the 20dB attenuator option for the signal handling capability it provides (think probing the high voltage switching spikes produced by a SMPS), but I can't do without the sensitivity of a 1:1 throughput for small signal stuff either. A selection of AC/DC coupling is also mandatory, but the switching between the two at these frequencies is problematic.
So what I am contemplating doing now is making an eight independent channel design. Four of these channels will have 20dB fixed attenuation and four will have no attenuation. Two of the -20dB channels and two of the 1:1 channels will have fixed DC coupling while the other two channels in each group will have fixed AC coupling. Not quite as convenient as only four channels that can have their mode of operation switched, but far better from an electrical, performance and design perspective as all of the input stage switching (relays) and attendant layout compromises are eliminated. I'll probably house these in a 1U-height 19" rack case.
In addition to all of the above I can report happily that the OPA653 is a fantastic performer. In my entire ~8 hours of experimenting at the bench not once did I witness even the briefest burst of RF oscillation.