I don't think you're seeing any problem with the scope, it is just the nature of trying to use a 50 ohm feed through connected to the 1 meg-ohm input of a scope. A scope with a 50 ohm internal path is optimized for such things. Using a feed-through will only match a proper 50 ohm input at very low frequencies.
Here is an example of what my Keysight scope looks like with its native 50 ohm input, and then using a 50 ohm feed-through with the scope input back at 1 meg-ohm. It looks absolutely horrible using the feed-through. The third shot is the 50 ohm feed-through on its own just for reference.
I mean this scope is not designed for testing above 350MHz signal. It just not perform well and maybe it needs some hardware modification to make use of the high sample rate.
You cannot just hack it.. For higher frequencies you need to use 50 OHm path, and that has to exist in the scope from input connector to A/D converter. It has to be controlled impedance layout.
Basically, it has to exist as separate part of PCB made for just that purpose that is just not there on MSO5000 board.
Chipset and front end chip in MSO7000 and MSO5000 is identical and capable of same bandwidth (frontend chipset is capable of few GHz actually). It's just that your signal from input BNC cannot get to it without being destroyed.
For a scope of this class it is more important that it has good 300 MHz with good signal integrity(which is a miracle itself), that hacking it to 1GHz with distorted signal. You get worse scope actually, and much more noise...