Hi,
I have an old hp 8970a, but (I think) a good Keysight 346B noise source, i repeated your set up - and got similar results at high horizontal CRO speed.
At slower speeds (the other picture) if you 'squint' I think you can see the difference between the 'on' and 'off' periods.
I think your gear is ok.
I got my set up to test some very low noise factor rf amplifiers for a pre-amplifier in amateur radio earth - moon-earth experiments. I had modest success being able to measure the thermal rf noise of sun, tree and cold sky. Set up was for 1296MHz. Using the low level noise source 346a, I am reasonably confident the amps I built are about 35dB gain, NF about 0.6, they are a G4DDK design.
Regards Rob