It is now very clear something is somehow wrong. But now after your comment, more mysterious. And then even more after these two coaxials. I assume they both come also from separate generator outputs so that just gen out - cable - scope input.
Your assumption is correct.
When I look this image and previous, what FFT window you have used. (nothing to do with this wobbling)
For this kind of things I use always Flattop if not serious mandatory reason to use some other.
I've used Flattop window.
Just only for thinking what is big and what is small.
For just example imagined situation where all is pure resistive.
INPUT DATA
Set Source Impedance:50
Set Load Impedance R:45.25
Set Load Impedance J:0
RESULT
Absolute Load Impedance:45.25
Load Reflection Coefficient:0.0499
Load VSWR:1.105
Load Return Loss:26.04dB
Load Mismatch Attenuation:0.011dB
If pure resistive load 33.5 ohm
Absolute Load Impedance:33.5
Load Reflection Coefficient:0.1976
Load VSWR:1.493
Load Return Loss: 14.08dB
Load Mismatch Attenuation:0.173dB
Yes, the load matching isn't ideal - and the scope input is not pure resistive for sure either.
When thinking about it, isn't a VSWR >1.1:1 (10%) per definition nearly 1 dB ripple?
On the other hand, a VSWR of up to 1:1.5 is quite typical for proper 50 ohm inputs on DSOs.
So if I'm not wrong, there is rather the question why your coax measurements look so clean and wobble-free?
I am now more sure that you have some problem in some hardware you use. Scope. Cables. Generator.
I made new test. Parallel test. Using 2 RG316 cables with connectors what also mage perfect coaxial GND and center connection.
(and when I did this test I also find one my matched cable pair was bad so in this test now these cables are not perfectly equal length, nonsense here)
Signals come from SDG6000X. Both channels do same sweep 1MHz - 500MHz. Sweep time 300s.
Signal levels.
Gen Ch2 to scope CH4 roughly 0dBm. CH4 AC 50ohm.
Gen Ch1 to scope CH1 adjusted so that voltage level @1MHz is equal with CH4. CH1 AC 1Mohm - so really bad "noob style" mismatch when he forget termination. As can see there is just these waves and very roughly around same level as waves in your signal. My scale is 2dB/div.
In this test I use both voltage bands, I and II. As can see this change and it also change impedance as can see also CH4 small waves after 250-300MHz when voltage band I is in use.
If now look and compare very bad termination situation and good termination it is clear that there is really some big problem in some HW what you use. My upper trace (CH4) is as good signal transfer as can be with my available things here and lower trace CH1 have 50ohm source with 50ohm cable to scope 1M input. Not as bad as short or open but... still "horrible" and as can see least these waves are in same ballpark with you image.
But it also feels very very mysterious after all you have told... only can know
something is failed or other ways wrong in your setup.
Your previous image.
CH1 without 50ohm termination, just 1M input. RG316. Same kind of waves as in your images.
CH4 50ohm internal load resistor. RG316.
1MHz signal voltage levels matched.
Source. SDG6000X separate channels and both sweeps 1 - 500MHz and swp time 300s.
FFT window Flattop and max-hold.
You Wonder, In my pictures, the signal drawings are smooth.
Well... When I go on a bus as a gray-headed elderly people here in China, the young peoples get up beautifully and offer a seat place even though I often thank and show that I can stand.
Could it be that the test devices are behaving somehow proportionally same and behaving beautifully front of me.