If I understand correctly, you are using the x10 probe directly on the SMA connector with no 50ohm load on it. With your coax scenario, you are driving a 50ohm load. The amplitude with the x10 probe is more because it is driving less load.
Correct, yes, x10 probe with no load. Well, the capacitance of the probe is loading the Si5351, isn't it? At 125 MHz, that is 100 ohm.
If you put a 50ohm terminator or load directly on the SMA connector and then measure the SMA pins with the x10 probe, you should get more comparable results.
I tried, and this is the result.
I do get the same amplitude, yes! It also looks more like a square wave. Not as good as with the coax, though.
What is the interpretation? was I seeing the complete amplitude as there was no load at the Si5351? So with 50 ohm load, if its output impedance is 60 ohm, I should see ca. 1.3V, if its output voltage is 3V.
A scope will have a rolloff in frequency response. It doesn't stop instantly at its rated bandwidth, and most scopes will surpass their rated bandwidth. So you should still see some squaring off of the waveform.
You didn't say what scope you're using, but if it's a Rigol DS1054Z upped to 100MHz, testing has shown it to have a bandwidth of around 130MHz.
It is a DS1102. Even assuming 130 MHz BW, it should not be able to display a square wave at 125 MHz (the 3rd harmonic is at 375 MHz), and it should not measure below about 2.6 ns risetime. Which it does now also with the x10 probe.