Than I was looking for the limits what is possible.
RF Gen: 100 MHz, 1 uV RMS (2.8 uVpp)
SDS: 100mV/div (!!!), same FFT settings as before.
Now I need an explanation: 100 mV/div means about 1 V measurement range, 12 bit resolution means about 250 uV resolution. Why it is possible to find the 100 MHz peak of 3 uVpp in the FFT?
You're not really seeing it. The SDS2000X HD quite obviously happens to have a spurious response at exactly 100 MHz. This is why it often is a good idea to avoid such straight frequencies for critical tests. Even better: check in advance with a zero signal.
Dead giveaway: the level of a 1 µVrms signal should be -107 dBm, which is far below the noise floor, as expected.
You see -98 dBm, which corresponds to 2.8 µVrms. With a gross error like this, you should immediately get suspicious.
In principle, you could measure such low signals if you had a much longer FFT available - unfortunately, 2 Mpts won't be enough. For the 12-bit machines there is definitely enough (particularly LF) noise for dithering, so that you can exceed the 73 dB dynamic inherent to a 12-bit system even in the most insensitive 100 mV/div setting. It is differeent with just 8 bits.