Thank you TimFox. I agree with what you wrote and that's what I was expecting.
I made sure that all the impedances are set to 50 but the numbers are off.
At 1Vpp from the wave generator, which btw is the wave generator embedded in the DSO I use, I measure 150mV (rms) on the SA.
With 2 Vpp I measure 208 mV (rms), while I was expecting 707 mV (rms)
Of course, the SA could be out of calibration. Does it have a calibration routine or a calibrator output itself?
What RBW and frequency sweep parameters did you use, at what generator frequency?
(Please be quantitative on the relevant parameters.)
In cases like this, I would also check the DSO if it is also the generator, since there could be a problem common to both functions.
It would be a good idea to use a third independent method to measure the voltage from the generator.
It is very suspicious that when you double the voltage, you did not get twice the result.
Also look for distortion: if you are overdriving the SA input or overloading the generator output with a 50 ohm load, you may have less voltage at the fundamental frequency than you expect.
In my own use of an SA, I sometimes find that a squirelly measurement is due to my accidentally selecting dBV when I meant dBm, or similar.