With the signal generator putting out 10 V, and the scope with 1 mV/division (closer to 100 uV/step) that calculates to 100 dB, but I don't expect much more than 70 dB. I just got it working and haven't had time to stress it yet (feel free to hack it). The scope only claims 80 dB or so isolation between channels.
It uses as much data as will fit in a 30 k sample window and extracts (correlates) the signal components at the test frequency. SNR is reasonable -- I think I pick up more from my environment and stray coupling (e.g. 10 V to 1 mV input range is actually hard to isolate). As you can see, though, the plots are quite smooth.
Here's a partial output from the log file showing 200 points per decade (stepping from 1 MHz to 10 MHz). Data is quite noise-free (Note the raw Sin/Cos phase varies because of the t=0 trigger point of the scope; it doesn't affect the results):
Sample 196, 9549925.860 Hz, 500 MS/s, 59948 points; 1145 cycles @ 52.4/cycle
Ch1: Sin, Cos = 1.6443, -1.5963; Mag = 4.58345, Phase = -44.15 deg.
Ch2: Sin, Cos = 0.0063, 0.0021; Mag = 0.01333, Phase = 18.06 deg.
Ch2:Ch1 = -50.73 dB @ 62.21 deg.; Z = 1.3513 +j 2.5805
Sample 197, 9660508.790 Hz, 500 MS/s, 59986 points; 1159 cycles @ 51.8/cycle
Ch1: Sin, Cos = 0.1239, 2.2847; Mag = 4.57618, Phase = 86.90 deg.
Ch2: Sin, Cos = -0.0058, 0.0035; Mag = 0.01350, Phase = 148.84 deg.
Ch2:Ch1 = -50.61 dB @ 61.94 deg.; Z = 1.3825 +j 2.6099
Sample 198, 9772372.210 Hz, 500 MS/s, 59965 points; 1172 cycles @ 51.2/cycle
Ch1: Sin, Cos = -2.2754, -0.2295; Mag = 4.57382, Phase = -174.24 deg.
Ch2: Sin, Cos = -0.0026, -0.0063; Mag = 0.01368, Phase = -112.60 deg.
Ch2:Ch1 = -50.49 dB @ 61.64 deg.; Z = 1.4156 +j 2.6388
Sample 199, 9885530.947 Hz, 500 MS/s, 59987 points; 1186 cycles @ 50.6/cycle
Ch1: Sin, Cos = 0.0164, -2.2851; Mag = 4.57037, Phase = -89.59 deg.
Ch2: Sin, Cos = 0.0061, -0.0033; Mag = 0.01385, Phase = -28.25 deg.
Ch2:Ch1 = -50.37 dB @ 61.34 deg.; Z = 1.4482 +j 2.6658
Sample 200, 10000000.000 Hz, 500 MS/s, 60000 points; 1200 cycles @ 50.0/cycle
Ch1: Sin, Cos = 1.5814, -1.6418; Mag = 4.55909, Phase = -46.07 deg.
Ch2: Sin, Cos = 0.0068, 0.0018; Mag = 0.01401, Phase = 14.97 deg.
Ch2:Ch1 = -50.25 dB @ 61.04 deg.; Z = 1.4830 +j 2.6969
Done
I intend to do a de-embedding step where I loop though the frequency steps without the unit-under-test connected and store that data to subtract from the measurements. However to expect 80 dB at 10 MHz needs a more sophisticated test fixture than a few crocodile clips.