Very nice project! I like UI design very much (used to R&S screens as well). Agree that touch interface on such a small display could possibly make things only worse. I am not sure about SI5351 use for LO - because of it's jitter. 70 ps pp jitter will introduce phase noise into 1st 40MHz IF. This can degrade phase resolution and not only. VNA shall have low phase noise LO/clock sources through all it's paths, down to final ADC clock. Just checked that PLL of stm32 you use have 60 ps pp jitter. Not so critical as for 40MHz IF, but anyway worth to check using online ADC aperture jitter calc.
https://www.maximintegrated.com/en/app-notes/index.mvp/id/3359
https://www.maximintegrated.com/en/app-notes/index.mvp/id/4466
https://www.maximintegrated.com/en/design/tools/calculators/general-engineering/jitter.cfm
http://n2pk.com/VNA/JitoPN.html
You could possibly do quick & dirty tests by running LO of 1st IF @ 27MHz - from TCXO. Most likely MCU will not mind throttling and will perform OK'-ish at 27MHz clock (w/o any internal PLL). Also two cents: you could consider to use lower (audible) frequency for 2nd IF and sample it using external 24bit (audio) ADC, thus with comparably small effort getting significant dynamic range improvement.
Your thoughts?
Thanks for the comment!
You are right, SI5351 is infamous as being not the most low noise synthesizer. I admit, I have not paid much attention to noise of oscillators, as currently dynamic range is mostly limited by leakage anyway. But probably if leakage terms could be measured with less noise it would help to improve the calibration. Thanks for the links, I will take that into a account. Also maybe trace noise in low amplitude measurements could be improved. But these are the things to improve when my systematic error issues will be perfectly solved.
About 1st LO @ 27 MHz: If I assume my block diagram, the 1st LO tracks the RF signal with difference == IF1. I could try IF1 at 27.1625 MHz for example, which would allow to use 27 MHz OCXO for 2nd LO to obtain IF2 of 0.1625 MHz.
I have tried clocking MCU from RF OCXO before, but I did not see any difference/improvement. Currently MCU (and its ADC) is free running/not sync with rest of the system, REF and RX signals are sampled simultaneously and amplitude&phase is obtained relatively. It should be only important that MCU clock does not drift during acquisition of current data chunk which is less than few 10s of ms anyway. But probably I did not understand Your note about MCU throttling
About more ADC resolution - currently I think 12 bits are plenty, again due to mostly leakage/systematic error not noise limited. But lower IF2 and external ADC would be easily possible.