So I've been playing around a bit. I've also tested a Mini-Circuits ZFSC-2-1 (5 MHz to 500 MHz) and an Anaren 10014-3 quadrature hybrid (500 MHz to 1 GHz) at various frequencies and tone spacing. It seems that at very narrow tone spacing like 1 kHz I'm indeed hitting a limitation of the analyzer. I currently have the ZFSC-2-1 hooked up and I'm getting 3rd order products lower than -97 dBc with 3 kHz spacing using the method described by G0HZU, i.e., zooming in on the 3rd order product. This is a bit challenging because sweeps tend to get slow, and one has to determine the right amount of input attenuation to keep the noise floor at reasonable levels without introducing IMD in the first mixer of the analyzer.
Not entirely sure, but maybe with some more effort one could make better measurements also at 1 kHz spacing. I think this contribution summarizes it very well:
To explore the limits of a decent modern swept spectrum analyser (swept LO with a wideband digital IF) it's best to use wide tone spacing and then select a very narrow span to zoom in and just look at the IMD tone itself. This assumes the analyser has a suite of analogue pre-filters ahead of the ADC and this is generally the case for a high end spectrum analyser. These pre-BPFs will typically be 2 or 3 times wider than the digital RBW filter and the analyser should select the optimum pre-BPF automatically. The BPF protects the wideband ADC from having to cope with the main test tones if you set the span much narrower than the tone spacing.
It should be possible to achieve the SFDR on the datasheet for the analyser using this method assuming the signal generators and the combiner are not limiting the performance. With a RBW of 1Hz some analysers should manage 115dB IMD3 SFDR. Older classic analysers like the HP 8568B use analogue RBW filters and this analyser can typically achieve just over 100dB IMD3 SFDR using the smallest RBW of 10Hz. This classic old analyser typically has a mixer IP3 of +13dBm and a DANL of -140dBm in a 10Hz RBW. So the IMD3 SFDR will typically be about 102dB.
For the FSIQ the IMD performance is quoted for tones at –30 dBm with ∆f >5 x RBW or 10 kHz, whichever is greater, and 3rd order products at < -74 dBc (TOI 22 dBm typically). So it seems I'm asking for too much here when I try to measure everything in one sweep.
Edit: After thinking a bit and reading some application from the manufacturers, the excessive IMD I have seen was likely due to the ADC being used to implement the digital RBW filters. Quoting from
this application note:ADCs are non-linear components whose intermodulation distortion follows rules which are different from other common RF components. Their intermodulation distortion is not specified using the TOI, but is included in the spurious free dynamic range (SFDR) specification. SFDR covers not only the intermodulation products, but all unwanted signals. Since SFDR specifications are given in "dB below full scale" (dBFS), it is essential to scale the signal correctly before applying it to the ADC. On a spectrum analyzer, the so-called IF gain, which is often coupled to the reference level, is used to optimize the signal level in front of the ADC.
The intermodulation products of an ADC remain more or less constant, independent of its input signal level. This in return means that the SFDR is dominated by the signal level, which means that a higher ADC input level will result in the same increase in SFDR.
Recommendations:
1. The best method to get around the ADC related IMD contribution is to avoid having two tones at the ADC input simultaneously by selecting an appropriate tone spacing (e.g. > 5 MHz for the R&S FSW and R&S FSV).
2. If the tone spacing is fixed and cannot be changed, the CW tones should be close to the full scale level of the ADC. The R&S FSW in default setting automatically takes care about the signal scaling for the ADC, using as much of the ADC’s scale as possible and avoiding an ADC overload at the same time.
This is the reason why the IMD products did not depend on the reference level setting. Also, it is good to keep in mind that the ADC is not included in the TOI specification. Easy to forget when you sit in front of the instrument.
The FSIQ and FSEA only use digital RBW filters for RBW settings < 1 kHz; the larger RBW filters including 1 kHz are analog (LC and crystal). When digital filters are used, the 3rd IF at 21.4 MHz is downconverted to 25 kHz and the sampled by an 18 bit ADC at a rate of 200kHz, with the analog filter section still in the signal path. Unfortunately I cannot find anywhere in the manuals which analog filter is used with which RBW or span setting. But since there is an 1 kHz crystal filter, there is a chance that it is switched in front of the ADC with narrow spans or RBWs. So it may be possible to measure 3rd order products with these instruments even when the two tones are closely spaced.
/Edit
I'm stating to believe that it is the darned splitter itself that is causing the IMD.
Might be, especially if that splitter has some magnetics core, but I don't have enough practice to know what IMD levels to expect.
From the datasheet it's not clear if that model is a resistive splitter. Since the specs for it specify a minimum frequency, that might be a hint that the splitter has some ferrite inside. Ferrites have hysteresis and saturation curves, they are not linear devices. Therefore, ferrites materials are expected to introduce distortions (and thus to produce IMD).
It's a splitter with 3 dB loss in both arms, with 30 dB of isolation, so it cannot be resistive. And I just popped the lid to confirm that.