The software I'm using doesn't seem to implement that. I read through the code for the TDR measurements and it just does the IFFT. Nothing fancy.
Anyways I swapped the two mixers and now I seem to have >50dB isolation across the entire band. I'm not exactly sure why this happened. The blue line is what I get when I connect a cable, and the brown line is with nothing connected. The difference should be the isolation. There is a peak in the blue and brown lines near 22 GHz but that is just because the mixers are running out of LO power, and one of them seems to be significantly better at lower amplitude than the other (probably because they're using different diodes). In my next design I'll probably pay $30 more and buy higher power amplifiers.
I'm not sure what's causing the remainder of the error. If I disconnect the ADC the noise floor is around -90 dB, so it's not the ADC. It's not the RF relay either.
Edit: my IF bandwidth is 1.3 kHz. Here is the frequency response of my "filter" (which is just correlating the sequences 1, 0, -1, 0, 1, 0, -1, 0, ... and 0, 1, 0, -1, 0, 1, 0, -1, ... with the received signal at 48 ksps). The IF frequency is 12 kHz.