So it turns out that on my unit, the spectrum analyzer, signal generator, and two-port insertion loss features seem to work fine, but the one-port insertion loss and return loss measurements give incorrect results. Since there's no block diagram available, I ended up taking apart and reverse engineering the receiver board, shown below. I don't have much previous RF experience, so corrections/suggestions are welcome!
It seems that this unit is a superheterodyne receiver using double conversion, with the 1st LO at 1.5 - 3 GHz, the 2nd LO at ~3 GHz, and a final IF at ~135 MHz. The unit seems to have two separate receiver bands, one for < 2.7 GHz, and the other for 2.7 - 4 GHz ??. There are 2x 5-bit (<= 31 dBm) digital attenuators on the input path, and a 0 - 90 dBm electronic attenuator on the output, although the module seems capable of up to 130 dbm attenuation.
Since this unit is designed for portable operation, the internal cables are short and the assemblies are all stacked on top of one another, making it difficult to probe the receiver board with the unit running. I ended up removing the connection panel and flipping the receiver board, which gave me access to its top side. Since the spectrum analyzer and signal generator features work independently, I'm guessing that the fault probably lies on the feedback path from the source output on to the input path, which runs from the coupler through some filters and SPDT switches (from D to A on the annotated image). Interestingly enough, some rework seems to have already been performed on the filters on this path, probably by the factory?
Next, I plan to probe this path while operating the unit in return loss mode, but this will be a bit tricky because I only have a 100 MHz scope, and no other spectrum analyzers or high-frequency probes. However, I do have a RTL-SDR and a LimeSDR, so I can probably make do, and just need to compile the software stack on my laptop.