EDIT: may also check the power supply pins on the PLL for that modulation pattern
...
I've also been around to all the power supply pins looking for 100kHz with a scope. I'll have a look with the SA if the chip is overly sensitive to that.
Ok, according to the SA, all PSP pins are <6mVpp for both 1GHz and 100kHz, except the internal compensation node, Cc.
Cc has a stronger 1GHz component of ~63mVpp. Since that's an internal node there's not much documentation for it so that might be normal. There's a decoupling cap on it to ground and the spec says it should be 10nF. I'm measuring it as 1nF, but that's in-circuit so my reading might be off (but likely not that much). Squeezing a 100nF next to it with a tweezers (I didn't have a 0402 10nF) doesn't affect anything anyway.
I don't think there's much more I'm going to do with it at this point until we hear from Rigol. To be fair to them and let them support customers their way, I've also opened up a case through their web site and I've referenced this thread. If I get anything towards a resolution that didn't also make it here, I'll post back.
I'm just pursuing the 5us jitter problem in the case.