I've made SMA kits like this for years. I would have expected slightly better than this, maybe -30 dB up to 6 GHz (I didn't go much beyond that).
A couple of thoughts for getting higher in frequency though:
- try four 200 Ohm resistors?
- test it removed from the steel cavity, or enlarge the cavity behind the connector
- don't worry about the resistor tolerance, 1% is like -50 dB reflection coefficient.
- use Suhner / Telegaertner connectors instead of eBay cheapies
- look very carefully at the Smith chart, and then start fiddling with the solder around the centre pin, and possibly the ground, to adjust it.
For the last point, before you try to read the Smith chart, you need to port extend. To my eye your chart looks like it has a half-wave-at-10ghz of uncorrected cable.
- First calibrate with a better cal kit than yours
- then install your short, and adjust the port extension so that you get a clean short circuit on the screen. Cal kits have various reference plane positions, not necessarily at the back of the basic 4-hole SMA. I assume your short is made from copper tape flat on the back of the connector.
- now try the load, you should see just one tail of L or C, not a spiral.
- a spiral means you need a better connector.
- it's probably inductive. Add a blob of solder, or a short wire perpendicular to the Rs, or bring some ground closer to the pin.
- or add a small disk of something, to enlarge the pin to twice its diameter, before connecting the resistors (and switch to 0603, etc). We would do this with the centre pin of a PCB-mount connector, tweaking the size of the pad and the ground clearance for best VSWR.
Thank you for fighting the good fight, plotting Return Loss as a positive number.