kladit, you say calibrated, but: when you measured the attenuator terminated on a 50 ohm termination, you should have obtained ~the same results as with the LOAD from the calibration kit.
After a successful calibration, you get a large RL with LOAD reference on port 1 (or, the same, a small dot in the center of the Smith chart). Approximately the same with the terminated attenuator.
I assume you made a "1 port" or an "enhanced response" calibration, otherwise 1-port measurements are not correct.
If calibration failed (no expected readout of references after it), you may have an analyzer problem.
If calibration was successful, but you get wrong RL measurements with att+load only, you have a problem in the att+load OR in the reference LOAD.