I just identified the value I entered, which triggered the observed -40dB linearization behavior. I thought, I had to enter the precise resistance of the load in the Z0 setting of the load (which is of course wrong, since it is an impedance, not a resistance). So, I put 51,06Ω there, which then triggered the described behavior. Just entering e.g. 53 as the Z0 parameter for the load in the Custom Ideal Standards set can trigger the same effect. The more the value differs from the other Z0 values for the open, short and thru, e.g. 50Ω, the worse the return loss will get.
I'm not sure I understand what you are after.
1-port SOL calibration basically works as follows: You have three standards with
known S11(f). The VNA's calibration procedure mesures their response, and you tell the VNA the known ground truth S11(f) values of your standards. Then the VNA can calculate the error correction terms for subsequent real measurements. It does not matter if the load standard is exactly 50 ohms or not -- you just need to know its
actual S11(f). The same applies to the short and open standard as well. They do not need to be ideal (you cannot realize ideal standards anyway), but you only need to know their actual S11(f).
If you tell the VNA that your standards are ideal (although they are not), then you have to live with the resulting error. This may be good enough at low frequencies if the load standard is very close to 50 ohms, but at UHF or GHz frequencies every millimeter of length begins to matter, and parasitics play more and more a role.
Instead of dealing with tabulated S11(f) of the standards directly, usually a parametric model with a low number of parameters is fitted to the measured S11(f) of the standards in order to
reduce dimensionality, which also aids noise reduction and enables re-calculation of S11(f) for arbitrary frequencies from the model parameters. In this case you have to tell the VNA the (known) model parameters of each standard instead of S11(f).
EDIT: If you calibrate the VNA using the actual model parameters of the standards and then measure the load standard (with calibration applied), then the VNA is expected to display the actuall S11 of the load standard.
OTOH, if you tell the VNA that your standards are ideal (although they are not) and then measure the load standard (with calibration applied), the the VNA is expected to display S11=0, even if the true S11 is different. Btw, note that you'll never see S11=0 in practice, due to noise and uncorrectable residual errors. IMO you can interpret your green curve as S11=0 + some residual errors + noise.