Except that when using the antenna analyzer, I made no changes to any SA setting.
As far as what the SA is reading, thinking about it, it's all good. The SA knows what the attenuation is set to, so can always display the power being applied to the input port.
I did test my external attenuator and it is indeed bad.
The only potential issue left is why when using the antenna analyzer did I initially get several high SWR readings, when I tried the Smith chart I got the overload warning, then things worked fine. I did not change any SA settings during these tests with the antenna analyzer.
The SA still acts odd sometimes, so I think I will still replace those switches.