Thats a reasonable perspective, you can always use a limiting amplifier to remove AM, no so with PN.
Remember that both AM and PM are small at any reasonable offset, so for the purpose of first order analysis you can consider whichever one you don't care about to be zero providing your detection arrangements do not gratuitously amplify whichever one you are trying to cancel.
Consider, that while the measured amplitude of the PN in a quadrature measurement depends on the level of the measured signal, and so is really the product PN * (1 + AN) where the nominal carrier level is 1, AN the amplitude noise component will generally be very much smaller then the carrier level, so unless you have a spur at a really silly level or something the AN component can be ignored, similarly the in phase measurement of amplitude noise is really AN * (1 + PN) where providing PN is much smaller then 1 it can be ignored.
Regards, Dan.