There's nothing about a BNC to N adapter that would intrinsically damage your SA. In fact, it's a very good idea to put some kind of adapter on the SA--even a N-N adapter (which some people call a "connector saver"), and leave it there for good, so any connections you make will be on the adapter. That way, if/when you manage to damage the threads, you just have to replace the adapter, and not repair the SA.
That said, it's worth the investment to get decent quality adapters. The difference is that the cheap adapters have imperfect tolerances on the threads, and sometimes rough threads, and can scratch the high quality N adapter on the SA. If you buy even good adapters on Ebay, you should try to look carefully at the pics for nicks or damaged threads, and look for some kind of return policy.
When going to be using the SA for an extended period, I leave a high quality 10 dB attenuator on the input. This not only protects the connector, but also protects the input against ESD damage.