Nah, frequency accuracy/resolution/rbw isn't the limiting factor these days, here's the problem:
* SA wants filters
* VNA doesn't need filters
* Filters are unstable, reflective, absorptive, and expensive
Both get filters: YIG filters are wildly unstable by VNA standards and mashing together a traditional YIG-based SA with reflectometers to achieve VNA capability would end in disaster. Just imagine transplanting the multi-dB amplitude accuracy spec from a YIG SA to a VNA. Yikes! High-IF architecture and filter banks make this much less of a problem, taking it from "disaster" to "tradeoff," but it's still there.
Neither get filters: Ok, the VNA doesn't want filters and doesn't need filters, what about the SA? Can it live without them? Sure, but now you have a ZIF SA with image rejection so bad that the phantom signals take up 80% of your graticule or SIS that makes wideband signals look like bathtubs at completely the wrong levels and hides intermittent signals. Not great.
-------------------
Life is tradeoffs. What are the options?
Separate SA and VNA: Best of both worlds, but you pay for it in money, space, weight, and convenience.
One port gets filters and the rest don't: This is the compromise that most combined SA/VNA units seem to strike. The ports aren't identical anymore, but if you need SA you use the SA port and for VNA you often have a situation where one reflection is less important and you can stick that on the SA port and live with the slightly degraded performance. It's pretty ok, all things considered.
No ports get filters: You see this on high-end VNAs because it is assumed that any engineer using one is a big boy/girl and understands how ZIF/SIS works and what the resulting "Spectrum Analyzer" can and can't do.