Uh huh, not too sure about what the problem is exactly either.
As an obvious side note, you could just have two separate output drivers, one differential, the other not (and possibly with higher current for the single-ended one if I got it correctly), both fed by the same digital input signal? They could go to separate connectors as well, as using a single connector for both modes could be confusing for the end-user.
Except if your design is ultra cost-constrained, really... I wouldn't bother doing it another way. Especially if you intend on driving standard RS-485 receivers, which may be what you're implying with "proper differential receiver". Better use standard drivers so they are conforming.
Just a thought, we don't know squat about your application, so...