Actually I cannot use 5W5 or 9W4 connectors. The D-Sub female connector is already fixed. This is on an Anritsu MG3700 Vector signal generator. The only way to get the IQ signals out is through a Female D-Sub connector 15 pin. So I need to solder a Male D-Sub to connect to that.
It
uses differential output, so I think it needs to use balun transformer to convert differential output to unbalanced coax cable...
So, you're needs to solder balun transformer wires to D-Sub connector and then solder coax cable to transformer output. And all wires needs to be short enough to not add impedance mismatch.
If you have broken Anritsu Dsub to BNC IQ converter, you can disassemble it and see how it works. I bet it uses transformers, because it's the cheapest solution, so you can take transformers from broken converter and use it in your converter.
For low frequency it may need to use active circuit with operational amplifier to convert differential line to unbalanced coax, because transformers don't works good on low frequency.