As a side note: I don't think I need that high frequency response for the transformer is required...up to 1MHz signal will do to test compensation loops.
Beautiful! We have all the numbers we need to work with!
Suppose the transformer is 1:1 at 50 ohms. It should be made of 50 ohm transmission line, in maximum electrical length 1/8 wave at 1MHz. If velocity factor is 0.67, then 25 meters is the allowable length.
Which is a pretty damned good bit of wire!
It should be something like #34 AWG or finer magnet wire (cough*), twisted together. This won't be quite as low as 50 ohms, but it'll be close enough.
Good cores:
Hi-mu toroids, especially nanocrystalline (obtain from Mouser or Elna Magnetics?)
Silicon steel, with fine laminations (under 10 thou), GOSS or better
Pot cores (hi-mu ferrite, e.g. 3E27 or better)
Making a core size choice isn't obvious. You do have a known quantity of wire to wrap around the thing. You can probably get on the order of 500 turns around a 5uH/t^2 core. Which is 1.25H, which gives a -3dB cutoff at 6.4Hz. Not too bad. Actually half that, because you'll have a 50 ohm signal generator, acting in parallel with the 50 ohm secondary termination resistor. So 3.2Hz.
You should be able to obtain higher inductivity (>20uH/t^2) in the fancier steels (NiFe, permalloy) and amorphous and nanocrystalline materials. This would push the cutoff below 1Hz.
The winding should be bifilar, wound in layers (with tape between each), to prevent too much crosstalk from adjacent layers. That will help avoid weird dips and peaks in the response.
*Checking... you may actually want much heavier wire, since 25m of #34 is 21 ohms. And that's just for one winding! #28 is about the smallest you can get away with (< 3dB insertion loss), and #25 would be ideal...
Needless to say, larger wire will invite much larger cores and windings. A UR style core (the kind they make flyback transformers with) might be attractive, because of the wide cylindrical winding area available.
Needless to say, any excuse you can make to reduce the winding length will save on both resistance and HF response.
Tim