Re spectrum analyser: Might be useful to make friends with someone at a local Uni EE dept. or a radio-comms service centre or avionics tech at an airport or a tech at a local radio/tv transmitting station. Any of these guys will be more than happy to help out someone who has built something from scratch, and might even give you some free stuff!.
Re the note on your schematic, top right, "measure power here" , you can't actually measure "power" without changing the power at that point, you can measure voltage however, so from
http://www.minicircuits.com/app/AN40-012.pdf we can see that 0dbm=224mV , and -10dBm=71mV
Lot of useful stuff here:
http://www.minicircuits.com/applications/applications_notes.htmlAlso note the MAR-02SM is the same as MSA-0286
http://www.minicircuits.com/pdfs/MAR-2SM+.pdf The AD9581 datasheet says nominal output is 10mA into 200ohm , that is peak current, so RMS is 7.07mA, I^2R says that's 10mW = 10dbm (and 1.414V RMS = 2v peak = 4vpk-pk)
If you use both outputs and a transformer, you get 20mW out of transformer , half of this goes into your sourcing resistor R24 = 50ohm, and the remaining 10mW (10dbm) goes down the line.
Your power splitter R43 R44 (each 50ohm) loses 6dB so 4dBm (.35v) goes one way and 4dbm (.35v) the other .
If you allow, say, 2db loss across the transformer and filter, then its only 2dBm at this point , so you could in theory replace the amps with a piece of wire and follow with a 2dB attenuator.
Note that without the amp, the voltage on one output port depends on whether the other port is open or has 50ohms load.
So available power with ATTA = 0dB is 2dBm at DDS_OUT_A
So to setup the levels we need to work backwards, for an -02 amp the 1db compression point is ~4.5dBm , so lets say we use 3dBm at this point, and follow with a 3dB pad (this value pad ensures reasonable tolerance to mismatched loads) to give 0dB out the connector. Given the -02 has 3dBm at it's output, and 13dB gain, the input needs therefore to be -10dBm , so ATTA needs to be -12dB.
You won't get 0dB across the full frequency band, the lower frequency -1db point is determined by your transformer , and the upper frequency by the filter.
Note also the values for the filter are tied to the reference clock, so if you change from 180MHz, you need different filters.
Note there isn't much headroom on the AD9851 output with 200ohm load and 10mA out, so you might need to mark the pot at 3900ohms.
(Also you might consider using a DAC or an electronic pot here then you can do software level calibration vs frequency , bit of an overkill really)