The phase difference plot is quite interesting, you seem to have 20-30 picosecond "jumps" that gradually get worse the further the measurement goes. In the beginning they're few and far apart, they gradually get more frequent until they dominate the plot. That doesn't look like flicker noise. It also doesn't look like the noise plots Corby has posted here.
For your measurement setup, I'd not try to exactly phase align the start and stop. How about going for a nice 90° phase offset?
Yup I saw those needle like jumps also. However there could be a ton of reasons for these, I don't know yet where I should start looking!
These spikes could come from:
a) the HP 5335A. I have repaired it once. So far I didn't have any problems with it, but maybe it has an instability which is just so small that I didn't notice it so far?
b) I have not locked the 10 MHz references of my test instruments together, this means: the two signal generators and the counter do not operate on the same reference. Could this make a difference?
c) I guess the HP 8341A has not a very good stability. Even though the data sheet of this signal generator doesn't specify stability, it specifies phase noise, and this is not very good. Maybe my 8341A is unstable? but again, I thought at least for the noise floor test this should really not matter!
d) maybe it was not a good idea to use switching regulators for the supply rails? I use TRACO TDN5 WI series, they operate at >100kHz and I have added an output filter with ceramic capacitors and inductors. I have tested the output voltages with a HP 34401A, 15.02V and -15.01V and 5.01V. That looks indeed really good.
I don't think there is a problem with the circuit itself, but it is rather a malfunction of some kind I'd say. If I compare my circuit to Corby's, the difference is in the input stage (MC1650 vs. more modern ECL gate MC100EL16). The OpAmp I use (LT1007) has lower noise; besides that I also use the LM311.
I have AC coupled my mixers. Could that make a difference? I used ceramic caps for that. Will try to remove them tomorrow and replace with shorts. However I don't really like the DC current through the mixers! Also my coupling caps to the ECL gates are ceramic. Could this be a problem? it is X7R 100nF. Shall I test C0G?
The fact that the spikes are rare at the beginning and become more frequent the longer I measure indicates that they could be related to some sort of thermal problem. Something in the circuit is cold at the beginning, and as it warms up, it produces the spikes.
I see that the DC/DC converter for the 5V rail is quite warm, even though it is specified for 1A output but it currently needs to deliver only 300mA. Maybe this could be a culprit. On the other hand it has an efficiency of 80%, so it dissipates around 0.5W, should get a bit warm I guess.
I'll try to measure the power supply rails tomorrow with the scope and see whether there is a lot of dirt. And also check without the AC coupled mixers. At the moment I am quite disappointed as I expected much better performance for this effort!
I bet it's the powersupply. Hopefully. This would be a place which would be easy to fix :-)
I'll post my timelab files tomorrow.