[Help] Symmetricom SyncServer s200 tracing & troubleshooting

[AlexJackson]

Last week (Aug 2) I bought a "as-is for parts/repair" SyncServer s200 off the usual marketplace for relatively cheap. Model was a base model with no OCXO and an old gps module. I have a handful of parts from a previous s200 that died that I kept. Unfortunately, I'm not anywhere near a "pro" at fixing things especially computer related. I can chase my way around a board a little bit with a meter to check for fuses and easy stuff but im certainly not an expert.

For the first few days, this one would lock up, and the status lights would glow red. power cycling would not fix it. Some screwing around led me to messing around with the cover off and it powered on. On the mainboard there is a handful of status LEDs. I don't ever remember a particular internal LED being lit on my previous unit (labelled DS2). So I messed around with the PSU wires and seen the LED turn off when messing with the wires. After a few minutes, it would boot ok. Subsequently it might be fine for 30 minutes, or several hours in which it would randomly lock up and id have to repeat the wire wiggle procedure. It locked up 7 times in a few days requiring me messing with it.

The s200 can be easily modified to the s250 model by adding bnc jacks to the board footprints (the signals are already there) and using a few commands to edit the eeprom to show model 250, and set the oscillator as OCXO. It is also trivial to physically add the 10MHz OCXO and replacing the bad gps module. The whole walk-through can be found in the metrology section here on EEVBlog forums. While having the board out, I re-flowed the power connector with my iron just in case. I also disk imaged the disk. The system booted up fine, checked the options & settings and all was good... Until another lockup a few hours later, and red status lights. Ive read elsewhere that a lockup and the front leds turning red typically means a failing power supply. So spent the remainder of the week (aug 5 - aug 9) fiddling with the wires to get it to work with the thought "ill mess with it this coming weekend when i have all my stuff and more time".

Fast forward to Thursday of that week (aug . There were no hanging/crashing... Great. Friday also no hanging/crashing. Great! I can reassemble it and install it in the rack! Its polling ntp, locked on 8-11 satellites and locked onto the time and serving ntp requests without issue.

Come Saturday. Let me try updgrading the OS again (from 1.10 to 1.36) since its always failed on my old long-gone syncserver. After an upgrade, the system resets settings to default. You can use the keypad on the front to change LAN1's address assignments. Turns out the keypad doesnt work anymore along with other weirdness. So since I have my stuff I check power 12v (yellow) is ok, 5v (red) is ok, and the -12v (blue) is ok, if a little 'high' (-13.1v). At this time the DS2 LED is lit, but the system is "working" and not locked up. Also noticed another internal LED thats usually lit, wasnt lit (PLL Lock). So I probed around with my meter. There are 2 regulators (both labelled p565d2T) on the front right (U1 & U3). I had 5v on the inputs & 3.3v & 2.5v on the outputs respectively. The only thing I "found" was a set of headers (P14) near the gps module that are labelled 12v, 5v, 3.3v, 2.5v, 1.8v, & 1.2v. 1.8 & 1.2 are lacking any voltage on them. I dont know if these are related to the unpopulated parts on the board. Probing those pads with my meter gave no continuity either. Seen nothing in the logs you get access to in the UI related to no timing sources and typical post boot messages.

I went as far as restoring the disk image of the OS before making this last set of changes (1.10) and the system still behaves the same. HOWEVER with issues.
• Keypad doesnt work
• Internal PLL LED is not lit (it used to be)
• Internal DS2 LED is lit (used to only be lit when locked up)
• System information page in the webUI said "No sync source", all 4 inputs are "Locked" Oscillator type is "unknown", hardware clock time is "Wed Dec 31 19:00:20 1969" & hardware clock "unlocked"
• ALARM front panel LED is green (should be red with no sync source)
• NTPD still works
• WebUI still works
• GPS not working
• SYNC light is red (should be red with no sync source, green when locked to pps, gps, irig, 10mhz, orange when synced with ntp. When SYNC is red OR orange, ALARM should be red)

Ive since went to version 1.36 and NTP syncs waaaaay the hell faster and the UI is much more responsive than 1.10 but still exhibits the behaviors above.

When left in this state, it does get an NTP sync to symmetricoms & time.nist servers and the SYNC light turns orange. I did some basic probing with the scope. OXCO is outputting 10.00000MHz according to the counter using my DS1054z. I did find a number of test points that are 2.5v & 2.8v in the middle of the board. Id have to remove the board from the chassis to probe the stuff on the bottom. I suspect the power supply may have killed something somewhere, but if anyone has any ideas where I can check next I'm all ears. There are a crap ton of test points scattered around the board but ive been unable to find a service manual and im not the best person with a scope and certainly not in the industry but im certainly willing to try. Sadly my SMD experience is a lot to be desired. I also lack a hot air station.

Tips, places to look, or something?

[aeg]

Maybe the 1.36 upgrade hosed the EEPROM. Do you have an EEPROM backup from before the upgrade?

[AlexJackson]

Ohh! I didn't think about that. No I do not have an original backup of that eeprom, but I do have my previous eeprom from my old syncserver. Swapped it out (luckily they're socketed ATMEL 93C46, 8 pin dip) and it didnt make any difference but it did remind me of a couple other things to try.

Allegedly when you do an OS update on the disk it checks for which oscillator is used and sets the new value into the eeprom, and sets it to the appropriate oscillator. I'm also running an aftermarket replacement receiver and not the stock one. So I had a thought what if I did the upgrade and it didnt recognize the new gps module, and for some reason didnt see the OXCO. Maybe it borked there? However, SSH'ing into the server and getting to a command prompt and running the program that can set values to the eeprom, my original settings and data is still there. There is also a program that can read/write to the FPGA and trying to read the date from the FPGA gives me the 1969 date (ntpd still has the correct time/date). Regardless, I re-imaged my OG OS back onto the disk, and reinstalled the oem gps, and did the OS upgrade cycle again to see if it would redo the eeprom setting. Still no change to the behavior.

This leads me to the thought that the FPGA possibly has ok power, but isn't somehow getting the data from the rest of the system (bad trace, regulator or other component?) and the webUI cant cope with some hardware weirdness thus cant pull any of the information from the FPGA. The newer OS/webUI seems to handle the issues much better than the older versions. Sadly, the FPGA is on the bottom of the board so I will have to dismantle the system again to get to it. Or maybe the upgrade hosed the eeprom that handles the FPGA program? (U40 holds model number, oscillator type, etc... U34 I dont know)