Tektronix TDS3032 Boot Problem Challenge

[mikest]

 I have a Tek TDS3032B scope that is challenging me with a boot problem and I thought I would reach out and see if someone may have seen something similar in the past.

 This TDS3032B hangs at the splash screen on power on and If I leave it there for about 3-4 minutes and do a quick power cycle it boots normally and runs fine.  At first I thought it was a temperature problem but after considerable time with a heat gun and freeze I cannot change the fail patten at all.  Short answer is that the scope does not seem effected at temperature extremes. From the boot log (see below) it is consistently hanging during the code decompression. 

 Some comments on what's been looked at so far:

- MPC860 seems halted or idle, DRAM refresh is active, TA* goes inactive, no reset, NMI or interrupt that I can see.
- Power supplies look very good and rise quickly
- Clocks look fine.
- NVRAM / RTC has been upgraded
- Network works fine when scope is running
- Has the same problem on 3.35 firmware and after recent flash of 3.41.  Boot sector has never been re-flashed but that could be in the cards..
- When the scope is up and running it runs fine Diag will loop all day with no error, SPC always passes. It functions well and I cannot effect it with any temperature extremes or vibration.
- Later in the boot log I have seen a machine check on occasion but it handles that and recovers on its own and boots normally.

Its really a challenge with no schematics or code and my hat is off to you folks that have unmasked what you have in this forum its been very beneficial to me in getting this far!

Here is the boot log showing one hang up followed by one successful boot.  Note I have added some comments:

// First power up

UUUUUUUUUUUUUUUUUUUU
Boot flash checksum OK
Code flash checksum OK
0x00800000 bytes of RAM found
DRAM address line check OK
DRAM data line check OK
NVRAM address line check OK
NVRAM data line check OK
TBC ACQ SRAM address line check OK
TBC ACQ SRAM data line check OK
TBC Display SRAM address line check OK
TBC Display SRAM data line check OK
SuperI/O ID 0x000000b2
Loading Image
Uncompressing code from 0xffc40010 size 0x001e0927

// Hangs here at splash screen

// After 3-4 minutes power cycle to successful boot

UUUUUUUUUUUUUUUUUUUU
Boot flash checksum OK
Code flash checksum OK
0x00800000 bytes of RAM found
DRAM address line check OK
DRAM data line check OK
NVRAM address line check OK
NVRAM data line check OK
TBC ACQ SRAM address line check OK
TBC ACQ SRAM data line check OK
TBC Display SRAM address line check OK
TBC Display SRAM data line check OK
SuperI/O ID 0x000000b2
Loading Image
Uncompressing code from 0xffc40010 size 0x001e0927
Enabling cache
Entering usrRoot
Enabling the MMU
sysClkRateSet using 50 MHz
Enabling I/O system
excInit called
Class 17 Level 3  hwResetTBC(): Installing Machine Check Exception handler. 18:04:38 02/07/2024 (thread: StartUp)
TBC reset count = 2
Class 2 Level 3  acqWaitForAcqDone: timed out, sysjmp = 0x5
 18:04:39 02/07/2024 (thread: StartUp)
 Doubling core clock ... complete
 Attaching network interface qu0... done.
Attaching network interface lo0... done.
NFS client support not included.
Starting VXI-11 service...Core channel...Abort channel...ready.

Any thoughts or insight most welcomed!
Mike

[sicco]

The point where it fails is probably just a little later, when the CPU clock PLL switches from 25 to 50 MHz. Possibly the PLL in the XPC860 no longer locks with 50 MHz. As it’s intermittent, it may well be related to the 3V3 power rail dipping at that point, possibly a bad cap in the power 15 -> 5 or 5 -> 3.3 supplies. Try a steady 1+ amp capable 3.3V source on the 3V3 rail instead of the default TO220 LDO and see if it then always boots fine.

[mikest]

 Sicco, thanks so much for the suggestion!  I did scope the supplies but a short single shot event would be difficult for me to to see, they do ramp quickly.  It is intermittent but always boots successfully from power down wait two to three seconds and power on.  If I power off for ~5 minutes or more it will hang at code decompress.   Your thought on the PLL is really good and I will look at that in the morning when i get back out there, maybe interesting to take a look at the MPC860 datasheet to understand any special power treatment for the PLL clock gen.

   I really appreciate the considerable work you have done in helping us on the forum have a realistic chance at repairing the TDS3000 series through your tools and insight, your prior posts have helped me considerably.

[mikest]

 Putting a supply on the 3.3 rail did not change anything with regard to the fail pattern. Further look at the supplies did not reveal anything, they are very stable, within a few millivolts of target with very low noise.  I think Tek put some thought (and $)  into power design here.. I do think they followed MPC860 reference design for PLC power L/C filter isolation.  One question that is come up is the thought that the serial boot logging is synced (blocked I/O) to the boot tests, sending a line of boot log completely before the next test or setup begins. Do we know this is the case or is the output buffered and the tests / setup can be steps ahead of the boot log output?  Since there are not that many and the scope is 20 years old I am going to replace the electrolytics, a show of optimism for a eventual successful repair for sure..

 Dave on the TekScopes forum was suspicious of the DRAM and indicated it could be  an intermittent failure mode. He suggested further cold / hot testing and see if there are any effects on the hang-up pattern. Going to try that as well.
 
 As a side I am trying (early stages) to bring up BDM and it says my Mini Module is not compatible. I have the FTDI FT4232H Mini Module and it returns a device code of 0x07 which I think it does not like.  I have not gotten to looking at the code yet and assume the compatible devices is in a header file, when it goes through the four devices it states that none are compatible but states the FT4232H is one that its looking for.   Are there any command line arguments dealing with device type by chance?  Device type on the module cannot be changed via their programming tool.

[sicco]

I think the boot log serial output stream is buffered. So the next test already commences before the serial text line has made its way fully to the 38400 baud tx pins.
If the CPU then freezes, then we're not getting the full picture.

Zooming in on the PLL possibly (not) locking, there's a capacitor on the PCB bottom right under the XPC860 that has the PLL VCO voltage. See picture, red circle. Can you measure with a multimeter how many volts over that cap, both before and after it hangs. And when it does not hang? The CLKOUT pin should also be accessible I think, and show the frequency that the XPC860 runs at. I thin it also clocks the SDRAM.

On the BDM program with FTDI - send me a PM and I can share a more recent version that also has RAM testing, and can do -c models reflashing as well. Quite likely that the FT4232H works with that version.

The RTC does tick and maintains time OK?

 

[mikest]

 Thanks again sicco!  Think you maybe on to something that's not quite right with it.  Here is what I saw with the measurements you requested:

DVM across C655 :  Hung at Decompressing: 1.655v       Running after Successful boot: 1.659v   (also have GND referenced if helpful)

Frequency checked at suspected CLKOUT via under 860 and TP next to DRAM: 25mhz in both hung and running state

Shouldn't CLKOUT be running at 50mhz following boot?   I checked this a couple of weeks back and I thought I saw 50 but I don't think I checked in the hung state until you pointed it out.

I have never seen a issue with the displayed Date & time. I watched it for about and hour and it seems to be keeping accurate time.  Puzzling at 25mhz... Only had a chance to do one Hung / Running test cycle, will try to a couple more tonight. Thanks for the insight into the serial buffering, they really should block the I/O for something like this at boot time.. once its up and running it is fare game.

I was able to get the Mini Module to be accepted by the BDM, will send a PM.

[inse]

There is a dedicated RTC so the displayed time would not be influenced by an unstable master clock.

[pschirrer]

Hi Guys,
I also have a dead TDS3014B with only the splash screen showing. The channel relays do not click either following boot. The keys on the front are inactive (none of the LEDs light up).
The 5V, -5V and 3.3V supplies seem fine. Both crystals are working (48MHz and 75.75MHz).
Where can you get the boot log at (UART ? Connector ? Pad on the PCB ?).

Any pointer welcome.

Regards

[sicco]

https://www.eevblog.com/forum/repair/tektronix-tds3012b-repair-help-needed/msg3329696/#msg3329696 reply#3.