Tektronix MDO3014 failure, MDO3000 series

[GridWork]

So 5 years ago I took some hard earned money an plunked it down on a brand spanking new Tektronix MDO3014 scope plus the logic analyzer expansion. Great running scope, did everything I needed it to until this last week. Fast forward to current time, the scope would just keep resetting. Turn it on, it would run for a few minutes, then act like it got a power cycle. First things first, check prime power, yup that's good, stable and nothing else plugged it has issues. Next, I open the unit to check the 12V power supply, check, it's good (even monitored the output with another scope for glitches). I start checking other test points and find that during the reset the lower voltage outputs from the mezzanine board are also dropping out. It is unclear if that's normal during a power cycle, or possibly the cause. I investigate for hot components and can't find anything. There have been no error messages prior to a restart, just bam reset.

I contacted Tek tech support, they recommend upgrading the firmware (I was running 1.18 just fine) and sure enough during the firmware update, RESET! So now the scope has a discombobulated front panel microcontroller (read: buttons, knobs and lights don't work anymore).

Next step, request a service quote for getting it back up and running. Tektronix wants $2710 plus shipping to repair the scope. 5 years old, 1424 hours on it and it's essentially garbage. 

Has anyone else had experience repairing the original problem of the random resets? Not really sure where to start looking yet. I think that the main acquisition board is ok, when it ran it had no problems getting a signal, and it was nice and clean. Nothing hot that I could tell. Any help out there?

[darkstar49]

Gee... I'd kill the one that advised a firmware update on a scope that does reset randomly !!!!!!! 

And 3K is the price for an MDO3014 with 3 years warranty (and fresh cal) at a major broker here in Europe... Tek's proposal is plain unacceptable, because even if you have it repaired, you'll still be left with a scope without warranty whatsoever, excepted on the repair.

At this stage, you're correct... if you don't manage to fix it yourself, you're left with a piece of electronics garbage...

[GridWork]

If I could just get the code back and stable on the front microcontroller, I might be able to troubleshoot what's going on with the rest of the thing. It sure puts the cost of ownership up there. Right now I think it is setting at around $1338 per year. I deliberately went for the Tektronix because I have had many in my professional career that are 20 years old and still functional. Even my TDS210 that was purchased new in 1998 still works.

I only did the firmware update because it was running stable for enough time to give me some confidence. The repair cost would have been the same either way.

[kcbrown]

This sounds like very poor design on the part of Tektronix.  It should never be possible to brick an expensive device like this through an incomplete firmware update.  It should have a bootstrap algorithm in ROM that performs a checksum verification against the flash and boots from an earlier version of the code that is also in ROM if the checksum fails.

The "service manual" for this scope is laughably worthless, not even worthy of its title.

[darkstar49]

wasn't there a way to have a 'forced update', even if the scope was unresponsive, by putting some file in the root directory, or renaming the update file ??

I can't imagine Tek has to replace h/w because the update went wrong... I'm pretty sure there's a way (although maybe not public) to push an update on a bricked scope...

[GridWork]

Since the cost of repair is so high, and fixed regardless of the state of the equipment, I'm going to give it a shot this weekend. I will try and post status updates if I get anywhere.

[darkstar49]

Since the cost of repair is so high, and fixed regardless of the state of the equipment, I'm going to give it a shot this weekend. I will try and post status updates if I get anywhere.

have you tried putting a file named "forceupgrade.txt" in the root of the USB drive ? (content doesn't matter, can be empty, or maybe must be empty, try both...)

[GridWork]

Since the cost of repair is so high, and fixed regardless of the state of the equipment, I'm going to give it a shot this weekend. I will try and post status updates if I get anywhere.

have you tried putting a file named "forceupgrade.txt" in the root of the USB drive ? (content doesn't matter, can be empty, or maybe must be empty, try both...)

Just saw that in the firmware readme. I have tried it, and the front panel microcontroller is still shut down. It did get through the firmware with a green check mark at least.

[GridWork]

Scope is apart for troubleshooting.


So far I've been able to see the voltage feeding the CPU board dropout, it mimics the Power ON/OFF button behavior. The firmware on the front panel microcontroller is still AWOL, multiple attempts to force an update have been unsuccessful. I have tracked down the power supply remote ON/OFF


The now brainless microcontroller.


I've tried to Telnet/SSH into the scope, but no real responses. The Web interface does work, however it only shows two channels.
<EDIT> The screen capture shows all four channels, had to reseat the front end board and all worked ok.

[darkstar49]

It won't solve your power issues, but at this stage, I'd already get in touch again with Tek, saying this upgrade was done upon their advice, and that you're now at least at a point where you could have the forced upgrade go through, although without success for the front-panel controller (the Freescale MCU on the 3rd pic). Maybe they could already give you some quote for fixing that part, don't know whether service centres are able to, but there surely is a way (JTAG...) to flash that MCU specifically, without swapping the board.

I'm really not a fan of Tek's approach to have a 'dumb' PSU delivering 12V, and provide all other voltages on the mainboard itself... had similar issues on a DPO4034 a while ago, and had to sell it pretty cheap after numerous (and unsuccessful) hours of troubleshooting... (have to say I'm not very at ease with such problems, being more on the software side).
 

[m k]

The Web interface does work, however it only shows two channels.

What else it says, right or wrong model?

Is it old or new firmware?

[GridWork]

It won't solve your power issues, but at this stage, I'd already get in touch again with Tek, saying this upgrade was done upon their advice, and that you're now at least at a point where you could have the forced upgrade go through, although without success for the front-panel controller (the Freescale MCU on the 3rd pic). Maybe they could already give you some quote for fixing that part, don't know whether service centres are able to, but there surely is a way (JTAG...) to flash that MCU specifically, without swapping the board.

I'm really not a fan of Tek's approach to have a 'dumb' PSU delivering 12V, and provide all other voltages on the mainboard itself... had similar issues on a DPO4034 a while ago, and had to sell it pretty cheap after numerous (and unsuccessful) hours of troubleshooting... (have to say I'm not very at ease with such problems, being more on the software side).
 

The power issue is definitely not from the power supply, it stays rock steady at 12V during the reset event. Having a dumb power supply isn't so bad, it theoretically allows easy future replacement for what is likely to break during normal use. Power lines can get some crazy energies on them.

Tek isn't budging on the price to repair much. The only other option I was given was $230 an hour for a technician to troubleshoot + a mandatory $650 calibration fee, not much better. And to add insult to injury, the repair only has a 90 day warranty in either case. <GRRR>

[GridWork]

The firmware is whichever version got loaded using the "forceinstall.txt" option.

If anyone out there has an MDO3000 series and happens to have a Freescale debug module, would you be willing to download the code from the microcontroller so that I can try and re-flash mine?

One of the other things I am noticing is that the quality of the assembly is not up to par for a mid level (and in this price range) oscilloscope. The hand soldered point do not have solder filling the hole, just a light surface fillet on the back side.

[GridWork]

New update:
I spent most of the weekend trying to trouble shoot what's going on. So far here is what I know:

• The main application processor is being reset due to a global reset from the supervisor circuit.
• The supervisor is causing a reset because main voltage is dropping out.
• The voltage dropout is due to the power controllers on the mezzanine board shutting down.
• I am trying to route out where the shut down is coming from. The 1.8V controller "RUN" comes from the power good on the 5.0VD controller.
• It appears that something on the main board tells the AUX DC/DC controllers to start up, not sure where that is yet.

I will try and get some pictures posted of the board and attach to this message.

[kcbrown]

New update:
I spent most of the weekend trying to trouble shoot what's going on. So far here is what I know:

• The main application processor is being reset due to a global reset from the supervisor circuit.
• The supervisor is causing a reset because main voltage is dropping out.
• The voltage dropout is due to the power controllers on the mezzanine board shutting down.
• I am trying to route out where the shut down is coming from. The 1.8V controller "RUN" comes from the power good on the 5.0VD controller.
• It appears that something on the main board tells the AUX DC/DC controllers to start up, not sure where that is yet.

I will try and get some pictures posted of the board and attach to this message.

This is actually good news if I'm reading it right.  You might not need to re-flash the front panel controller after all, if it's being reset in the way you suggest.  Smells like a hardware fault.

The very first thing I'd look at is the actual voltages going into and out of the various voltage controllers, see how they vary over time.

[GridWork]

New update:
I spent most of the weekend trying to trouble shoot what's going on. So far here is what I know:

• The main application processor is being reset due to a global reset from the supervisor circuit.
• The supervisor is causing a reset because main voltage is dropping out.
• The voltage dropout is due to the power controllers on the mezzanine board shutting down.
• I am trying to route out where the shut down is coming from. The 1.8V controller "RUN" comes from the power good on the 5.0VD controller.
• It appears that something on the main board tells the AUX DC/DC controllers to start up, not sure where that is yet.

I will try and get some pictures posted of the board and attach to this message.

This is actually good news if I'm reading it right.  You might not need to re-flash the front panel controller after all, if it's being reset in the way you suggest.  Smells like a hardware fault.

The very first thing I'd look at is the actual voltages going into and out of the various voltage controllers, see how they vary over time.

I've been looking at the voltages on all of the DC/DC outputs (and the 12V power supply output), there doesn't appear to be any increased current draw prior to shutdown. There are two dual channel Linear controller chips responsible for the 5.0VD, 3.3V and 1.8V, they are being commanded off from a discrete FET pair (on the backside of the board), but I'm not sure what from. The front panel micro appears to be dead, i.e. no code. It was working prior to the re-flash even when the intermittent reset started.

This scope has been sitting on my desk for the last 5 years, and actually un-plugged during a good chunk due to lack of use.

One of the really frustrating things is the small components, traces and vias. For as much space as appears on the boards, there is no reason to use the grain of sand parts that they have. 0603's would make for something much more repairable.

[kcbrown]

I've been looking at the voltages on all of the DC/DC outputs (and the 12V power supply output), there doesn't appear to be any increased current draw prior to shutdown. There are two dual channel Linear controller chips responsible for the 5.0VD, 3.3V and 1.8V, they are being commanded off from a discrete FET pair (on the backside of the board), but I'm not sure what from. The front panel micro appears to be dead, i.e. no code. It was working prior to the re-flash even when the intermittent reset started.

This scope has been sitting on my desk for the last 5 years, and actually un-plugged during a good chunk due to lack of use.

One of the really frustrating things is the small components, traces and vias. For as much space as appears on the boards, there is no reason to use the grain of sand parts that they have. 0603's would make for something much more repairable.

It's probably a bill of materials thing.  The front-end likely needs the grain-of-sand parts because (I would expect) they have less in the way of parasitic characteristics, and since they're already specifying those parts, they may as well use the same types for the rest of the scope.  That would also simplify the pick and place requirements since fewer strips of parts would be needed to feed the pick and place machines.

You looked at the DC/DC outputs.  What of their inputs?   If you can identify which one (if any) is getting a failed input, that might lead you down the right path.  If you've got a cracked trace or something, you might be up against a thermally-generated failure and the only way you'd be able to see that is by looking at the inputs right at the voltage controller.  But if the voltage regulators are also controlled by a FET pair, and you've already identified the signal from the FET pair as causing the voltage controllers to shut off, then there's nowhere to go except back to the source that feeds the FETs.

If you've got a microscope, you should use it to look for cracked solder joints and other types of damage that could be responsible.

Sounds like tracing these things out is a real pain.   

Do you have close-up photos of the area around the FETs (both the side where the FETs are located and the other side)?  It might be helpful for us to see what you're up against here ...

[GridWork]

I've been looking at the voltages on all of the DC/DC outputs (and the 12V power supply output), there doesn't appear to be any increased current draw prior to shutdown. There are two dual channel Linear controller chips responsible for the 5.0VD, 3.3V and 1.8V, they are being commanded off from a discrete FET pair (on the backside of the board), but I'm not sure what from. The front panel micro appears to be dead, i.e. no code. It was working prior to the re-flash even when the intermittent reset started.

This scope has been sitting on my desk for the last 5 years, and actually un-plugged during a good chunk due to lack of use.

One of the really frustrating things is the small components, traces and vias. For as much space as appears on the boards, there is no reason to use the grain of sand parts that they have. 0603's would make for something much more repairable.

It's probably a bill of materials thing.  The front-end likely needs the grain-of-sand parts because (I would expect) they have less in the way of parasitic characteristics, and since they're already specifying those parts, they may as well use the same types for the rest of the scope.  That would also simplify the pick and place requirements since fewer strips of parts would be needed to feed the pick and place machines.

You looked at the DC/DC outputs.  What of their inputs?   If you can identify which one (if any) is getting a failed input, that might lead you down the right path.  If you've got a cracked trace or something, you might be up against a thermally-generated failure and the only way you'd be able to see that is by looking at the inputs right at the voltage controller.  But if the voltage regulators are also controlled by a FET pair, and you've already identified the signal from the FET pair as causing the voltage controllers to shut off, then there's nowhere to go except back to the source that feeds the FETs.

If you've got a microscope, you should use it to look for cracked solder joints and other types of damage that could be responsible.

Sounds like tracing these things out is a real pain.   

Do you have close-up photos of the area around the FETs (both the side where the FETs are located and the other side)?  It might be helpful for us to see what you're up against here ...

Pictures inbound:
MDO3000 Mezzanine board
TOP

BOTTOM

LTC3850

So what I have found is that the LTC3850 on the Left supplys the 1.8V and 5.0V Digital. The LTC3850 on the right supplies the 5.0V Analog. The LTC3850 on the right PGOOD output feeds the RUN for the one on the left. The 5.0VA (right) is the controller that shuts down first. The RUN1 on it gets commanded off from Q305 (back side, bottom right). Now I've got to figure out what drives it.

[kcbrown]

So what I have found is that the LTC3850 on the Left supplys the 1.8V and 5.0V Digital. The LTC3850 on the right supplies the 5.0V Analog. The LTC3850 on the right PGOOD output feeds the RUN for the one on the left. The 5.0VA (right) is the controller that shuts down first. The RUN1 on it gets commanded off from Q305 (back side, bottom right). Now I've got to figure out what drives it.

If I'm reading it right, the code on those transistors is I09, which makes them an SST109, an N-channel JFET (see https://www.sphere.bc.ca/download/smd-codebook.pdf).  The gate is on pin 3, the single pin on the left side in the photo.  That connects to R321, which connects to the via just next to the lower left corner of Q305.  That via dives through to the other side, but the other side is unpopulated.  In fact, the way the other side is unpopulated, and the way these components seem to connect to the unpopulated side, makes me wonder if this is the right place to be looking.   This is really weird.

Do you have a macro lens or something that you can use to get close-in shots of these two areas?

Back side:



Front side:



I've also attached a horizontally flipped version of the front side.  I've tried to match the boundaries and resolution as much as possible so that you can overlay one on top of the other to make it easy to see where any given via goes through and connects to:



The problem is that the original images aren't quite high enough in resolution for me to be able to clearly make out the traces and where they go.  Additional side lighting from the bottom right might help with that.

[GridWork]

So what I have found is that the LTC3850 on the Left supplys the 1.8V and 5.0V Digital. The LTC3850 on the right supplies the 5.0V Analog. The LTC3850 on the right PGOOD output feeds the RUN for the one on the left. The 5.0VA (right) is the controller that shuts down first. The RUN1 on it gets commanded off from Q305 (back side, bottom right). Now I've got to figure out what drives it.

If I'm reading it right, the code on those transistors is I09, which makes them an SST109, an N-channel JFET (see https://www.sphere.bc.ca/download/smd-codebook.pdf).  The gate is on pin 3, the single pin on the left side in the photo.  That connects to R321, which connects to the via just next to the lower left corner of Q305.  That via dives through to the other side, but the other side is unpopulated.  In fact, the way the other side is unpopulated, and the way these components seem to connect to the unpopulated side, makes me wonder if this is the right place to be looking.   This is really weird.

Do you have a macro lens or something that you can use to get close-in shots of these two areas?

Back side:



Front side:



I've also attached a horizontally flipped version of the front side.  I've tried to match the boundaries and resolution as much as possible so that you can overlay one on top of the other to make it easy to see where any given via goes through and connects to:



The problem is that the original images aren't quite high enough in resolution for me to be able to clearly make out the traces and where they go.  Additional side lighting from the bottom right might help with that.

I do have a macro and have gotten some more detailed pictures, just ran out of time getting some update here. I think I may have mistyped the transistor, I think that some how Q304 is the one initiating the shut down. It's the only active device I have found connected to the "RUN" pin on the 5.0V DCDC. All of the other components attached to the "RUN" are anodes from various diodes on the board.

There may be another development here on the scope. The tech support individual I've been e-mailing mentioned a service note "CS-SUP-0110" about scopes randomly powering down. I have to see if the RMA department changes this from a customer funded repair to something different.

[gord]

I have a 5 year old MDO3000 too. It has had very little usage, and in recent weeks I have noticed it resetting. In my case I have only seen this happen during startup. The display is blank when normally the splash screen should be displayed, and it power-cycles, and this happens several times until it manages to fully start up, and it then reports no errors.
There was an older firmware release note that discusses a fix which restarts the MDO3000 if the display fails to start up, but I do not know if this is what is happening, or if the MDO3000 software would restart everything if the display does not respond after startup as well. This could all be unrelated and I may have jumped to the wrong conclusion.

[GridWork]

I have a 5 year old MDO3000 too. It has had very little usage, and in recent weeks I have noticed it resetting. In my case I have only seen this happen during startup. The display is blank when normally the splash screen should be displayed, and it power-cycles, and this happens several times until it manages to fully start up, and it then reports no errors.
There was an older firmware release note that discusses a fix which restarts the MDO3000 if the display fails to start up, but I do not know if this is what is happening, or if the MDO3000 software would restart everything if the display does not respond after startup as well. This could all be unrelated and I may have jumped to the wrong conclusion.

It sounds similar to what was happening to mine, however I get the splash screen pretty much every time. Sometimes it just seems to hang there, sometimes it will go through the boot. It doesn't show that anything is wrong in the POST results. My issue got compounded when it did one of these resets during an update to the latest (1.30) firmware. That reset corrupted the code in the front panel microcontroller. Even with the latest firmware it still does the restarts.

[m k]

Is it staying up in the cold?

How far back Tek is supporting?
Link seems to have MDO3000 Series_v1.10

http://twintech21.cafe24.com/v3data/1279

[GridWork]

Is it staying up in the cold?

How far back Tek is supporting?
Link seems to have MDO3000 Series_v1.10

http://twintech21.cafe24.com/v3data/1279

Not sure what you mean by staying up in the cold? The most recent firmware is v1.30, which didn't fix the problem.

[m k]

If you have a place where the machine can be in low ambient temperature maybe it is not resetting so eagerly.

Then if you downgrade far far away back maybe your corrupted part gets some new old code.
Assumpton there is that upgrade is upgrading only those parts that are included in that upgrade.

Are you back in square one but with new firmware?