Tektronix MDO3014 failure, MDO3000 series

[GridWork]

If you have a place where the machine can be in low ambient temperature maybe it is not resetting so eagerly.

Then if you downgrade far far away back maybe your corrupted part gets some new old code.
Assumpton there is that upgrade is upgrading only those parts that are included in that upgrade.

Are you back in square one but with new firmware?

Helpful hint: For Tektronix scopes if you create a "forceinstall.txt" file on the USB flash device containing the .img file, then the scope will do an upgrade, regardless if the version is newer or not.

I did this rolled it back to 1.18, and the front panel micro is still codeless. Also tried 1.26 and 1.30 (the newest).  Makes sense as there should be boot code to tell the microcontroller how to deal with the incoming code. If that got corrupted during the reset, then it wouldn't know what to do. Only the programming port or JTAG port would be able to load new code.

I'm still not sure if it's a thermal issue entirely. Since I cannot get the firmware to re-flash the front panel micro, the scope will either get the recycle bin, or sent back to Tektronix to get rejuvenated.

[m k]

If you like somebody here will buy it from you for sure.

I don't know nothing about your specific device but don't let it disturb you.
Put device specifics aside for a moment.

Do you know what is in that image file, do you know that front panel code is there?
It's a good practice to upgrade all but do you know it.

I understand that bootloader is a special thing and is not changed during normal write operation.

Is the upgrade a script driven, can you edit it?
If you can find a spot where front panel is accessed you can add a flag how it is done, like is it skipped or actually written.
Maybe it's even saving write cycles and reads first, if that fails there should be a note somewhere.

[GridWork]

So to add an update to the MDO saga, the tough decision was made to send the scope back to Tektronix for evaluation. There was apparently no way (for me) to get the code back on the front panel microcontroller.

To answer a previous question, the firmware updates are a .img file about 40MB for early versions and 60+MB for later versions. I tried loading several different versions (at least Tek makes them available, not just the latest), none would get the front panel working.

Replacing the scope with anything equivalent would end up costing the same or more than the repair, so I'm going to see if Tektronix sees this as a manufacturing defect or customer responsibility.

Thanks to all that had suggestions and put some time in trying to help!

[kcbrown]

So to add an update to the MDO saga, the tough decision was made to send the scope back to Tektronix for evaluation. There was apparently no way (for me) to get the code back on the front panel microcontroller.

Did they manage to do anything for you on this?

[GridWork]

So to add an update to the MDO saga, the tough decision was made to send the scope back to Tektronix for evaluation. There was apparently no way (for me) to get the code back on the front panel microcontroller.

Did they manage to do anything for you on this?

The scope is still back at Tektronix, repairs are being finalized. The end damage was they are replacing the front panel board (I think) for $600, plus another $600 for Calibration (this one is required for any equipment going back) plus the cost of shipping. Half of what was originally quoted, which I can live with even though it makes me grumpy.
<Insert Grumpy_Cat meme>

I think that the speed things are moving is very much dependent on the times, capacity everywhere seems to be reduced.

[kcbrown]

The scope is still back at Tektronix, repairs are being finalized. The end damage was they are replacing the front panel board (I think) for $600, plus another $600 for Calibration (this one is required for any equipment going back) plus the cost of shipping. Half of what was originally quoted, which I can live with even though it makes me grumpy.
<Insert Grumpy_Cat meme>

Normally I would have thought you might want to get a parts unit and perform the replacement from that, but EBay doesn't seem to have any such thing and the used units are generally up in the $5k range from what I can tell.  So considering that, $1200 doesn't seem terrible.

I think that the speed things are moving is very much dependent on the times, capacity everywhere seems to be reduced.

Yeah, it's not unexpected given the way the virus has been handled.

[Howardlong]

FWIW, my MDO3014 has started randomly hanging or resetting at boot time over the past few weeks. Occasionally the only way to get it to boot is to pull the AC out, let the caps drain (turn on with the AC unconnected speeds this up).

I hadn't used it for quite some time, maybe a couple of years, but occasionally it has the odd feature that is useful and unique on my second bench such as MagniView.

I've had it for about 8 or 9 years from new, and it's only done 546 power cycles. Typically it's lived in its Tek carrying bag for most of its life in a dry inside storage room.

Once it's up, now it'll stay working perfectly for hours until I finish for the day, but when the problem first started happening it did hang or boot two or three times while in active use. Restoring a setup or setting default setup would do this.

Anecdotally, there is a strong correlation between the hanging and the relays clicking away.

Also anecdotally, it seems to be getting better rather than worse (touch wood, cross fingers etc).

My hypothesis at the moment is that the problem may be due to a relay getting stuck, and that active use has re-wetted the relay contacts.

The scope's self test and SPC run OK.

[Howardlong]

Anecdotally, there is a strong correlation between the hanging and the relays clicking away.

Also anecdotally, it seems to be getting better rather than worse (touch wood, cross fingers etc).

About three days ago the scope suddenly completely crashed again while using it, and it failed to ever successfully reboot, stopping at various times during the boot process. After at least half a dozen attempts to get it to boot (turn it off & on, pull the AC cord for a while to let the caps discharge etc), I gave up and put it away in the cupboard to look at later.

A day later, with a view to repairing it, I pulled it back out of the cupboard, and it booted fine. It ran a couple of dozen self-tests fine too. However I noticed that the relays don't click when running a self test. Furthermore, if I set up a semi-complex setup to save in an effort to get the relays clicking, apart from setting the 50/75 ohm termination on a channel, the relays won't click... unless you have the TPP analogue probes plugged in. When doing the self test, you are directed to remove all probes, so the relays aren't cycled.

I plugged the probes back in, and I haven't been able get it to fail again after 1.5 days of solid use.

I now have a new hypothesis... the TekVPI smart probe interface is suffering intermittent connections, leading to a 1-wire comms failure, thus rebooting or crashing the scope. (Previous analysis of these TPP probes show they use a 1-wire solution).

So the next time I get a failure and it won't restart, I'll try rebooting with the probes disconnected.

[Howardlong]

Anecdotally, there is a strong correlation between the hanging and the relays clicking away.
...

About three days ago the scope suddenly completely crashed again while using it, and it failed to ever successfully reboot
...

I plugged the probes back in, and I haven't been able get it to fail again after 1.5 days of solid use.

I now have a new hypothesis... the TekVPI smart probe interface is suffering intermittent connections, leading to a 1-wire comms failure, thus rebooting or crashing the scope.

I've been using the scope daily for further 8 days, and (cross fingers, touch wood) it's worked flawlessly.

Unless it fails again, I'm going to put this down to dirty contacts & intermittent connections on the TekVPI probe connections causing a 1-wire timeout, crashing the scope. This correlates to the failures seemingly coinciding with when the relays on the input attenuator start clicking away.

So my advice is, if your MDO3000 starts hanging or crashing randomly, either at boot or while in use, the first step is to remove the probes and see if it works.

My unit had been in storage room for many months with the probes removed, so although it was stored inside its Tek bag with its front panel cover on, it's quite conceivable that the TekVPI connections had become a little tarnished, or one or more of the the TPP probes' pogo pin for 1-wire became similarly dirty. I haven't (yet) cleaned the connections, although that will probably be a next step if these problems recur.

When on the bench, typically it's set up for days if not weeks or months at a time with the same TPP probes left permanently connected, so it's a little strange this behaviour just suddenly started, but not beyond the realms of possibility.

[stj]

interesting thread,
you should probably dump those 1wire devices and keep a note of their serial number and part id.
it's a bit lame though that a 1wire device-search failure didnt just result in a probe message on the screen!

[Howardlong]

interesting thread,
you should probably dump those 1wire devices and keep a note of their serial number and part id.
it's a bit lame though that a 1wire device-search failure didnt just result in a probe message on the screen!

Yes it isn't great if my hypothesis is true, although definitely better than some kind of internal failure!

I also have an MDO4000C at my other bench space that's lived out on the bench 24/7 for six years, also with TPP probes. It's not my go to scope on that bench though, it's only had 247 power cycles and 350 hours on, but it enjoys a couple features my daily driver doesn't have (e.g. 60ps MagniVu, USB-HS decode and waveform histograms). I do cycle the probe connections on occasion though with both high speed differential and single ended FET probes, so maybe that has some kind of cleaning effect.

[northhants]

Just found that my (work) MDO3024 does not like starting when too warm (like me).

It refused to start this morning. It would get as far as the splash page and no further. It had done this occasionally recently, but previously did start on the 2nd or 3rd try. Not this time.

But today is also the hottest day so far this year, and I do not have air conditioning in my home office.

I also don't have a temperature chamber at home. But the 'scope just fitted in my fridge. After about 20 minutes cooling down, it started OK.

[Howardlong]

Just found that my (work) MDO3024 does not like starting when too warm (like me).

It refused to start this morning. It would get as far as the splash page and no further. It had done this occasionally recently, but previously did start on the 2nd or 3rd try. Not this time.

But today is also the hottest day so far this year, and I do not have air conditioning in my home office.

I also don't have a temperature chamber at home. But the 'scope just fitted in my fridge. After about 20 minutes cooling down, it started OK.

I'm sure you may well have already tried this. but did you try booting it without any probes plugged in (cf my problem immediately above)?

[northhants]

Just found that my (work) MDO3024 does not like starting when too warm (like me).

It refused to start this morning. It would get as far as the splash page and no further. It had done this occasionally recently, but previously did start on the 2nd or 3rd try. Not this time.

But today is also the hottest day so far this year, and I do not have air conditioning in my home office.

I also don't have a temperature chamber at home. But the 'scope just fitted in my fridge. After about 20 minutes cooling down, it started OK.

I'm sure you may well have already tried this. but did you try booting it without any probes plugged in (cf my problem immediately above)?

Sorry - I should have included that I tried the other suggestions first (remove mains (and obviously reconnect!), then remove probes). Still stuck at the into display.

Error Log reports 0 system errors

Also, Powerups: 579
Operating time: 18988 hours

Just tried power cycling (I'm stopping now, so doesn't matter if it doesn't start!) - and it started!

[Howardlong]

Just found that my (work) MDO3024 does not like starting when too warm (like me).

It refused to start this morning. It would get as far as the splash page and no further. It had done this occasionally recently, but previously did start on the 2nd or 3rd try. Not this time.

But today is also the hottest day so far this year, and I do not have air conditioning in my home office.

I also don't have a temperature chamber at home. But the 'scope just fitted in my fridge. After about 20 minutes cooling down, it started OK.

I'm sure you may well have already tried this. but did you try booting it without any probes plugged in (cf my problem immediately above)?

Sorry - I should have included that I tried the other suggestions first (remove mains (and obviously reconnect!), then remove probes). Still stuck at the into display.

Error Log reports 0 system errors

Also, Powerups: 579
Operating time: 18988 hours

Just tried power cycling (I'm stopping now, so doesn't matter if it doesn't start!) - and it started!

Has this gone away now (as mine did once I'd figured out I had a probe interface issue), or is it still a problem?

Once it's actually started, does it work flawlessly? (In my case, it would crash or reboot anything from a few minutes to a few hours after switching on, as well as on boot).

[northhants]

Just found that my (work) MDO3024 does not like starting when too warm (like me).

It refused to start this morning. It would get as far as the splash page and no further. It had done this occasionally recently, but previously did start on the 2nd or 3rd try. Not this time.

But today is also the hottest day so far this year, and I do not have air conditioning in my home office.

I also don't have a temperature chamber at home. But the 'scope just fitted in my fridge. After about 20 minutes cooling down, it started OK.

I'm sure you may well have already tried this. but did you try booting it without any probes plugged in (cf my problem immediately above)?

Sorry - I should have included that I tried the other suggestions first (remove mains (and obviously reconnect!), then remove probes). Still stuck at the into display.

Error Log reports 0 system errors

Also, Powerups: 579
Operating time: 18988 hours

Just tried power cycling (I'm stopping now, so doesn't matter if it doesn't start!) - and it started!

Has this gone away now (as mine did once I'd figured out I had a probe interface issue), or is it still a problem?

Once it's actually started, does it work flawlessly? (In my case, it would crash or reboot anything from a few minutes to a few hours after switching on, as well as on boot).

Once it started it stayed working. I do not think I have ever seen it reboot after power on.

[Howardlong]

Once it started it stayed working. I do not think I have ever seen it reboot after power on.

OK, so it only fails at startup, and it still fails at startup (sometimes)? Or have you just left it switched on (understandable!).

Yours has about 50x the hours of mine, but only about double the power cycles.