Intel Atom C2000 Failures

[bson]

Does anyone know more about the Atom C2000 family failures?  In the errata sheet, Intel states:

AVR54. System May Experience Inability to Boot or May Cease Operation
Problem: The SoC LPC_CLKOUT0 and/or LPC_CLKOUT1 signals (Low Pin Count bus clock
outputs) may stop functioning.
Implication: If the LPC clock(s) stop functioning the system will no longer be able to boot.
Workaround: A platform level change has been identified and may be implemented as a workaround
for this erratum.
Status: For the steppings affected, see Table 1, “Errata Summary Table” on page 9.

The LPC is a PCI-to-ISA bridge controller and is one of the two supported BIOS boot locations; the C2000 can either boot from SPI (default) or LPC/ISA (set via external sense pins at powerup).  This is fixed in a stepping, and curiously the "fix" consists of eliminating the ability of muxing the LPC bus pins with GPIO - they no longer become software selectable.  This is pretty much ALL I've been able to find on the subject.  There's a workaround which consists of adding an external 100 ohm resistor, but it's not clear what pins this is added to.  It's added across two pads on a connector on some Synology NAS units, so it's not an output current limiter but almost certainly a stiff pullup or pulldown.  This leads me to suspect it really goes on a configuration sense pin.  Intel hasn't made their "platform level change" public.  Tracing it out on a board is kind of hard since the SoC is a large BGA package that would need to be desoldered.

Does anyone know more about this?  Like, for example, where the resistor is added - in particular is it added to the LPC clock outputs, or to the sense inputs? 

It's also not clear if the clock output actually fails, or this is merely a convenient symptom any engineer with a scope can identify.  (The two LPC clocks are only 25MHz.)  Some possible root causes I can think of are:

1. The sense input pullup is underdimensioned and fails, resulting in the CPU trying to fetch boot firmware from SPI.
2. The sense configures it for LPC boot while the pins are reset to GPIO, resulting in duplicate pin drivers that short out internally.
3. 1+2 -  multiple sense inputs with slightly different thresholds result in inconsistent pin configuration with both pin drivers enabled.
4. The clock pin output driver actually dies.

#4 sounds simple and straightforward, but somewhat implausible to me.  This isn't Intel's first rodeo, and besides how would an external resistor help with this?

Here's the C2000 family datasheet:
https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/atom-c2000-microserver-datasheet.pdf

Errata:
https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/atom-c2000-family-spec-update.pdf

[amyk]

Some more info here: https://www.mail-archive.com/list@lists.pfsense.org/msg10553.html

[Monkeh]

#4 sounds simple and straightforward, but somewhat implausible to me.  This isn't Intel's first rodeo, and besides how would an external resistor help with this?

From what I've been able to gather, this is exactly what's happening - the resistor is connecting a different clock pin to the LPC bus. This is not the first time Intel has had consistent aging failures like this.

[bernroth]

I made a post some time ago:

https://supportforums.cisco.com/t5/firewalling/clock-signal-repair-pictures-isr4300-asa-isr4400/m-p/3088505/highlight/false?attachment-id=107384

The fix applied by Cisco is by putting a 110 ohm pull-up resistor from either LPC_CLKOUT0 or LPC_CLKOUT1 to +3.3V

I don't want to talk about their repair quality 

Currently I am trying to fix another Cisco router. I found the signals LPC_CLKOUT0 and LPC_CLKOUT1.

Does anyone know which one of these pins requires the pull-up?
Maybe I'll just put two pull-up resistors

[hfiennes]

Ok, a very late reply but my Synology DS415+ box died, and the 100 ohm resistor (across pins 1 & 6 of a 12 pin, 2mm header) made it work again.

It's all back together now so I can't really verify this theory, but do we know *how* the clock output dies? The register article quotes intel as saying "a degradation of a circuit element under high use conditions at a rate higher than Intel’s quality goals after multiple years of service.".

https://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/

Could the issue be the PFET in the clock driver dying? If that was the case, a strong pull-up on the clock line - meaning only the NFET needs to be functional to get a clock out of the pin - would indeed solve the issue. This seems to square with the Cisco fix too - it's just a strong pull-up.

(it also means that there's 33mA being sunk by the driver 50% of the time, which makes me fear for the longevity of the fix - and whether something like the smart pull-up on an I2C FM+ bus would stress the D2000 less)

[rthorntn]

I have five c2xxx Supermicro motherboards, one is a "2013" A1SAi-2750F and the rest are "2015" A1SAi-2550F's.

Basically I'm wondering out loud if I should preemptively mod these, or just wait for them to die and fix them, I don't run them 24/7 atm and I wouldn't put anything business critical on them now, how would one go about figuring out where to stick the resistor?

Thanks.

[EEVblog]

Ok, a very late reply but my Synology DS415+ box died, and the 100 ohm resistor (across pins 1 & 6 of a 12 pin, 2mm header) made it work again.

It's all back together now so I can't really verify this theory, but do we know *how* the clock output dies? The register article quotes intel as saying "a degradation of a circuit element under high use conditions at a rate higher than Intel’s quality goals after multiple years of service.".

https://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/

Could the issue be the PFET in the clock driver dying? If that was the case, a strong pull-up on the clock line - meaning only the NFET needs to be functional to get a clock out of the pin - would indeed solve the issue. This seems to square with the Cisco fix too - it's just a strong pull-up.

(it also means that there's 33mA being sunk by the driver 50% of the time, which makes me fear for the longevity of the fix - and whether something like the smart pull-up on an I2C FM+ bus would stress the D2000 less)

I just shot a video on this after finding a DS415+ in the dumpster!
Yes,m the resistor fix works, and I assumed it was bypassing a clock somehow but couldn't trace exact details.

[Mazian]

Quote from: rthorntn on November 18, 2019, 05:42:45 am

I have five c2xxx Supermicro motherboards, one is a "2013" A1SAi-2750F and the rest are "2015" A1SAi-2550F's.

Basically I'm wondering out loud if I should preemptively mod these, or just wait for them to die and fix them, I don't run them 24/7 atm and I wouldn't put anything business critical on them now, how would one go about figuring out where to stick the resistor?

Bit of a late followup, but... I'm also running an A1SAi-2750F, and a friend pointed me at the very helpful DS415+ video.  A user on another forum found the pins for a similar board, and the manual for the A1SAi boards has the same header shown on page 43, the JTPM1 header.  I made a 100 ohm jumper and popped it onto the board across pins 1 (LPC clock) and 9 (+3.3V):



Can't be 100% sure it's doing anything, since my board hadn't died yet, but at least it didn't make it worse!

[awallin]

Thanks for posting this! 

We tried this on our SuperMicro C2000s, and it does work! One of these was run 24/7 since 2015 and died last week - now back from the dead 
IIRC these are SuperMicro 5018A-MLTN4 with A1SAM-2550F mb.

[Grunchy]

Supposedly the Western Digital DL2100, DL4100, and DX4200 are also susceptible to these catastrophic disasters.
However Western Digital has apparently taken the attitude that they are not going to do anything other than let the chips fall where they may and people can use their warranty protection (or not).
https://translate.google.com/translate?sl=nl&tl=en&u=https%3A%2F%2Fnl.hardware.info%2Fnieuws%2F51213%2Fwestern-digital-geeft-statement-over-c2000-defecten-onderneemt-geen-actie

However, I do have a dead DL4100 and I'm game to try wiring in a 110 ohm resistor between +3.3V and LPC-CLKOUT0/1. Why not, the box is dead anyway.
However some smart wag said that he heard there is no pin connection from anywhere on the motherboard to either of the LPC-CLKOUT pins. Would us DX4200/DL2100/DL4100 owners therefore consequently all be SOL?
Rats.

[nigelwright7557]

i do like a good bodge to get things working.
i recently messed up a pcb by accidentally swapping power pins to an ic.
luckily it was cut and strap to fix it.
with the onset of cheap Chinese pcb's it isnt always an expensive problem.

[wraper]

Ok, a very late reply but my Synology DS415+ box died, and the 100 ohm resistor (across pins 1 & 6 of a 12 pin, 2mm header) made it work again.

It's all back together now so I can't really verify this theory, but do we know *how* the clock output dies? The register article quotes intel as saying "a degradation of a circuit element under high use conditions at a rate higher than Intel’s quality goals after multiple years of service.".

https://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/

Could the issue be the PFET in the clock driver dying? If that was the case, a strong pull-up on the clock line - meaning only the NFET needs to be functional to get a clock out of the pin - would indeed solve the issue. This seems to square with the Cisco fix too - it's just a strong pull-up.

(it also means that there's 33mA being sunk by the driver 50% of the time, which makes me fear for the longevity of the fix - and whether something like the smart pull-up on an I2C FM+ bus would stress the D2000 less)

I just shot a video on this after finding a DS415+ in the dumpster!
Yes,m the resistor fix works, and I assumed it was bypassing a clock somehow but couldn't trace exact details.

As I understand it, it's simply a strong pull-up which mitigates/replaces a dead high side switch in CPU.

[pf]

On my DS415+ I tried various values of the resistor between pins 1 and 6. Starting with 100 ohms and increasing the value in roughly 100-ohm steps, I found that 750 ohms was sufficient to restore the LPC clock signal. 1000 ohms did not restore the LPC clock signal.

[nevusZ]

https://www.eevblog.com/forum/repair/barracuda-networks-f80-repair-(atom-c2000-bug)/

hello,
how can i find out if i can fix my board the same way?

[hindenbugbite]

I was recently hit by this bug on a QNAP TS-251+ with an Intel Celeron J1900. The resistor mod did work. In my case it was a 180 ohm resistor to ground.

The scope traces I took before and after shows the strong pull down resistor loading down both the high and low level of the clock. In effect pulling it back into CMOS input threshold range:



While I'm glad it works again, I don't believe this to be a reliable long-term fix. Intel's errata noted that this issue doesn't occur if LPC is 1.8V which suggests the lower power reduces degradation. The strong pull down here only continues to stress the push-pull FETs, of which the pull part has already degraded.

If more time permits, I might try to disconnect the LPC clock from the CPU and inserting an inline buffer to regenerated the clock. A buffer should help reduce the load current that the output has to provide which hopefully will provide a longer term fix. The QNAP has an impedance matching resistor between the clock output and the rest of the board so it is possible to buffer the clock as long as latency is kept in check. (maybe < 10ns) The issue is that the clock now has a 1V offset which won't register with most CMOS logic. One option is to go analog and use an op-amp to offset the clock back down near 0V and even add a gain to make it full swing again.

[Whales]

Woah, I have a bad intel board with soldered processor.  Last I checked the LPC wouldn't get to a state where it output the right signals for things to continue.  Going to get it out from storage and scope the clocks now!  Thankyou for this topic, even if it doesn't end up being the cause I'm sure to have some fun.

[jesse1329]

Hi!

I was troubleshooting a gigabyte motherboard with the celeron J1900. I couldn't figure out what the problem was keeping it from booting. I pretty much decided the SoC was not enabling the bus to talk to the BIOS chip. I had the strange idea to probe the lpc clock pin and found the scope image I've attached. That's one sick puppy. So I searched for lpc clock on J1900, and realize now this is an old issue I've heard about.

I tried a 150 ohm pull-down thinking that might get some devices switching but you can see, it doesn't even swing rail to rail. Pull-up and pull-down? Or is the only way is to inject a new 25 Mhz clock?

[Foxxz]

Thanks for posting this! 

We tried this on our SuperMicro C2000s, and it does work! One of these was run 24/7 since 2015 and died last week - now back from the dead 
IIRC these are SuperMicro 5018A-MLTN4 with A1SAM-2550F mb.
(Attachment Link)

Wow! I have been running one 24/7 since 2015 as well and just before Christmas is became completely unresponsive and dead as a doornail. Nothing on the screen. Reset/powercycle did nothing.

I replaced the system with something more modern but I wouldn't mind reviving this system.

[hindenbugbite]

I tried a 150 ohm pull-down thinking that might get some devices switching but you can see, it doesn't even swing rail to rail. Pull-up and pull-down? Or is the only way is to inject a new 25 Mhz clock?

I am skeptical that injecting a standalone 25MHz clock would lead to a solution here because this is a bus clock and needs to be in phase with the data line. Your scope trace might have both a voltage and a timing problem if it really is a saw tooth. 3V CMOS needs below 0.8V for low and above 2.1V for high, so I would have guessed a pull down would have done something too. But if the timing has shifted from the edge quality then it would be hard to recover.

Just curious, could the saw tooth trace be due to your scope/probe bandwidth?

[wraper]

I tried a 150 ohm pull-down thinking that might get some devices switching but you can see, it doesn't even swing rail to rail. Pull-up and pull-down? Or is the only way is to inject a new 25 Mhz clock?

It should be a pull-up, not pull-down.

[jesse1329]

Just curious, could the saw tooth trace be due to your scope/probe bandwidth?

Yes. I checked and I needed a 10X probe to view this one correctly with my scope. So this is what a healthy clock looks like, I suppose. I'm going to check the BIOS chip more closely. LPC bus comes out of reset on power on, but no data observed, but the BIOS chip is on SPI up to 100MHz. Still wary about the SoC issue as there is no other explanation, so I'll keep checking.

[wraper]

Yes. I checked and I needed a 10X probe to view this one correctly with my scope.

Probe in 1X mode is way worse than a simple coaxial cable. It's good up to a few MHz (sine) at most.

[hindenbugbite]

Yes. I checked and I needed a 10X probe to view this one correctly with my scope. So this is what a healthy clock looks like, I suppose. I'm going to check the BIOS chip more closely. LPC bus comes out of reset on power on, but no data observed, but the BIOS chip is on SPI up to 100MHz. Still wary about the SoC issue as there is no other explanation, so I'll keep checking.

That definitely a better looking clock for 3.3V CMOS. I don't recall what the initial traffic on the LPC bus looks like but it would seem logical that the BIOS is waiting for the CPU to query it. So there should be some data packet to get things started.

While this bug is mainly attributed to the LPC CLK line, I believe it can affect any of the LPC lines. The CLK is the first to fail because it is continuous and sees the most transitions.

[Foxxz]

Quote from: rthorntn on November 18, 2019, 05:42:45 am

I have five c2xxx Supermicro motherboards, one is a "2013" A1SAi-2750F and the rest are "2015" A1SAi-2550F's.

Basically I'm wondering out loud if I should preemptively mod these, or just wait for them to die and fix them, I don't run them 24/7 atm and I wouldn't put anything business critical on them now, how would one go about figuring out where to stick the resistor?

Bit of a late followup, but... I'm also running an A1SAi-2750F, and a friend pointed me at the very helpful DS415+ video.  A user on another forum found the pins for a similar board, and the manual for the A1SAi boards has the same header shown on page 43, the JTPM1 header.  I made a 100 ohm jumper and popped it onto the board across pins 1 (LPC clock) and 9 (+3.3V):



Can't be 100% sure it's doing anything, since my board hadn't died yet, but at least it didn't make it worse!

Can confirm this fixed the dead system I had

[jesse1329]

That definitely a better looking clock for 3.3V CMOS. I don't recall what the initial traffic on the LPC bus looks like but it would seem logical that the BIOS is waiting for the CPU to query it. So there should be some data packet to get things started.

While this bug is mainly attributed to the LPC CLK line, I believe it can affect any of the LPC lines. The CLK is the first to fail because it is continuous and sees the most transitions.

I am able to see some activity on the SPI bus for the BIOS chip just before it reboots. The BIOS seems ok on my scope from what I saw, I haven't bothered putting my logic analyzer on there, since I don't think I'll matter. So there isn't really much left to check. I'm curious what other I/Os are affected on the SoC. I thought I saw the RTC might be affected. Who knows what else could be? I don't think Intel really explained what fully is affected. I am thinking these chips are ticking time bombs, depending on the board implementation and use case. I had someone contact me saying their identical board died too in the last few weeks, some time after I put out my questions on what has gone wrong.