Beckman Coulter Optima XE-90 Ultracentrifuge SMPS Premature Failure

[fmashockie]

Hey y'all.  Just thought I would share this here for engineers in a similar role as myself with limited resources.  I work as an in-house engineer servicing/repairing lab equipment at a biotech company.  For about 90% of the 400 pieces of equipment I am responsible for, the manufacturers do NOT provide service manuals (they provide user manuals, but they are limited in scope) and/or schematics.  Not even block diagrams of the internal circuitry.  So it can make electronic repairs difficult because the circuits I encounter for some of this equipment are complex.

I wanted to share specifically about this Beckman Coulter Optima XE-90 ultracentrifuge that just had a catastrophic failure of its switchmode power supply (SMPS).  There are two reasons I put this topic in this forum.

1) The SMPS literally failed right after the warranty expired.  The warranty for this power supply (it is a Huigao Magnetics AWM700T60V(4A)SF) is 3 years according to its datasheet on their website.  The centrifuge has a manufacture date of 11/13/2020 on its label.  So it literally failed right after 3 years.  And catastrophically I might add.  All internal fuses blown, 2 power MOSFETs on one of the output modules shorted, and some of the SMD components (a ceramic cap, resistor, and SOT-23 TL431) literally FELL OFF the PCB.  Never seen anything like it before.  Good luck getting the schematic for this power supply.  It is a Chinese company and they never responded to my request for one (not saying all Chinese-made electronics are bad, just makes communication with them for support difficult).

2) The power supply in our centrifuge has 3 outputs according to its label - a 24V 8A, and two 60V 4A outputs that are in parallel for a total output of 8A.  The problem is that the 60V outputs are only 36V each.  And I have no idea what the max output for the 36V modules are.  Each time I tried to test the supply with my electronic load, when I got above 4A (the max I got it to was 5A), either the temp protection would kick in or the MOSFETs would short on one of the '60V' outputs.  This really confused the heck out me when I was trying to repair it.  Because I was chasing a red herring.  I was convinced there was still something wrong with the unit because of the label, yet all of my testing indicated the unit was fine.  Also, the PCBs that this SMPS supplies didn't have any test points that indicated what the output was (i.e. a 60V or 36V testpoint).  There was one for 24V output which supplies the embedded PCB, however. 

I just figured I would share this because these pieces of equipment aren't cheap.  And they are not cheap to fix either.  Good luck getting any info from Beckman Coulter to help you.  They will only allow one of their own service techs touch them.  You might as well be leasing the thing from them because they sure as hell don't act like you own it.  One of the reasons I'm a big advocate of right to repair!  But anyway, here's some links to some videos I did on repairing the SMPS.  They are too long and not the best production/editing, but hopefully they might help someone in the same position as me some day.

Part 1
             

Part 2 (fixed)

[NiHaoMike]

Just install some Mean Well modules as a replacement?

[fmashockie]

I may go that route if this thing fails like this again.

[fmashockie]

Welp I was dead wrong (wouldn't be the first time  ) about this power supply having only 36V outputs instead of 60V.  During some additional testing I found that the power supply was outputting up to 60V during operation.  This was happening when the unit's Peltier element or thermoelectric module (TEM) was cooling at max setting.  I wasn't even sure exactly what the 60V output was powering until now.  This is the challenge involved with working on equipment without service manuals/schematics which is why I am big advocate of right to repair.  Also, according to the system monitoring application in the unit's software, the TEM was pulling upwards of 8-8.5A!  The power supply is only rated for 8A on the 60V output! 

I still think this topic is worth keeping in this forum because of how quickly after the warranty this power supply failed (almost exactly 3 yrs which is the length of the warranty).

I do have some questions though for anyone who might have some input:

1) Can anyone speculate as to why this power supply was only outputting ~30V on the 60V outputs when it was removed from the unit?  The 24V output was putting out 24V.  There are these logic connectors on each of the output modules (see photo).  But the 24V module has that input as well.  Potentially how is the circuitry this SMPS supplies triggering the SMPS's full output?

2) When I performed load testing on the SMPS out of circuit, I could only drive it up to 5A, before two of the power MOSFETs on one of the 60V output modules shorted out.  In parallel, the two 60V modules should be able to handle 8A.  Also, as I mentioned earlier, the current measured on the 60V output by the centrifuge's software was measured at over 8A.  How could it handle 8A in-circuit, but not out of circuit?

I apologize for my first post without having collected more information.  Also, for anyone who needs it, the service code for putting this centrifuge into service mode is 372466.  I don't believe the manufacturer hands this out to customers and this code allows you to perform calibrations of the unit's rotor motor, temperature control, and vacuum pump.


 

[helius]

Forgive me if this is too obvious and you have already checked it, but the "extra connections" could be for remote voltage sense wires. Many power supplies will either oscillate or not regulate at the correct voltage if output sense is not wired up, because its feedback circuitry is controlled by it. Some supplies have a documented way to switch between remote sense and local sense.

I will watch your videos at some point but there is no time at the moment

[fmashockie]

Thanks for your comment helius.  I guess that makes sense.  So this supply has 3 'modules' for each output (here's a link to the manufacturer's website advertising the PSU https://en.huigao-magnetics.com/encate_proinfo/77.html#main_frame).  There's a 24V 8A module, and two 60V 4A modules connected in parallel at the output.  Each module has a connector by the output called the 'module logic connector'.  It has the following pinout description:

1 = +Sense V1
2 = -Sense V1
3 = V_prog
4 = IShare
5 = NC
6 = +Inhibit
7 = -Inhibit
8 = +Dock
9 = -Dock
10 = NC

Now the centrifuge utilizes those connectors for the 60V outputs.  But it does not for the 24V output.  What I don't understand is why the 60V outputs only regulate at ~34V when the supply is powered up by itself. The 24V output regulates at 24V and it has that same logic connector.  What makes the 60V outputs different?

At this point the centrifuge is running fine after my repair so now I'm more curious about how these modules work and the difference between the 24V module and the 60V modules.  But thanks for taking the time to respond!

[fmashockie]

Update: 7 months later this power supply (Huigao Magnetics AWM700T60V(4A)SF) has had a catastrophic failure again!  Same two high voltage switching MOSFETs on the same 60V module were found shorted and the internal fuse blew.  This also happened in the first failure 7 months earlier.  However, there are some differences:

• Failure on 11/2023 - a TL431 SOT23 and some other SMD components literally fell off the 24V module. 
• Failure on 6/2024 - The Ct cap for the oscillator of the UC2845 on 60V module with blown MOSFETs had fractured causing the oscillator frequency to jump up to 2MHz (should be ~250kHz).  So this high frequency was driving the MOSFETs to failure.
• Discoloration of the transformer casing on the 60V module with blown MOSFETs.  Possibly due to it overheating due to higher than normal switching frequency.

I repaired the supply by replacing the two HV MOSFETs, the fuse, and the Ct cap for the UC2845. 

I also reached out to Beckman Coulter to see if they have any reports of premature failures of this supply in the field.  They told me that they haven't gotten any reports like this and that the supply has reputation for being 'bulletproof'.  It is also $7500 for a replacement  (Beckman Coulter part # is B05755). 

This supply in my opinion has a design flaw.  It cannot dissipate heat effectively.  And that is causing thermal stress of internal components.  Also, I think Beckman Coulter should have picked a more powerful supply.  The thermoelectric module (TEM) is pulling up to 8A at max setting.  The two 60V modules in parallel only provide a max of 8A.  Wouldn't it be better have overrated the supply (ex. 10A or more)??

I think this belongs in the dodgy tech thread because of how expensive this equipment is.  The centrifuge is $100k new and a replacement supply is $7500.  It is only 3-4 yrs old.  And the supply had the first failure literally right after the warranty ended.  Therefore, the price tag demands superior quality than what I've observed here.  Plus, this is an ultracentrifuge.  They can be extremely dangerous to life/property if they aren't working properly.

  Here's a video from my YT channel documenting the repair. 

[AVGresponding]

I'd personally want some headroom, 20-30%, but I'm very risk-averse.

Is there no way to retrofit any sort of active cooling?

[fmashockie]

I'd personally want some headroom, 20-30%, but I'm very risk-averse.

Is there no way to retrofit any sort of active cooling?

Thats a good idea!  I didn't even consider that.  I'd have to give it some thought though.  I was thinking just finding a supply from an alternative manufacturer with more power output.  A lot of manufacturers make configureable/modular supplies like this.  The problem I'm having is finding one that matches the output logic connectors on this one.  I'd need one with all of the following:


1 = +Sense V1
2 = -Sense V1
3 = V_prog
4 = IShare
5 = NC
6 = +Inhibit
7 = -Inhibit
8 = +DCOK (edited)
9 = -DCOK  (edited)
10 = NC

Having trouble finding some with these I/Os, or they have them all but not the correct voltage output. 

[helius]

What the heck is "+Dock"? Did you mean DC_OK?

[fmashockie]

What the heck is "+Dock"? Did you mean DC_OK?

yes my bad it is DCOK.  Sorry for the transcription error

Also, that reminds me it would need a main input connector as well with the following:

1. GL INH-
2. GL INH+
3. GL DCOK E
4. GL DCOK C
5 GL ACOK E
6. GL ACOK C
7. 5V SB
8. 5V SB return

The other connectors are for the configurable output modules.