Ubiquiti US-48-500w switch no boot. Main PSU works.

[acedogblast]

Hello everyone. I am in the process of fixing a Ubiquiti US-48-500W network switch. It came to me with the following situation:
1. Fans spins when power is applied. There is no power switch or any other buttons except for the reset button which does not do anything.
2. None of the front port status leds or the main indicator led on the left side surrounding the Ubiquiti logo work or show any sign of life.
3. Console serial gives me no output at all.

Opening up the switch shows no visual defects. The main PSU does not have any puffed up caps and is built rather well using Nichicon and Rubycon electrolytic capacitors.
Time to measure voltages starting with the main PSU which provides 3 voltage rails (24V for main PCB, 26.5V and 53V for POE injection boards). There are no stand by supply for this PSU so all voltage rails turn on together. All 3 of these voltage rails work and I have verified with my oscilloscope that ripple was in acceptable ranges (rms less than 10% of nominal). This confirms that main PSU is not the problem which is a common issue with these Ubiquiti switches.

Switching focus to the main PCB there are 2 main Broadcom switch ICs with each having it's own 5v, 3.3v, and 1.0v supplies. There is also a single 1.5v for the ddr3 dram module. Checked all of these and found them to be acceptable. Note there are also a "12V" rail for each of the 2 main ICs but measuring them gives me 6.3V DC. This at first confused me but as it turns out that it is intentional as it is an intermediate voltage that feeds into the 1.0V rail.

So all voltages are present and with in acceptable ripple. The only thing that I know that can cause a no boot situation is a bad or corrupted firmware. The NOR flash chip containing the firmware is on the back side of the PCB. It is a MX25L25635F-10G which if you done repairs on PS4s you would recognize it. Tried pulling the firmware off the chip and after a few tries got a full dump of the firmware on my laptop. Using binwalk shows me that the firmware on the switch is "Ubiquiti 5.64.8.13083". Comparing it to the same version firmware downloaded on the Ubiquiti website shows that the first 2 entries "Ubiquiti firmware header" and "Ubiquiti partition header" with U-boot is missing/corrupted on the switch.

Figured that was the problem so I attempted to flash the latest firmware (6.5.59) to the flash chip but could not write. I then de-soldered the chip and replaced it with a brand new one since I had some and soldered the new one with the latest firmware back on the main PCB. Reassembled the switch back together and attempted to boot it back on. Unfortunately still no boot. I figured that this would be the fix but no sign of life on the front leds or on the console serial port just like the beginning.

Going to try to flash the firmware again with the same version found on the original corrupted firmware to see if that does anything. Really hopping that this switch is repairable as there is no visual damage what so ever. It would be a real shame if the main switch ICs are dead but I don't have any indication or test that would prove it?

Do any of you guys have any suggestions on what else I could check for?

Edit: I may have done my ripple measurements wrong. Will redo them.

[Swake]

Do you know what preceded this situation with the switch? FW update; power failure; something else?

For what it is worth, the reset is somewhat a specific procedure:

    Switch off switch by unplugging the cable
    Wait for 20 seconds to discharge all PSU condensators.
    Press reset button while connecting the power cable
    Keep reset button pressed for 20 seconds on power up.
    Release reset button
    Power off again
    wait again 20 seconds
    Power on

I would also try this with an older FW version, even down to v4 or V3, just in case there is something particular with that version.

[acedogblast]

Tried with an older but confirmed stable firmware version v4.3.20 and still no boot. Tried reset procedure Swake suggested but still nothing. Both heatsinks do get a bit warm to the touch after a minute the switch on but not very hot so I assume that's fine.
I have also tried connecting directly to the serial pins on the main PCB instead of going through the rj-45 console jack(it is just a TI MAX3321 line driver). Still no output.

[acedogblast]

Do you know what preceded this situation with the switch? FW update; power failure; something else?

I have no details on when or how this issue occurred.

[Swake]

Some ideas:

Is the µC talking to the Flash? Try to scope the signal. If not, are the tracks still connected?
Is the power good on µC and on flash itself? (Is it 3V or 3,3V?)
Is clock present on both chips?
Check for reset pin and signal on the µC and on the flash.

FYI:
I've had the 24port/250W version of that switch. It takes 30 seconds or so before the white LED comes on and another 2 or 3 minutes to boot to blue light (and some disco lights on the ports at the end ).

[acedogblast]

I tried probing the CS pin on the SPI flash chip and it won't go low. The SCLK pin isn't giving me anything but noise. The power pins for the SPI flash is good at 3.3v. Both crystals are giving me a nice solid 25 MHz sine wave. I left the switch on for 5 minutes and the status led never turns on.

Re-checking the ripple on the power rails again give me good indicators. Main 24V supply has ~15mVrms ripple and 210mVpp. There are 4 separate and independent 3.3v rails on the main PCB. The worst ripple was only 19mVrms and 70mVpp which is definitely fine.

I am guessing now that either there is a bad connection or the main IC itself is bad. Not sure if I am able to reflow the main ICs with just a hot air gun but I might be worth a try. I have never done a large BGA before.

[Swake]

Wouldn't the clock for the flash chip come directly from a crystal? I don't know but if so you can surely trace that signal.

No shorts to GND on any signals to the flash?

Try with the PoE boards disconnected. (Have found some pictures on the net and they appear to be separate boards.)

Is there another µC or only those 2 Broadcom chips? If both are exactly the same you could try swapping them (after the reflow tentative). My thinking is that one of them should be the main/master CPU and the other is a slave.

[fzabkar]

Tried pulling the firmware off the chip and after a few tries got a full dump of the firmware on my laptop. Using binwalk shows me that the firmware on the switch is "Ubiquiti 5.64.8.13083". Comparing it to the same version firmware downloaded on the Ubiquiti website shows that the first 2 entries "Ubiquiti firmware header" and "Ubiquiti partition header" with U-boot is missing/corrupted on the switch.

Could you upload these firmware dumps? I expect that the various firmware modules would be checksummed.

[acedogblast]

Here is the original firmware on the switch: (https://drive.google.com/file/d/1KDWkVgfMXbQ6-uDoA8ZgMXFaI3DmztjE/view?usp=drive_link)
The new ones can be downloaded from Ubiquiti's website.

As for the swapping the main chips I don't think that is within my lab's capabilities. The only other large ics on the board are Broadcom "B50282C1KFBG" which from what I can gather online are ethernet PHYs. These should not affect the boot process.

[acedogblast]

From what I have seen from this video () I measured the resistance of the inductors to ground. All except the 1.0v rails are above 1KOhms. The left side 1.0V inductor measures 12.0 Ohms and the right side 10.1 Ohms. Not as the very low 2 Ohm the video has but could be an indicator of a almost shorted cap. My PCB is not discolored. Going to measure all of the mlcc caps on the back to see if any are shorted.

[fzabkar]

All except the 1.0v rails are above 1KOhms. The left side 1.0V inductor measures 12.0 Ohms and the right side 10.1 Ohms

Vcore load resistances are usually on the low side. The fact that both cores produce similar measurements suggests that these rails are probably OK.

I notice that there are two similar blocks of code at 0xE0000 - 0xFDFFFF and 0xFE0000 - 0x1EDFFFF. These appear to be two different Ubiquiti firmware versions.

At 0x1EE0000 there appears to be a lonely 32-bit value which I suspect is a CRC.

I'm wondering whether the two aforementioned blocks are the active and inactive firmware modules. Perhaps a firmware update writes new code to the inactive module, and then makes it active if the update completes without error.

Edit:

0xE0000 - 0xFDFFFF, size 0xF00000, Ubiquiti 5.64.8.13083
0xFE0000 - 0x1EDFFFF, size 0xF00000, Ubiquiti 5.43.36.12724

[Swake]

You're likely correct saying there is room for 2 firmwares in the flash. I would think that, given there is a specific reset procedure when an upgrade fails, the 'second' firmware is some recovery firmware possibly with limited functionality. Also from what I remember, under normal conditions, the text exposed on the console shows that an image is selected prior to the boot. I don't remember if it was possible to intervene manually in this process.

On the other hand we might think that the code in the flash is a composition of a firmware as posted on their website + some other things making the 'dual boot' possible. That would make it difficult to rebuild it.
Could you compare the dumped firmware structure with the new ones you have flashed? (sorry can't help with that, I have no access to a full computer for a couple of days)  As you could read that old flash, I suspect that chip was ok. Might as well assume what was on it is still valid too.

Do you have access to a thermal camera? If not you can use a very volatile liquid and spray that on the suspected area. Check for evaporation, if the surface is hot it will evaporate much quicker that on other parts. Takes only seconds but you can see the difference easily.  Isopropyl Alcohol (IPA ) works well.

[Swake]

An example on how to detect a hotspot with IPA:

https://youtu.be/6Z8rFuM_67Y?list=PLbRsB6WOs7Fup7nPzU0NOCIVLFisn9eyl&t=381

[fzabkar]

I think I understand how the firmware updates are unpacked into the flash IC, but I need the actual .BIN that was used on this device.

Never mind, I found it:

https://community.ui.com/releases/USW-Firmware-5-64-8/c2f999a4-2135-4c9d-a9aa-075acf2c0547
https://dl.ui.com/unifi/firmware/US24P250/5.64.8.13083/US.bcm5334x_5.64.8+13083.210706.1047.bin

[fzabkar]

The .BIN file has a 0x104-byte main header:

Code: [Select]

Offset(h) 00       04       08       0C

00000000  55424E54 55532E62 636D3533 3334782E  UBNTUS.bcm5334x.
00000010  76352E36 342E382B 31333038 332E3231  v5.64.8+13083.21
00000020  30373036 2E313034 37000000 00000000  0706.1047.......
00000030  00000000 00000000 00000000 00000000
........
00000100  00000000
Then there is a 0x40-byte header:

Code: [Select]

Offset(h) 00       04       08       0C

00000100           EF6BED74 00000000 50415254      ........PART
00000110  752D626F 6F740000 00000000 00000000  u-boot..........
00000120  00000000 00000000 00000000 00000000
00000130  00000000 1C000000 00000000 0006C616
                     ++++++          ^^^^^^^^
00000140  000C0000
          ========
It defines a firmware component called "PARTu-boot" with an actual size of 0x6C616 and a reserved size of 0xC0000.

Offsets 0x144 - 0x6C759 (size 0x6C616) are written to 0 - 0x6C615 in the flash.

This is the next 0x40-byte header in the .BIN:

Code: [Select]

Offset(h) 00   02   04   06   08   0A   0C   0E

0006C750                           3C8E DE4F 0000            ......
0006C760  0000 5041 5254 6B65 726E 656C 3000 0000  ..PARTkernel0...
0006C770  0000 0000 0000 0000 0000 0000 0000 0000
0006C780  0000 0000 0000 0000 0001 1C0E 0000 0000
                                     +++++++
0006C790  0000 00EA 8DC0 00F0 0000
               ^^^^^^^^^ =========
Similarly, it defines a firmware component called "PARTkernel0" with an actual size of 0xEA8DC0 and a reserved size of 0xF00000.

The corresponding data are written to 0xE0000 in the flash.

The end of the .BIN contains a 0x100-byte digital signature.

Code: [Select]

Offset(h) 00   02   04   06   08   0A   0C   0E

00F15550                           84EF DBFE 0000            „ïÛþ..
00F15560  0000 454E 4453 3BDB D0EC 8911 EBFE 2235  ..ENDS;ÛÐì‰.ëþ"5
00F15570  4E07 C206 E0CA 164F D2DF 43AC 1823 CCBE  N.Â.àÊ.OÒßC¬.#̾
00F15580  B18E 600E 2051 841C 9CDF B42B 25AE 2815  ±Ž`. Q„.œß´+%®(.
00F15590  6DAA 0ED5 19DF 9359 8574 02A7 A139 D7CF  mª.Õ.ß“Y…t.§¡9×Ï
00F155A0  CC16 D5B7 0E61 4E1C 477D 641A 1544 33FF  Ì.Õ·.aN.G}d..D3ÿ
00F155B0  3808 4027 D1DE 74C9 ADB7 347F 7C70 D87C  8.@'ÑÞtÉ.·4.|pØ|
00F155C0  6D68 20EC BE8C 7A5F 5AC5 16F7 B62D 32A2  mh 쾌z_ZÅ.÷¶-2¢
00F155D0  85A0 91E5 F53B 1204 2010 0F7B 9C5A E8E4  … ‘åõ;.. ..{œZèä
00F155E0  5725 4F64 06F5 E328 3D13 08E5 21D5 F43E  W%Od.õã(=..å!Õô>
00F155F0  807D A14B C3B5 67D1 3C26 35CB 909C 1580  €}¡KõgÑ<&5Ë.œ.€
00F15600  3742 2D67 352D D8CB B9BE 9DB8 FD5F B7E3  7B-g5-Ø˹¾.¸ý_·ã
00F15610  E59A 6774 6DB8 0268 02E4 218F 226F 4017  åšgtm¸.h.ä!."o@.
00F15620  06F3 12E9 7B1D 46FC 4FE5 CED1 B106 21F4  .ó.é{.FüOåÎѱ.!ô
00F15630  96D8 B468 1387 78D0 E7D9 BD92 39F7 C8D2  –Ø´h.‡xÐçÙ½’9÷ÈÒ
00F15640  5630 1ED7 BF8D 0DBC A0CE 47FC 5852 54EC  V0.׿..¼ ÎGüXRTì
00F15650  451D EDC0 715E D8C5 5894 6EA7 1FAF B41D  E.íÀq^ØÅX”n§.¯´.
00F15660  38F0 174F 8BBF 0000 0000                 8ð.O‹¿....
There are small data areas at 0xC0000 and 0xD0000 which are not touched by the update.

I believe that the first 32-bit dword of each 0x40-byte header is a CRC/checksum for that particular module.


Edit:

I was wrong about the CRC. The main header has a 32-bit CRC at the end. It is the same algorithm as is used by PKZIP. This is the default CRC-32 algo in HxD (freeware hex editor).

Code: [Select]

Offset(h) 00       04       08       0C

00000000  55424E54 55532E62 636D3533 3334782E  UBNTUS.bcm5334x.
00000010  76352E36 342E382B 31333038 332E3231  v5.64.8+13083.21
00000020  30373036 2E313034 37000000 00000000  0706.1047.......
........
000000F0  00000000 00000000 00000000 00000000
00000100  00000000 EF6BED74
                   ^^^^^^^^
I was also wrong about the 0x40-byte headers. They are in fact 0x38-byte headers.

Code: [Select]

Offset(h) 00       04       08       0C

00000100                             50415254              PART
00000110  752D626F 6F740000 00000000 00000000  u-boot..........
00000120  00000000 00000000 00000000 00000000
00000130  00000000 1C000000 00000000 0006C616
00000140  000C0000
The CRC is calculated over the header and the corresponding module. It is the same algo as before. It is located at the end of the module.

[acedogblast]

I do not think the contents of the firmware matters at this point since the flash chip is never read by the main IC because the chip select pin on the SPI interface never gets pulled down.

[fzabkar]

I found this:

DS1000 Ethernet Access Switch Hardware Specification 1.0:
https://www.opencompute.org/documents/ds1000ethernetaccesswitch-docx-pdf

Pages 38 and 39 show typical PSU specs for these devices. I don't think that 10 ohms is too low a resistance in this case.

The firmware update suggests that your SoC is a member of the BCM5334X family.

BCM53346 - 64 Gb/s Multilayer Switch:
https://docs.broadcom.com/doc/53346-PB

[asis]

Hi,

In my practice, there have been several failures of Cisco Systems network switches (C2811; C3500; C2960-48TC-L; Cisco C2960G-24TC-L; Cisco Wi-Fi AIR-WLC2125;).
All failures were due to a power outage.
If it is not possible to communicate with the device through the console port (RS232) using the ROM monitor Console (ROMMON), all attempts to restore functionality will be in vain.
At least, if the IOS is not destroyed, the SoC, when powering up during initialization, should respond with replicas via RS232 about the stages of executing the SETUP functions.
All SW updates are usually done via a TFTP server.

I think the SETUP mechanisms for checking your switch should be the same.
And, the fact that there are two SW images in the flash memory is normal.
One image is original - at the production stage, the other as a result of the last update.

[acedogblast]

 I guess the last resort is the try reflow the main BGA ICs? If that doesn't work I suppose this means that the switch is truly dead then?

[Swake]

You made absolutely sure those BGA' chips have power where it needs to be?
That chip should at least try to talk to the flash (something you could monitor with the scope). If that is not happening and you have power and clock to that chip, then there is not much left...

With a hot air gun and a little carefulness you cannot break many things more than these are already. You're only reflowing, not reballing.
- Take detailed pictures just in case you blow off some other component.
- My own preference is to apply heat masking around the chip. You probably already know about the green (=lower temp) or orange (=higher temp) kapton tape. If you don't have any aluminium tape or copper tape will do, or even aluminium foil, worst case some sheets of metal cut from a can, whatever deflects heat from the other places is a good measure.
- Be extra careful if you have components on the bottom side of the PCB.
- Plenty of examples on youtube.


I have successfully reflown several GPU's in the oven of the kitchen (just don't tell your other half).

[acedogblast]

I already confirmed that all voltage rails are present  and working. I already scope probes the spi flash and found that the chip select never got pulled low which is needed for the flash to talk to the main IC.

I will definitely take your advice on taking as many pictures as possible just in case. I will try to reflow the left main ic as that is what I suspect is the master chip. I will also reflow the single  ddr module as well just in case. For reflow tools I have a hair drier and a 700w hot air gun. I plan to suspend the board using the hair drier to heat the bottom while I use the hot air gun to heat the top side. Should I try to remove the heatsinks or keep them on?

[asis]

I give 0.5 percent ......

[Swake]

You absolutely need to remove the heat sinks.
You do not want to heat the bottom at all, else all those small components will fly away...
Do only 1 chip at a time and then retest. Else you'll never know where the issue was and that is something very helpful should it occur again.

Exercise on some other scrap board if you can.

[acedogblast]

I have not yet attempted to reflow the BGAs yet just I just had an thought.

The left side 1.0V inductor measures 12.0 Ohms and the right side 10.1 Ohms.

Both the left side and the right side ICs are the exact same chip and their VRMs are identical so my theory is that if there is a connection issue with the left IC which I assume is the master IC that handles the booting process then a loose/open connection on the left IC would cause the 2 ohm resistance difference. I am going to see if I can get some clamps to put some pressure on the heatsink on the left IC so see if that makes any difference.

[acedogblast]

Applying pressure to the main chips does not change anything.
I was able to remove the heat-sinks after careful prying between the chip and the heat-sink. Unfortunately I was scraping too hard and ended up almost removing the markings on the left IC  . I was more careful on the other chip and took pictures of them both.

Interestingly the 2 main switch chips are not the same model. U1 (left side) which I suspect to be the master chip is from what I can read is a "BCM53344A0KFSBLG". The right chip labeled U2 is "BCM53346A0KFSBG". This makes sense according to public documents Broadcom have on their website (https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm5334x). Notably this section:

BCM53344 includes up to four GbE or two GbE + two HiGig-Duo™ cascade ports for non-blocking 48-port design
BCM53346 includes up to four XFI/SFI uplink/stacking ports or two XFI + two HiGig-Duo™ cascade ports

I assume the HiGig-Duo is the interconnect between the 2 ICs. This makes the U1 chip handle half of the 48 Ethernet ports and the first 2 1Gbit SFP cages while the U2 handles the other half of Ethernet ports and the 2 10Gbit sfp+ cages.

That is all I have to report today. I have a vacation coming up this weekend so I am not sure when I can do the reflowing. The other good news is that both of these chips are available for reasonable prices in the typical Asian e-shops so replacing them if faulty is now a possibility.