RTO2000 RAM Artifacts

[jjoonathan]

I bought a RTO2014 the other day. Beautiful scope, but it has RAM artifacts. The seller made it right, but now I have to go and actually fix it 

My first step was to identify the RAM chip with the artifact, and I did this by cutting data lines and observing where the errors showed up (and then repairing the cuts, of course). I didn't have especially good luck in guessing which chip was responsible, but the silver lining is that I now have a mostly complete picture of how data is deposited in the RAM. Even better, I have a procedure to share with the world so that the next person to have this problem can simply drop some cursors on the waveform artifacts and immediately know which chip is faulty.

The architecture is that 4 10GS/s channels feed two ASICs and each ASIC farms the samples from its channel pair out to 14 W9751G6KB SDRAM chips. This will be different if you have option B110 extended memory, because the math doesn't work out for B110 to use the same chips. Each chip has 16 data lines and every 100ps one of the DRAM chips will receive one 8 bit sample from ch1 and one 8bit sample from ch2. The chips will be used in sequence until, 14 samples later, the first DRAM chip receives its second pair of samples. The especially good news is that this pattern is stable with respect to the timebase: if you hit PRESET and then activate Channel 2 (Firmware 5.35.1.0), you can simply drop a cursor on the first artifact past (or on) the origin, measure the picoseconds, and look up the chip on the image I attached. Be sure to use the BLUE numbers; the yellow numbers are my own labeling scheme relative to a chip I arbitrarily chose as 0 and are offset 200ps from the BLUE numbers, which are the ones that correspond to "picoseconds past the origin".

Replacement chips are on order and while they slowly make their way to me I have begun to practice BGA rework. I've done a few one-offs on smaller chips, but this will be my first chip with this many balls on this thick of a board. My first practice attempts were on the cheapest DDR4 SODIMMs I could find, because the LED testers were available on Amazon. Those went well! I think I have the hang of reballing them now, and my success rate is 100% on 4 chips out of 4. My second set of practice attempts were on a chunky but obsolete graphics card and those didn't go nearly so well. My 450W preheat plate was not up to the task. I had to crank the temperature setting very high just to get one chip off, let alone remounted. The thermocouple I had on the back side was always much lower than the setpoint, even if I insulated with aluminum foil. I have an 850W station on order, hopefully that works better. In the meantime, here are some pictures and that timing diagram I was talking about!

EDIT: compression did a number on the blue numbers, here is a higher res version. Note that the dropbox preview is also compressed, download the PDF for full resolution. https://www.dropbox.com/scl/fi/1roh0w3hl585cxuaygs77/DRAM.pdf?rlkey=7vpw6q7kcfpk3d3kpr7usy2qq&st=clrch9hl&dl=0

EDIT2: diagnostic procedure in video form. If you want the repair story, look for that video a few posts down.

[hansibull]

Thanks for the troubleshooting details! It was a joy to read. How much does one have to pay for a defective RTO2014?

[pdenisowski]

Wow.  Just ... wow.  Very, very impressed.

[tautech]

Is this a RAM SW management issue ?

[jjoonathan]

I was going to hold the video until I actually went ahead with the replacement, but I have enough footage that it probably needs to be split into two parts. See Part 1 here (and wish me good luck on Part 2):



Re: is it a RAM SW issue? No, it's definitely a hardware problem. Jump to 16:55 in the video to see where I cut traces to induce similar faults in order to figure out which of the 14 plausible candidate DRAM chips is the culprit.

Re: how much did it cost? Five shares of NVDA at time of purchase Whether or not this was a good deal depends considerably on whether or not the repair is ultimately successful.

[Harry_22]

Hi!
Good job!
Definitely I will use it for R&S diagnostic.

I think that there will be no problems with the replacement if you warm up the board well. The board is not simple with a very good heat dissipation. Lead-free soldering complicates the situation.

My advise is to remove two radiators and thermally insulate the ends of PCB that protrude beyond the bottom heating.

[DaJMasta]

Impressive sleuthing!  Good luck on the repair!

[nctnico]

Hi!
Good job!
Definitely I will use it for R&S diagnostic.

I think that there will be no problems with the replacement if you warm up the board well. The board is not simple with a very good heat dissipation. Lead-free soldering complicates the situation.

My advise is to remove two radiators and thermally insulate the ends of PCB that protrude beyond the bottom heating.

A pre-heater may not be enough to put enough heat into the board. Last time I did a similar repair job on a 10 layer board, I had to use a 1500W paint stripper heatgun (from the under side of the board) to get the temperature of the board hot enough for the solder to melt and get the BGA soldered. The problem with BGAs is the air between board and package which prevents heat from a hot air gun from above to heat the board properly through a BGA package.

[jjoonathan]

Thanks for the tips. This tracks with what I am seeing from northwestrepair on youtube -- he specializes in repairing graphics cards, which have this same sort of "hot chip surrounded by DRAM on a thick board" layout, and heat is a constant struggle. The R&S board is 2mm thick! My practice GPU is only 1.7mm thick -- I don't know the copper fraction here, but I definitely have work to do before I am comfortable with the practice GPU.

Yep, heat sinks will have to come off, so I have ordered replacement thermal paste.

I should also figure out what temperatures I can get away with. My thought here is to pick a decoupling capacitor and use the hot tweezers to figure out where the melting point of the solder is, and to sample a corner of the board and figure out where the resist starts to rapidly degrade. I also want to experiment with insulating the board. Aluminum foil is traditional, but I have really had to stack up the layers for effect on the GPU so I want to experiment with silicone cookwear as well.

In any case, I have plenty of things to fill the time while the chips ship

[nctnico]

I'm not quite sure you need to cover the board. The areas you don't heat, will cool down quickly. The only thing I typically protect (using kapton tape) are plastic parts which can melt. Another issue you'll likely encounter is warping of the board; I don't think this is avoidable. Also make sure to have ventilation as the board will likely start to smoke while doing the repair.

Worst desoldering job I ever did was removing a 486 with glued on heatsink from a motherboard.

Either way, this is going to be a tedious job with lots of room for error. The key is to heat the board gradually; don't keep the hot air pointed at one place and pray.

[Harry_22]

Hi Jjoonathan!

In the meantime I made an extension for the front panel. With a length of 20 cm a slight moire is noticeable in the image.
Connector brand Amphenol 88946-102LF and 70233-111LF.

PS
Waiting for the good news.

[jjoonathan]

Nice! Yeah, having an extension cable for the screen is a godsend. I'm glad they built one in on the RTO2000 and I remember it being a real pain on the RTO1000 so congrats on building one of your own!

The news right now is that I am waiting for thermal paste, which will arrive tomorrow 

[jjoonathan]

This story has taken some interesting twists and turns, here's an update!

Swapping DDR2 Chips

The replacement chips arrived from China, but they had gray crusty oxidized balls. I swapped them for fresh ones using the cheapo direct heat stencil kit that gets sold everywhere, which was straightforward from my practice with the DDR4. I should have ordered one of those heat-resistant pads with chip-shaped depressions to hold the chips while I beaded and wicked away the crusty old solder, but upside-down tape worked OK and I wound up with a bunch of chips with shiny silver balls and a melting point of 190°C rather than 220°C. Heat was still a struggle at this point so the extra 30°C of margin was welcome: I could copy the settings and thermocouple numbers that sufficed to remove the chip and be relatively secure in the assumption that it would do an OK job of melting the fresh BGA onto the board.

This went well! Old chip out, new chip in, booted up, problem looked fixed! And then it returned. More RAM artifacts  Different time offset, but same amplitude as before (8*LSB). Did another RAM chip suffer a failure on the same exact ball? It seemed unlikely, but maybe the heat flow or stress patterns conspired to kill Channel 2 bit 3 on multiple chips? I shotgun-replaced another 4 chips, going for the entire group of them in case I had caused this second failure by heating without reflowing and pushing the neighboring chip to maximum stress. I booted back up and... oh no, the problem was still there. And one of my induced glitches had become un-fixed when I bumped it with the soldering iron during the shotgun replacement. That was an easy fix, but left me back at square one: the 600ps 8*LSB glitch was still there. New hypothesis: the 8*LSB glitch never shifted in time, but I was upset enough to see it return that I forgot to carefully follow my own procedure and zoomed in before stopping the scope and *that* shifted it in time. The whole shotgun replacement had been a wild goose chase.

I took the 600ps RAM chip off and poked Channel 2 Bit 3 with the multimeter in diode mode. No diode. Uh-oh. My "steel wool" trick for hidden trace hunting didn't work here because the exposed ends of the vias were so small, so I injected a tone into the floating trace and hunted it out with an active probe. No contact. I would have to deal with the big chip after all. The bad joint wasn't under the RAM, it was under the big 1296 ball chip. Nooooo!

[jjoonathan]

Exorcising Demons from Intel PCH

And then the screen went red. Lol. Why did the screen go red? Demons? In my oscilloscope?

Well, I found the display datasheet and probed the LVDS lines with a diff probe. It turns out that the common mode on one of the LVDS lines had gone from 1.2V to 0V. Shout out to R&S ProbeMeter for making this a one step check rather than two step! According to the datasheet, the LVDS line in question carried the bottom 5 bits of the red signal, which explained the red tint.

I was hoping the LVDS line would lead to an easy-to-swap driver chip, but it led right to the Intel Platform Control Hub, a 989 ball 0.6mm pitch BGA. Ahhhhh!

Well, I removed the PCH with minimal fuss thanks to dialing in the preheat plate, but I did knock off a few unlabeled 0402s. In my defense, I did try to protect the 0402s, but I had just switched to a new roll of Kapton tape that was thinner and shriveled up more quickly than I expected. Fortunately, there weren't many strays, and though 2 of them were ambiguous the datasheet was available and got that sorted. These resistors were calibrating the DMI termination (DMI is the CPU-PCH interface, "PCI express but a little faster so that it can encapsulate at full speed"), so I'm VERY glad the datasheet was available.

I went a bit nuts with replacement solder mask for reasons that will become clear next post, but I have to say that I really love this stuff. It's easy to apply by dipping the end of a wire, it hardens quickly under UV, it doesn't smell, and after cured it's easy to sculpt with a knife or remove/try again.

I put on the new PCH, crossed my fingers, booted up, SUCCESS! COMPLETE SUCCESS!

Not bad for my first 989 ball 0.6mm BGA replacement. Which is only my second major BGA replacement, if you group together the 5 RAM chips and count them as the first.

[jjoonathan]

EDIT: the forum software seems to be dropping a random subset of uploaded images, so there's some churn on these posts.

Here's a template I made so that I could draw alignment marks for the PCH. The RAM chips and big R&S BGA came with alignment marks in the copper layer and solder mask, but the PCH was just plopped into place by a trusty machine that didn't need no registration marks so I had to score some marks for myself into the board with an xacto knife.

I also tried to use solder mask to make registration bumps, but they were less effective than hoped, probably due to all the other solder mask hitting other bumps and making the tactile landscape complicated. Fortunately it didn't make matters worse. Even though there was only about 0.3mm between board and package, the solder mask bumps were only about 0.1mm tall so there was plenty of clearance.

[jjoonathan]

As for how the LVDS broke in the first place, the symptoms appeared over a few boot cycles so I'm not 100% sure but either I zapped it with static or I accidentally plugged the screen into the instrument while powered and it took poorly to being hot-plugged. It could also be that I accidentally summoned some demons while performing the usual satanic ritual before every BGA removal, as one does. My guess is static, because I'd expect a hot plug problem to have enough power to insta-kill the common mode drivers, but it's hard to tell.

In any case, problem solved. I highly recommend the horned differential probes for chasing demons out of oscilloscopes.

[tv84]

 

When there's a will there's a way!

[jjoonathan]

That's the spirit! I'm not out of the woods yet though.

The RTC Chip

In parallel to the Intel PCH drama, I ordered a BGA reball jig for the RTC Chip with the poor contact beneath it. Before it arrived, I decided to try a reflow. This is where things went wrong.

When I booted the scope after the reflow, it went into "leaf-blower mode" and the mainboard stopped showing up on PCIe. Further inspection revealed that none of the power rails were coming up, and that the 1.2V core rail was ohmic short (vs 4 ohms for the other chip IIRC). I must have bridged something during the reflow, and I suspect this happened because I tried the "gentle tap" technique to see if I had successfully melted the solder. It didn't feel like I had gone too far, and I anticipated the possibility and simply planned to wait for the reball jig if it happened, but with a dead scope on hand I clearly must have tapped too hard and not felt it. In hindsight I should have checked resistances on the main rails -- any buck converters with a big inductor and certainly any multiphase buck -- before and after reflow. Not all hope was lost: I have seen many high quality R&S protection circuits and true to form they have 9 individually fused 12V domains on the mainboard, bus transceivers between them that handle power-outs, monitor circuits... they even have LEDs on the fuses. How nice is that? (01 blown fuse.jpg) There was a good chance I shorted a power stage without pulling current through the chip and in that case everything would be fine. I removed the chip, replaced the fuse with an, erm, slightly less elegant substitute (02 replacement fuse.jpg) and confirmed that the power rails came up by injecting 12V into the main bus. They did!

12V Mainboard: 4.45A (13V OVP, 5A/2ms OCP)
12V Mainboard+Frontend: 11.45A (14V OVP, 13A/2ms OCP)

The RTO even booted and enumerated the mainboard on PCIe, although with one of the RTC chips missing it did complain about GB Transceiver errors. I wonder why 

I inspected the RTC chip under a microscope and noticed that a bunch of pads looked partially or completely unwetted with solder. The pad corresponding to the malfunctioning DRAM bit looked especially bad. Welp, there's my problem! Everything tinned up nicely and in not too much time the jig arrived.

[jjoonathan]

Reball time! With a proper jig and a big BGA. No more of this DDR2 and $10 aluminum extrusion nonsense. The reball jig that you see in all the youtube videos is really great, but it demands more patience than might be apparent from watching youtubers who have everything dialed in because they do this professionally. Some learnings:

Spend the time to adjust all 4 corners so top of stencil is at the top of 4 test balls BEFORE adding flux. Otherwise you will almost certainly crash the stencil into the flux and have to start over, or you will lift the stencil so that balls can escape to the space between holes and have to start over. Starting over is costly enough that it is worth going to great lengths to level before adding flux.

Tape off the set screws. I thought I could just avoid bumping them. Nope. Just tape them.

Spend the time to mask off unused holes. If you are doing a one-off, it is tempting to just let the balls fall through unused spaces. Don't do this. The balls pick up flux and spread it into every crevice and it is a huge pain to clean out. It also prevents you from doing enough passes with the brush, and you will probably need a number of passes before you get the "feel" so that you can brush balls into stencil holes without bouncing them back out.

Less flux! I am usually a fan of the Louis Rossman school of "there is no such thing as too much flux" but this is an exception. If you have more than even a tiny amount of flux the surface tension draws the balls together the moment it starts to flow (06 chasing balls.jpg).

Finger method for spreding flux. I really tried to get the brush to work, but I never succeeded in avoiding small patchs of too much flux. By contrast, following northwestrepair's example and spreading with my finger, I never created a patch of too much flux, it was always all-or-nothing. I don't like messes but I can't argue with success. Finger technique is better, or at least easier.

Test your tongs BEFORE heating. None of my tweezers were wide enough to grip the chip and my tongs were too curved to do a good job. I eventually made a tool that worked by cutting a heavy silicone pad, but this was a terrible hack. This is something I should have sorted while I was ordering supplies.

As you can see below, I bought one of those new MiniHotPlates for this and it was delightful. As you heard from the lessons above, it took me a few tries but I eventually wound up with a reballed chip ready to install!

[jjoonathan]

Whoops, I see now that it was 36*36-1=1295 balls, not 1296 as I claimed before. I hope you will accept my humble apology for this egregious case of stolen valor.

[nctnico]

You are excused. Amazing work... 

[jjoonathan]

Haha, thanks!

Continuing with the story: I put the mainboard on the preheat plate, put thermocouples on both sides, and heated until the top was about 120C. I globbed on a ton of flux, carefully put the RTC chip in place, came in with the 800W hot air, pushed it up to 230C, and let it cool. High resolution, fast, responsive temperature readings from the DMM7510 were amazing. I'm sure you don't need a DMM7510 for these, but coarser/slower reading on the JBC station was by comparison almost completely useless to evaluate the temperature transients caused by angles/gestures/etc. I'm sure you don't need a JBC station either, my point is that you should invest in a fast and accurate thermocouple reader because it made waaay more of a difference than I thought going in. The reading on the hot air station bore little resemblance to the thermocouple reading, even if I stuck the thermocouple right at the tip of the nozzle, and the solder followed the thermocouple not the station. I don't know if this is a problem with my station or if air is just that bad at conducting heat and surface thermocouples are therefore much more necessary than I guessed, but there you go. Invest your money in a good thermocouple reader (I'm sure they can be had for far less than my repaired DMM7510, I didn't plan this just observed it) and invest your time in placing the thermocouples.

I checked the power rails and... ohmic short. Dammit. I suspect I know why, though: the globs of flux bubbled quite a bit while heating and when the solder balls melted and "schlurped" the chip down it was a rather violent event that ejected flux out the sides. I am not surprised that something bridged. Well, nothing to do but remove it, reball it, and try again with less flux. This time I painted the flux on with a brush immediately before putting the reballed chip in place. The balls melted, the chip aligned, and the ohmic short on the 1.2V rail disappeared! Yay!

I booted it up and... leafblower mode 

[jjoonathan]

Ok, debug time. The RTC chips have a 312.5MHz clock input that wasn't present if I powered the mainboard alone, because it turns out that it is derived from the 100MHz main clock sent up from the frontend board. Once I powered the boards together, 312.5MHz was visible on the input to both RTC chips.

The 312.5MHz clock is very familiar from this patent, which almost certainly describes the ADC/RTC interface: https://patents.google.com/patent/US7720108B2 . Also, this interface is AC coupled so at least the frontend board was protected from my DC mistakes.

Both RTC chips get 312.5MHz in, but only the non-replaced RTC chip sends a 400MHz clock to its DRAM. Hmmm. It has a current consumption profile that looks different, but this is probably just because it starts first. I don't like the way it drifts up, but that's hardly definitive.

What *is* definitive is the voltage monitoring. The R&S board designer really outdid themselves here: they brought every sundry power rail, even the little jellybean LDO rails, into a pair of 32-way AD muxes and they left a file in the firmware that records the names and nominal values of each one. Nice! The numbers even match up in the logical way, and they make it very quick to check all of the little rails. If you can boot the scope you can check through PCI Express, and I'm sure there is a UART backdoor or something, but even without knowing any of this you can just go from resistor to resitor on a multimeter and they are sorted in order of ascending voltage so you can tell when one is 0 and shouldn't be. How nice is that?

In any case, a 0.9V DDR termination rail wasn't coming up so that was the next thing to chase.

[jjoonathan]

So the 0.9V DDR termination rails are fed by these wacky little special-purpose LDOs TI 51100 with a 5V power rail that they don't use for power, pins designed to interpret sleep states from laptop 5V logic signals, a 1.2V power rail that they do use for power, and two 0.9V outputs, one of which terminates DDR command lines and one of which serves as a reference for the RTC chip to compare logic levels to.

The short was on the RTC side (uh-oh), so I disconnected the LDO and had my power supply feed 1A/100ms pulses (0.9V limit) into the shorted rail and had my multimeter report max voltage between cursors at about -1 and -3 seconds. I am sure there are short-hunting solutions out there with higher quality and lower price, so I'd be interested in hearing from anyone with more experience here, but my setup did produce the classic short-hunting dog-leg pattern, which I repeated from both sides to resolutely point the finger at a single short pad. I then removed the chip, masked off the short pad, reballed, and reinstalled. The 0.9V aux rail was no longer short, but its resistance didn't quite match the sibling rails (700 ohms, 700 ohms, 700 ohms, 400 ohms). Not good. Still, I applied 12V/12A to the mainboard + frontend and the 0.9 rail came up! Yay! But the 400MHz RAM clock didn't appear and the scope app stalled out waiting for the GB receivers.

I'm thinking this RTC chip is dead. When I reflowed it and bridged some of the balls, I must have pulled power through the chip and slagged important silicon. I didn't mention it before but there was discoloration next to the shorted pin that hints at some past excitement.

Fortunately, I accounted for this possibility. Remember earlier when I said "the seller made it right"? Well, I sent out a quote to R&S for a replacement mainboard some time ago, the quote came back, and for once it turned out that I had actually asked for enough money! My partial refund will cover the replacement mainboard.

[jjoonathan]

Bringing us to the point. There are two versions of the RTO2000 mainboard (at least). I have the old version with less memory. According to the declassification guide, there is a new version with more memory. I would like to quote this one too, because I want it  but I don't know the material number. If anyone has a RTO2000, especially one with the 1Gsa memory option, could you go to File>Maintenance>Mainboard  or Menu>Settings>Maintenance>Mainboard and tell me the Material Number? Here's mine (attached).

Also, I couldn't help but notice that a bunch of the numbers on the RTC chip match up exactly with those on the RTO1000. This is giving me some very bad ideas. I mean, how different could two custom silicon chips possibly be if some of the numbers on the package match up? Ebay prices are enough to knock some sense into me, but if I rummage around in the budget there is still a $400 difference between my ebay refund and the replacement mainboard quote, and if there are any other RTO1000 (or RTE) enjoyers out there with a half-dead mainboard that might contain a working RTC that they might be persuaded to part with for what is admittedly not a good price, perhaps we could see together where this dumb idea goes. Probably not, but I had to throw it out there.

That's all for now. I have video of most of this but gee, it was long enough in written form. Maybe once this has had a chance to settle I'll cobble something together and post it.