Author Topic: RTO2000 RAM Artifacts  (Read 860 times)

0 Members and 1 Guest are viewing this topic.

Offline jjoonathanTopic starter

  • Frequent Contributor
  • **
  • Posts: 813
  • Country: us
RTO2000 RAM Artifacts
« on: July 07, 2024, 01:28:05 am »
I bought a RTO2014 the other day. Beautiful scope, but it has RAM artifacts. The seller made it right, but now I have to go and actually fix it  :-/O

My first step was to identify the RAM chip with the artifact, and I did this by cutting data lines and observing where the errors showed up (and then repairing the cuts, of course). I didn't have especially good luck in guessing which chip was responsible, but the silver lining is that I now have a mostly complete picture of how data is deposited in the RAM. Even better, I have a procedure to share with the world so that the next person to have this problem can simply drop some cursors on the waveform artifacts and immediately know which chip is faulty.

The architecture is that 4 10GS/s channels feed two ASICs and each ASIC farms the samples from its channel pair out to 14 W9751G6KB SDRAM chips. This will be different if you have option B110 extended memory, because the math doesn't work out for B110 to use the same chips. Each chip has 16 data lines and every 100ps one of the DRAM chips will receive one 8 bit sample from ch1 and one 8bit sample from ch2. The chips will be used in sequence until, 14 samples later, the first DRAM chip receives its second pair of samples. The especially good news is that this pattern is stable with respect to the timebase: if you hit PRESET and then activate Channel 2 (Firmware 5.35.1.0), you can simply drop a cursor on the first artifact past (or on) the origin, measure the picoseconds, and look up the chip on the image I attached. Be sure to use the BLUE numbers; the yellow numbers are my own labeling scheme relative to a chip I arbitrarily chose as 0 and are offset 200ps from the BLUE numbers, which are the ones that correspond to "picoseconds past the origin".

Replacement chips are on order and while they slowly make their way to me I have begun to practice BGA rework. I've done a few one-offs on smaller chips, but this will be my first chip with this many balls on this thick of a board. My first practice attempts were on the cheapest DDR4 SODIMMs I could find, because the LED testers were available on Amazon. Those went well! I think I have the hang of reballing them now, and my success rate is 100% on 4 chips out of 4. My second set of practice attempts were on a chunky but obsolete graphics card and those didn't go nearly so well. My 450W preheat plate was not up to the task. I had to crank the temperature setting very high just to get one chip off, let alone remounted. The thermocouple I had on the back side was always much lower than the setpoint, even if I insulated with aluminum foil. I have an 850W station on order, hopefully that works better. In the meantime, here are some pictures and that timing diagram I was talking about!

EDIT: compression did a number on the blue numbers, here is a higher res version. Note that the dropbox preview is also compressed, download the PDF for full resolution. https://www.dropbox.com/scl/fi/1roh0w3hl585cxuaygs77/DRAM.pdf?rlkey=7vpw6q7kcfpk3d3kpr7usy2qq&st=clrch9hl&dl=0

EDIT2: diagnostic procedure in video form. If you want the repair story, look for that video a few posts down.



« Last Edit: July 07, 2024, 05:43:45 pm by jjoonathan »
 
The following users thanked this post: ddrl46, egonotto, tv84, hansibull, TurboTom, Qw3rtzuiop, 2N3055, pdenisowski, Martin72, RAQU_PL, Harry_22

Offline hansibull

  • Regular Contributor
  • *
  • Posts: 119
  • Country: no
Re: RTO2000 RAM Artifacts
« Reply #1 on: July 07, 2024, 07:07:27 am »
Thanks for the troubleshooting details! It was a joy to read. How much does one have to pay for a defective RTO2014?
 
The following users thanked this post: egonotto, jjoonathan

Offline pdenisowski

  • Frequent Contributor
  • **
  • Posts: 818
  • Country: us
  • Product Management Engineer, Rohde & Schwarz
    • Test and Measurement Fundamentals Playlist on the R&S YouTube channel
Re: RTO2000 RAM Artifacts
« Reply #2 on: July 07, 2024, 09:18:20 am »
Wow.  Just ... wow.  Very, very impressed.
Test and Measurement Fundamentals video series on the Rohde & Schwarz YouTube channel:  https://www.youtube.com/playlist?list=PLKxVoO5jUTlvsVtDcqrVn0ybqBVlLj2z8
 
The following users thanked this post: egonotto, jjoonathan

Offline tautech

  • Super Contributor
  • ***
  • Posts: 28966
  • Country: nz
  • Taupaki Technologies Ltd. Siglent Distributor NZ.
    • Taupaki Technologies Ltd.
Re: RTO2000 RAM Artifacts
« Reply #3 on: July 07, 2024, 10:17:19 am »
Is this a RAM SW management issue ?
Avid Rabid Hobbyist.
Siglent Youtube channel: https://www.youtube.com/@SiglentVideo/videos
 

Offline jjoonathanTopic starter

  • Frequent Contributor
  • **
  • Posts: 813
  • Country: us
Re: RTO2000 RAM Artifacts
« Reply #4 on: July 07, 2024, 05:41:39 pm »
I was going to hold the video until I actually went ahead with the replacement, but I have enough footage that it probably needs to be split into two parts. See Part 1 here (and wish me good luck on Part 2):



Re: is it a RAM SW issue? No, it's definitely a hardware problem. Jump to 16:55 in the video to see where I cut traces to induce similar faults in order to figure out which of the 14 plausible candidate DRAM chips is the culprit.

Re: how much did it cost? Five shares of NVDA at time of purchase ;) Whether or not this was a good deal depends considerably on whether or not the repair is ultimately successful.


 
The following users thanked this post: egonotto, tv84, TurboTom, 2N3055, Harry_22

Online Harry_22

  • Regular Contributor
  • *
  • Posts: 242
  • Country: hn
Re: RTO2000 RAM Artifacts
« Reply #5 on: July 07, 2024, 08:07:23 pm »
Hi!
Good job!
Definitely I will use it for R&S diagnostic.

I think that there will be no problems with the replacement if you warm up the board well. The board is not simple with a very good heat dissipation. Lead-free soldering complicates the situation.

My advise is to remove two radiators and thermally insulate the ends of PCB that protrude beyond the bottom heating.
 
The following users thanked this post: egonotto, jjoonathan

Online DaJMasta

  • Super Contributor
  • ***
  • Posts: 2324
  • Country: us
    • medpants.com
Re: RTO2000 RAM Artifacts
« Reply #6 on: July 07, 2024, 08:37:43 pm »
Impressive sleuthing!  Good luck on the repair!
 
The following users thanked this post: egonotto, jjoonathan

Online nctnico

  • Super Contributor
  • ***
  • Posts: 27423
  • Country: nl
    • NCT Developments
Re: RTO2000 RAM Artifacts
« Reply #7 on: July 07, 2024, 08:46:08 pm »
Hi!
Good job!
Definitely I will use it for R&S diagnostic.

I think that there will be no problems with the replacement if you warm up the board well. The board is not simple with a very good heat dissipation. Lead-free soldering complicates the situation.

My advise is to remove two radiators and thermally insulate the ends of PCB that protrude beyond the bottom heating.
A pre-heater may not be enough to put enough heat into the board. Last time I did a similar repair job on a 10 layer board, I had to use a 1500W paint stripper heatgun (from the under side of the board) to get the temperature of the board hot enough for the solder to melt and get the BGA soldered. The problem with BGAs is the air between board and package which prevents heat from a hot air gun from above to heat the board properly through a BGA package.
« Last Edit: July 07, 2024, 08:47:47 pm by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 
The following users thanked this post: egonotto, jjoonathan

Offline jjoonathanTopic starter

  • Frequent Contributor
  • **
  • Posts: 813
  • Country: us
Re: RTO2000 RAM Artifacts
« Reply #8 on: July 07, 2024, 09:27:51 pm »
Thanks for the tips. This tracks with what I am seeing from northwestrepair on youtube -- he specializes in repairing graphics cards, which have this same sort of "hot chip surrounded by DRAM on a thick board" layout, and heat is a constant struggle. The R&S board is 2mm thick! My practice GPU is only 1.7mm thick -- I don't know the copper fraction here, but I definitely have work to do before I am comfortable with the practice GPU.

Yep, heat sinks will have to come off, so I have ordered replacement thermal paste.

I should also figure out what temperatures I can get away with. My thought here is to pick a decoupling capacitor and use the hot tweezers to figure out where the melting point of the solder is, and to sample a corner of the board and figure out where the resist starts to rapidly degrade. I also want to experiment with insulating the board. Aluminum foil is traditional, but I have really had to stack up the layers for effect on the GPU so I want to experiment with silicone cookwear as well.

In any case, I have plenty of things to fill the time while the chips ship :)
 
The following users thanked this post: egonotto

Online nctnico

  • Super Contributor
  • ***
  • Posts: 27423
  • Country: nl
    • NCT Developments
Re: RTO2000 RAM Artifacts
« Reply #9 on: July 08, 2024, 03:51:59 pm »
I'm not quite sure you need to cover the board. The areas you don't heat, will cool down quickly. The only thing I typically protect (using kapton tape) are plastic parts which can melt. Another issue you'll likely encounter is warping of the board; I don't think this is avoidable. Also make sure to have ventilation as the board will likely start to smoke while doing the repair.

Worst desoldering job I ever did was removing a 486 with glued on heatsink from a motherboard.

Either way, this is going to be a tedious job with lots of room for error. The key is to heat the board gradually; don't keep the hot air pointed at one place and pray.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 
The following users thanked this post: egonotto

Online Harry_22

  • Regular Contributor
  • *
  • Posts: 242
  • Country: hn
Re: RTO2000 RAM Artifacts
« Reply #10 on: July 18, 2024, 12:01:52 pm »
Hi Jjoonathan!

In the meantime I made an extension for the front panel. With a length of 20 cm a slight moire is noticeable in the image.
Connector brand Amphenol 88946-102LF and 70233-111LF.

PS
Waiting for the good news.
« Last Edit: July 18, 2024, 12:04:39 pm by Harry_22 »
 
The following users thanked this post: egonotto, jjoonathan

Offline jjoonathanTopic starter

  • Frequent Contributor
  • **
  • Posts: 813
  • Country: us
Re: RTO2000 RAM Artifacts
« Reply #11 on: July 18, 2024, 05:40:15 pm »
Nice! Yeah, having an extension cable for the screen is a godsend. I'm glad they built one in on the RTO2000 and I remember it being a real pain on the RTO1000 so congrats on building one of your own!

The news right now is that I am waiting for thermal paste, which will arrive tomorrow  :D
 
The following users thanked this post: ddrl46, egonotto


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf