Agilent 16717A Comparator and ZoomChipSelTest failures

[keitheevblog]

Hi there,

I've got a misbehaving HP 16717A for my 16700A hp logic analyzer. The module had previously worked fine but has started throwing these test failures lately. I've been carefully removing the plastic runners from all my modules, cleaning them using IPA, applying a conformal coat on the runner areas, and then reapply the runners. Using a 3M ESD-safe electronics vacuum, I vacuumed both sides.

This particular module had the runners and old adhesive removed, and the area cleaned with IPA. A nearby 25 mil test pad looked dull and crusty, so I used a gentle brush with IPA. That didn't really do it, so I applied Deoxit D5 to the area, scrubbed to no real avail. I thought that the cleaning broke the trace on one of the sides of the pad, so I carefully scraped the soldermask off the trace, and measured a good tone between either side. And all the way to the yellow 16-pin (4816P-B07, film resistor, I think) packs. The trace isn't broken.

The blade now fails self-testing with cmpTest (Comparator test), and the zoomChipSelTest (Zoom Acquisition Chip Select Test)

Testing using pv with "debug d=9 r=9"

Code: [Select]

pv> x cmpTest
  Check POD1 Thresholds:
    Slot C, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot C, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot C, Chip 9: B ....BB.. .B.BBBBB  . BBBB..BB B.B.....  Cal Clk Levels
    Slot C, Chip 8: . BBBB.BBB ........  . B....... ........  Cal Clk Levels
    Slot C, Chip 9: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
    Slot C, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD2 Thresholds:
    Slot C, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot C, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot C, Chip 9: . BBBB..BB B..B..B.  B ........ .BB.B...  Cal Clk Levels
    Slot C, Chip 8: . BBBBBBBB ........  . ........ .BB.BB..  Cal Clk Levels
    Slot C, Chip 9: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot C, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD3 Thresholds:
    Slot C, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot C, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot C, Chip 9: . BB...... ........  . BBBB..BB ........  Cal Clk Levels
    Slot C, Chip 8: . B...BB.. ....B.B.  B BBBBBBBB BBBBBBBB  Cal Clk Levels
    Slot C, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot C, Chip 8: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
  Check POD4 Thresholds:
    Slot C, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot C, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot C, Chip 9: B ........ .BB.BB..  B BBB....B ........  Cal Clk Levels
    Slot C, Chip 8: . BBBBBBBB B..B.BBB  B ........ .B......  Cal Clk Levels
    Slot C, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot C, Chip 8: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
> Slot C: Comparator Test Failed!
Mod   C: TEST FAILED       # "cmpTest" (2, 2, -1)

So those are U8 and U9, which are the HP 1NB4-5040 ASICS. A normal passing test has "all decimal points." Not sure what the B indicates -- but it sure looks like something's hosed.

Next, the Zoom test fails spectacularly, seeing lines like

Code: [Select]

      Actual = 0x8808, Expected = 0xadff
      Actual = 0x8808, Expected = 0xadff
      Actual = 0x8808, Expected = 0xadff
     Slot E: FISO #2 failed.
    Slot E: Checking FISO #1...
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a

and

Code: [Select]

      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
     Slot E: FISO #3 failed.
    Slot E: Checking FISO #2...
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a

While sadly I think I broke my kind-of-expensive board, what the heck do you think I did? Did some part fail? Deoxit get stuck underneath a chip?

I don't have the skill or experience (access, or tools, honestly) to probe this board live. I'm just about ok writing this thing off, but wouldn't mind learning from my mistake.

I use a 3M mat with wriststrap religiously, careful with how I hold the board, what I touch.....I don't think it was ESD, but who knows?

Thanks for any guesses or manageable next steps.
Keith

[MadTux]

Those Deoxit sprays are cancer to electronics. The acid in these first dissolves oxide layer and then eats away good material.

I'd put that board into dishwasher or otherwise clean it with plenty of water and blow it dry afterwards with compressed air to get rid of any Deoxit residues

[keitheevblog]

I'd put that board into dishwasher or otherwise clean it with plenty of water and blow it dry afterwards with compressed air to get rid of any Deoxit residues

Sounds like a good idea. I was using IPA after the deoxit, but even the smallest spray on the lightest setting it goes all over.

[MarkL]

Hi Keith,

Does the self-test result in the same failed bit pattern every time?  Just wondering if something might be floating and settling to random values during each test.

It's been my experience that the first thing "pv" reports is bad is the thing to go after first.  The tests displayed after that could be a consequence of the first failure.

If it's the comparator test, you might want to look closely around U34, which is on the top side near the bottom edge.  It's an AD7841AS octal DAC (44-pin QFP) which provides the threshold voltages to the 1NB4-5036 comparators near the connectors.  It's parallel load, so perhaps something has happened to nCS, or the bus/address lines leading to the chip.  Also verify power and ref input pins are sane.

Could be anything with these boards, but at least it's a place to start.

EDIT: Fixed minor typo.

[keitheevblog]

Thanks much for chiming in Mark! I was about to ping @doricloon or @docben or some of those guys in the other thread, too!

Yes, the comparator test is the first one to fail. I'm almost certain that the Zoom test is failing as a result: from the service guide 16715-97003,

"Zoom Acquisition Test. The Zoom Acquisition Test verifies the data inputs to the 2GHz TimingZoom acquisition memory and that the TimingZoom acquisition clock is at the correct sampling frequency. Test data is created by clocking the comparators test port. "

So if comparators are failing, then this test would fail as well.

Maybe this naive question, but what's the best way to measure these voltages live? I've got some wirewrapping wire(AWG30) that I could solder to the pins in question on the DAC, and then run them out the side of the chassis. I don't see any other way to probe them in-situ.

There's nothing obviously wrong at first glance but I have to dig in. I'll rerun some tests and get some more data points.

Thanks

[keitheevblog]

I ran the cmpTest three times back-to-back. Here's a comparison (differences are highlighted in yellow) between the three of them.

The bit pattern is NOT the same.

Looked under 10x to the DAC, and all traces leaving the DAC. Everything "looks" healthy....at least to the eye.

I still have to figure out how to probe these..... some of those micro smd grabbers seem like they might work. The nice ones seem pretty pricey.

Thanks

EDIT: I guess it looks like the "Cal Clk Levels" are the only ones that differ. Not the "Cal Clk No Act." or the "Cal Clk Activity" --- not sure what the heck that means, but it does look consistent!

[MarkL]

On the probing, I've been fairly lazy and resort to a few methods (at least on a 16702B chassis since this is all I have):

0) Remove all the other modules.  They only get in the way and increase boot times.

1) Put the module in the bottom slot E.  Turn the whole unit upside down and remove the bottom cover.  Relocate the mouse/keyboard board elsewhere.  This gives you full access to the bottom of the module.  With some luck, the signal you want to probe makes an appearance on the bottom via a trace or via, which you can also verify with a continuity tester.

2) As you suggested, solder wire-wrap jumpers to probe points on the top and lead them out of the chassis.  I usually solder them to a 0.1" header so they're not flapping around.

3) Remove the front plate on the module.  This gives you somewhat better access to components near the edge, like the comparators, but the deeper the probe point, the harder it is to get in there.  If you remove the front plate, give yourself some pull loops through the empty holes to get the board back out.  Your knuckles will thank you.  Removing the front plate also eliminates the drudgery and wear/tear on the thumb screws, so you might want to do this anyway for faster insertion and removal.

4) Connect a 16701A/B expander chassis and remove the top and bottom covers on it, and put the module in slot J.  The advantage of the 16701 is that there isn't a large processor card obscuring access to the entire top of the module area.  There is, however, a smaller I/O card which is about half the module length, but access to the top of the module is at least better.  You may also want to relocate the expansion connector and its rather wide ribbon cable.  16701 expanders can occasionally be found very cheap because no one wants them, or the seller thinks they're broken because they plug them in and the power doesn't come on (power is controlled by the main chassis).  I got mine for around $50.

5) I do have a handful of the rather pricey micro-grabbers which I use occasionally, especially if I know it's a high speed signal that won't make it very far over a wire-wrap wire with much integrity.  Everyone should have a few.  But I usually end up doing most probing with wire-wrap jumpers, the main reason being that the micro-grabbers don't tolerate being jostled when repeatedly removing/inserting the card in the debugging process.  Taping down the wires leading to the micro-grabbers probe wires with Kapton helps.  Also, with the wire-wrap wire, I have more of that than micro-grabbers, so I tend to leave all the jumpers in place as I go in case I need to go back to anything.


What I've really been meaning to do is build a small right-angle adapter board to make the module stick straight out of the bottom.  Photo attached of what I mean, at least of the orientation.  Then both sides are accessible for probing.  Some additional support is likely needed.  And ideally, I would build two of them and use one in a second chassis (16700/16702, or 16701 expander) so that I could do real-time comparisons on good/bad modules of the same type without having to reboot to swap boards.  As you know, the boot time is rather terrible. 

I will look at your pv data to see if anything jumps out at me and maybe try a few things on the one 16717A I have here if I get any ideas.

EDIT: Typo.

[keitheevblog]

Hey MarkL,

I just wanted to say thank you very much for taking the time to post those "access" instructions. I managed to figure out a couple.....I did remove the faceplates, although unsurprisingly struggled to pull the board afterwards....good tip with the zipties.

What brand of micro-grabbers do you like? Usually I'm just looking for an excuse to supplement my toolbox anyways. One of my personal pet peeves is needing a tool, wire, adapter, cable, and just not having it on hand. Being an impatient gen x'er, the 2-3 day wait is miserable.

I've been toying with this moldable plastic. You heat up the plastic in boiling water, and then it can be formed for a a few minutes after. I'm contemplating making some sort of wire-holder adapter which would space wire-wrap wires 0.8mm apart to hold them in place, and allow something more substantial to be kapton taped to a board.... hrrmmmm.

I think the fact that only the "Calibration Clock Levels" differ on that debug output means something. I just haven't figure out what yet.

Thanks
Keith

[MarkL]

...
What brand of micro-grabbers do you like? Usually I'm just looking for an excuse to supplement my toolbox anyways. One of my personal pet peeves is needing a tool, wire, adapter, cable, and just not having it on hand. Being an impatient gen x'er, the 2-3 day wait is miserable.

Hi Keith,

I like the Mechano micro-grabbers.  There was thread on this a while back that might be useful.  My post, amongst other recommendations:

  https://www.eevblog.com/forum/projects/really-small-smd-grabbers/msg2021335/#msg2021335

HPAK calls them "QFP clips".  Keysight part #0960-2992 (long) and #0960-2995 (short), but don't order from them.  Their pricing is insane.  I got mine with a pair of complete probe kits (N2877A).  I suspect, but don't know for a fact, that Mechano is the actual manufacturer of all the look-a-likes that are resold through many channels.

There's also this clip which is good for passives and differential probing:

  https://www.eevblog.com/forum/testgear/best-way-to-securely-attch-probe-to-a-passive-smd-component/msg1310826/#msg1310826

Made by Hirschmann SKS, part #972416100:

  http://datasheet.octopart.com/972416100-Hirschmann-datasheet-21383182.pdf

I've been toying with this moldable plastic. You heat up the plastic in boiling water, and then it can be formed for a a few minutes after. I'm contemplating making some sort of wire-holder adapter which would space wire-wrap wires 0.8mm apart to hold them in place, and allow something more substantial to be kapton taped to a board.... hrrmmmm.

Sounds somewhat similar to the "SMD wedge adapter", which is also mentioned in that first thread:

  https://www.eevblog.com/forum/projects/really-small-smd-grabbers/msg2021632/#msg2021632

I have a few of the 3-pin versions in both 0.5mm and 0.65mm.   I've used them only occasionally for 167xx modules because they don't stay put reliably during card insertion/removal.  For stationary probing they're fine.

I think the fact that only the "Calibration Clock Levels" differ on that debug output means something.[/b] I just haven't figure out what yet.

I will probe some signals around the comparator on a good 16717A and post.  Maybe that will at least help point you in a direction.  Offhand it smells like a data path problem, but who knows.

[MarkL]

Ok, here's some things to look at on the comparator.  I've focused on comparator U69 (1NB4-5036) since it can be accessed from the bottom and has CLK from the pod on it.

For reference, here's the comparator to pod mapping:

16717A AD7841ASZ DAC to 1NB4-5036 comparator mapping

  DAC (pin)    Comparator   Pod/Bits (Clock)
  ----------   ----------   ----------------
  VoutA (2)        U70       1/7:0
  VoutB (44)       U69       1/15:8 (J)
  VoutC (43)       U4        2/7:0
  VoutD (41)       U3        2/15:8 (K)
  VoutE (37)       U72       3/7:0
  VoutF (35)       U71       3/15:8 (L)
  VoutG (34)       U6        4/7:0
  VoutH (32)       U5        4/15:8 (M)


It's easier if you have a script run the pv "cmpTest" for you repeatedly.  Here is a script to run the test every second:

Code: [Select]

#!/usr/local/bin/bash

(
  echo "s e"
  echo "debug d=9 r=9"

  while true; do
    echo "x cmpTest"
    sleep 1
  done

) | pv
If you don't have bash loaded, I'd highly recommend it.  Otherwise you can adjust the script for the HPUX default shell (sh?).

I think I've figured out most of the pins on the 1NB4-5036.  Here's my current best determinations (corrections welcome):

1NB4-5036 comparator pins
  based on use in 16717A module

   1  CmpIn
   2  -5.1V (decoupled to GND)
   3  CmpIn
   4  -3.2V (decoupled to GND)
   5  CmpIn (on U69 is clock in from pod 1, input voltage/10)
   6  GND
   7  CmpOut from pin 5
   8  nCmpOut from pin 5
   9  +3.4V
  10  CmpOut
  11  +3.4V (decoupled to GND)
  12  CmpOut
  13  +3.4V
  14  CmpOut
  15  +3.4V (decoupled to GND)
  16  CmpOut
  17  +3.4V (decoupled to GND)
  18  Low when test, goes 3.4V to ground, slow rise (probably a pullup)
      common to all comps
  19  no visible copper (n/c?), +2.1V steady
  20  +3.4V (decoupled to GND)
  21  CmpOut
  22  +3.4V (decoupled to GND)
  23  CmpOut
  24  +3.4V
  25  CmpOut
  26  +3.4V (decoupled to GND)
  27  CmpOut
  28  GND
  29  GND
  30  -3.2V (decoupled to GND)
  31  CmpIn
  32  -5.1V (decoupled to GND)
  33  CmpIn
  34  +3.4V (decoupled to GND)
  35  CmpIn
  36  GND
  37  50Mhz clock input, +/-5mV square wave during test
      common to all comps
  38  GND
  39  Vref
  40  +3.4V (decoupled to GND)
  41  CmpIn
  42  GND
  43  CmpIn
  44  +3.4V (decoupled to GND)

-5.1V is must be power

+3.4V probably power, is also on a couple of LVC08
      (on the top, U77 U78)

-3.2V could be power or bias

Vref is 136mV when set to TTL trig level
   +87mV when set to +1.0V
   -11mV when set to  0.0V
  -110mV when set to -1.0V


The comparison voltage (Vref) is pin 39, and it used as one input to all 9 comparator channels in the chip.  Each pod input is compared to Vref.  I didn't bother mapping each input and output (except CLK) since that's not the problem here.  Feel free to complete the table for future reference.

The test appears to proceed as described in the service manual.

A test clock is first applied to all the comparator chips on pin 37.  It's a 50MHz +/-5mV 50% duty cycle square wave.  When it starts it doesn't actually stabilize at 50MHz until after about 13us or so.

A low on pin 18 puts the chip in test mode.  In test mode the clock input is internally applied to all comparator inputs instead of the external pod inputs.  To perform the actual test, the system moves Vref above the clock, below the clock, and then to 0V.  This generates comparator comparator outputs of 1, 0, and then 50MHz.

You should probably also make sure +3.4V, -3.2V, and -5.1V are ok.

Each comparator output looks like it's split into two different voltage levels by whatever is inside the yellow resistor networks adjacent to them.  One set of signals goes to acquisition ASICs (U21 U22), and the other set goes to the zoom ASICs 1NB4-5040 (U8 U9 U10 U11 U12).  Perhaps this commonality is a clue since the zoom test is failing also.

The scope traces are labeled with the pin number.

[MarkL]

A minor update...

At some point during my probing, I killed one or both of my U69 and U70.  I didn't notice the board had started failing at some point since the PC driving the test was happily sitting in a loop on a different bench.  I'm sure it was the result of a slipped probe shorting out two pins somewhere.

I also noticed that U69.Test_Clk_CmpOut_zoom.png was not showing a nice sharp edge for the comparator output (pin 7).  It's a super fast comparator, and this would not do for the signal speeds this board is capable of.

Oh well, I've done worse.  Both comparators replaced.  Maybe only one of them was dead, but whatever it was it affected all of them.  No doubt it was interfering with one of the test signals common to all.

Below is a re-capture of the Test/Vref/CmpOut pins, U69.Test_CmpOut_Vref_fixed.png.  Nice sharp edges now with a fair amount of ringing on the rising edge.  Note the swing is now between 1.7V and 2.5V.  With the dead comparator and previous screen capture it was not at these levels.

U69.Test_CmpOut_Vref_start.png shows that the CmpOut duty cycle does not quite track the 50% test clock.  This is because Vref has just entered the trip point for the comparator.  As Vref gets closer to 0V, the output approaches 50%, as it should, as shown in U69.Test_CmpOut_Vref_after_20us.png.

This is more info than you need, but since I had to troubleshoot my board, I thought I'd post it for future reference in case anyone else lands on this thread looking for LA comparator info.

Also attached is a shot of the empty footprint for a comparator.  They are all seem to be the same, at least on the bottom.  In my 1NB4-5036 pinout description, I wasn't sure where pin 19 goes.  Under the chip I now see it goes into a via, but I can't find the other end of the via anywhere on the board.  It's still an unknown.


So...  Be careful probing, and use the little slotted protector caps that keep the probe tip on one leg at a time.

[MarkL]

We seem to be done with this thread, but I have a couple of minor updates before I forget.

Below is my updated notes on the 1NB4-5036 comparator pins after I could see the traces underneath.  It also occurred to me that the comparator pins I couldn't trace are probably not a N/C, but are likely going to one of the large BGA ASICs on an inner layer.

I also forgot to post my analysis of what's inside the 16-pin yellow resistor packs that's next to the comparators.  Attached.

Just puttin' the info out there.

Reverse engineering always involves some amount of guessing.  Corrections welcome.



1NB4-5036 comparator pins
  based on use in 16717A module

  ---------
   1  CmpIn
   2  -5.1V (decoupled to GND)
   3  CmpIn
   4  -3.2V (decoupled to GND)
   5  CmpIn (on U69 is clock in from pod 1, input voltage/10)
   6  GND
   7  CmpOut from pin 5
   8  nCmpOut from pin 5
   9  +3.4V
  10  CmpOut
  11  +3.4V (decoupled to GND)
  ---------
  12  CmpOut
  13  +3.4V
  14  CmpOut
  15  +3.4V (decoupled to GND)
  16  CmpOut
  17  +3.4V (decoupled to GND)
  18  Low when test, goes 3.4V to ground, slow rise (probably a pullup)
      common to all comps
  19  via under chip, +2.1V steady
      not common to other comps
      can't find other end, comes from big ASIC?
  20  +3.4V (decoupled to GND)
  21  CmpOut
  22  +3.4V (decoupled to GND)
  ---------
  23  CmpOut
  24  +3.4V     
  25  CmpOut
  26  +3.4V (decoupled to GND)
  27  CmpOut
  28  GND
  29  GND
  30  -3.2V (decoupled to GND)
  31  CmpIn
  32  -5.1V (decoupled to GND)
  33  CmpIn
  ---------
  34  +3.4V (decoupled to GND)
  35  CmpIn
  36  GND
  37  50Mhz clock input, +/-5mV square wave during test
      NOT common to all comps
      source clock split 4 ways with cap/res to get 5mV
      source clock common between left 4 and right 4
      can't find source end, each side comes from big ASIC?
  38  GND
  39  Vref
  40  +3.4V (decoupled to GND)
  41  CmpIn
  42  GND
  43  CmpIn
  44  +3.4V (decoupled to GND)
  ---------

-5.1V must be power

+3.4V probably power, is also Vdd on a couple of
      LVC08 on the top, U77 U78

-3.2V could be power or bias

Vref is 136mV when set to TTL trig level
   +87mV when set to +1.0V
   -11mV when set to  0.0V
  -110mV when set to -1.0V

[fisafisa]

hi guys, did you solve the problem?
Recently I worked on a similar fault and solved it.
The boards were of the 16740 family.
The problem turned out to be a resistor of 7.5K that failed open in some channels and in others had a larger value.
This issue seems it was common to all comparators on the top of the board.
Some failed during testing.
I ended up replacing all 4 on both boards.
The resistor was used in a voltage partitor to generate the reference voltage in pin 39.
When correct, in circuit the resistor measures 7.05k.
The last I changed was measuring >8K and was on a pod were I had failures associated to the zoomChip.
The pod itself seemed to work fine.
After the change all the test passed again.

F

[MarkL]

hi guys, did you solve the problem?

I think Keith will have to answer that.  He sent me a couple of cards for further investigation and repair, but we concluded the above debug info was not from one of them.

Recently I worked on a similar fault and solved it.
...

There was another person who had a bad 7k5 resistor at the comparator input, same as your board:

  https://www.eevblog.com/forum/repair/series-defect-on-agilent-167xx-boards/msg2719080/#msg2719080

I've found a lot of bad resistors on 165xx/167xx series boards (I'd say on the order of 15 on 20 boards), and in all parts of the circuitry.  Some resistors have been off, but most completely open.  And a few have been associated with setting the output of a voltage regulator, which was particularly bad.

I have a bunch of other test equipment from HP/Agilent from this same era, and I've never seen so many bad resistors.  The bad resistors are usually, but not always, near an area of corrosion.  I can't say the corrosion caused it, but it is a suspicious coincidence.

[MarkL]

Here is an update to the 1NB4-5036 pin usage posted above.  There are a couple of changes:

- Pin 29 is an input (not a GND)

- The test clock input is 50mV (not 5mV)

Didn't want to leave bad info out there.  Thank you to user shakalnokturn.

1NB4-5036 comparator pins
  44-pin LQFP, 0.8mm pitch
  based on use in 16717A module

  ---------
   1  CmpIn
   2  -5.1V (decoupled to GND)
   3  CmpIn
   4  -3.2V (decoupled to GND)
   5  CmpIn (on U69 is clock in from pod 1, input voltage/10)
   6  GND
   7  CmpOut from pin 5
   8  nCmpOut from pin 5
   9  +3.4V
  10  CmpOut
  11  +3.4V (decoupled to GND)
  ---------
  12  CmpOut
  13  +3.4V
  14  CmpOut
  15  +3.4V (decoupled to GND)
  16  CmpOut
  17  +3.4V (decoupled to GND)
  18  Low when test, goes 3.4V to ground, slow rise (probably a pullup)
      common to all comps
  19  via under chip, +2.1V steady
      not common to other comps
      can't find other end, comes from big ASIC?
  20  +3.4V (decoupled to GND)
  21  CmpOut
  22  +3.4V (decoupled to GND)
  ---------
  23  CmpOut
  24  +3.4V
  25  CmpOut
  26  +3.4V (decoupled to GND)
  27  CmpOut
  28  GND
  29  CmpIn (previously said GND - wrong!)
  30  -3.2V (decoupled to GND)
  31  CmpIn
  32  -5.1V (decoupled to GND)
  33  CmpIn
  ---------
  34  +3.4V (decoupled to GND)
  35  CmpIn
  36  GND
  37  50Mhz clock input, +/-50mV (approx) square wave during test
      NOT common to all comps
      source clock split 4 ways with cap/res to get 50mV
      source clock common between left 4 and right 4
      can't find source end, each side comes from big ASIC?
  38  GND
  39  Vref
  40  +3.4V (decoupled to GND)
  41  CmpIn
  42  GND
  43  CmpIn
  44  +3.4V (decoupled to GND)
  ---------

-5.1V must be power

+3.4V probably power, is also Vdd on a couple of
      LVC08 on the top, U77 U78

-3.2V could be power or bias

Vref is 136mV when set to TTL trig level
   +87mV when set to +1.0V
   -11mV when set to  0.0V
  -110mV when set to -1.0V

[mbalmer]

I've recently come across this information, too, and I'm trying to diagnose a board with similar issues -- failing the Comparators and ZoomChipSelTest tests.

I've removed the adhesive plastic runners and cleaned heavily with IPA, but need some guidance as to which chips are which when it comes to the test.

I recognize the eight 1NB4-5036 comparator chips, but it's unclear which chips the test calls "chip 8" and "chip 9". I'm also curious to know if anything came of either Keith's or Mark's testing to revive their failing boards.

I do have a bunch of test results from running pv on the analyzer itself, but I'm not sure how helpful that will be.

Any help would be appreciated.

[MarkL]

What model board do you have?  A 16717A or something else?  There's a large number of models that have the same basic control circuitry and software.

Assuming for the moment a 16717A, "chip 9" is U21 and "chip 8" is U22.  These are the two large ASICs with heatsinks.  Each ASIC is responsible for half the pods.  As the pv tests are running, it's telling you which ASIC, and which half of the board, the failure is associated with.  (I'm guessing those chip "names" were established when the system was originally being developed.  They didn't bother mapping them to real U numbers since pv output was for internal use only.)

Usually the first failure reported by pv is the one to investigate first.  pv does not understand failure dependencies, so even if a failure occurs in one test, pv may try to use that failed sub-system to verify other sub-systems in subsequent tests, and those tests will fail too.

The output results from pv with debug turned on can be very useful.  Please post the pv results from the "all tests" summary (no debug turned on), and then the output with all debug turned on from manually running the first failing test.

[mbalmer]

The board in question is a 16717A, and I have five more of them in varying states of (dis)repair.

That said, here's the results:

Test results from x modtests. Mod D is the device under test.

Code: [Select]

Mod   D: TEST passed       # "vramDataTest" (2, 0, 1)
Mod   D: TEST passed       # "vramAddrTest" (2, 0, 1)
Mod   D: TEST passed       # "vramCellTest" (2, 0, 1)
Mod   D: TEST passed       # "vramUnloadTest" (2, 0, 1)
Mod   D: TEST passed       # "chipRegTest" (2, 0, 1)
Mod   D: TEST passed       # "bpClkTest" (2, 0, 1)
Mod   D: TEST FAILED       # "cmpTest" (2, 2, -1)
Mod   D: TEST passed       # "icrTest" (2, 0, 1)
Mod   D: TEST passed       # "flagTest" (2, 0, 1)
Mod   D: TEST passed       # "armTest" (2, 0, 1)
Mod   D: TEST passed       # "vramSerDataTest" (2, 0, 1)
Mod   D: TEST passed       # "vramSerCellTest" (2, 0, 1)
Mod   D: TEST passed       # "clksTest" (2, 0, 1)
Mod   D: TEST passed       # "calTest" (2, 0, 1)
Mod   D: TEST passed       # "zoomDataTest" (2, 0, 1)
Mod   D: TEST passed       # "zoomMasterTest" (2, 0, 1)
Mod   D: TEST passed       # "fisoRedundancyTest" (2, 0, 1)
Mod   D: TEST passed       # "zoomAcqTest" (2, 0, 1)
Mod   D: TEST FAILED       # "zoomChipSelTest" (2, 2, -1)

Results from tests using x cmpTest and x zoomChipSelTest with debugLevel set to 9 and resultLevel set to 9. There are 4 successive tests shown below.

Code: [Select]

  Check POD1 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: B B..BBBBB BBBBBBBB  . ........ ........  Cal Clk Levels
    Slot D, Chip 8: B B....B.. .BBBBBBB  B B...BBB. .BBBBBBB  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD2 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: B ........ BBBBBBBB  B BBBBB.BB ........  Cal Clk Levels
    Slot D, Chip 8: B BBBBBBBB B....BBB  B BBBBBBBB B.....B.  Cal Clk Levels
    Slot D, Chip 9: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD3 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . ........ ........  . ........ .B.B....  Cal Clk Levels
    Slot D, Chip 8: . BBB..... ........  B B..BBBBB BBBBBBBB  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
  Check POD4 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . .BB..... ........  . .B...... ........  Cal Clk Levels
    Slot D, Chip 8: . ........ .BBB....  B BBBBBBBB B...BBBB  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
> Slot D: Comparator Test Failed!
  Check POD1 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Levels
    Slot D, Chip 8: B B....B.. .BBBBBBB  B B...BBB. .BBBBBBB  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD2 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . BBBBB.BB B.......  B B...BBB. BBBBBBBB  Cal Clk Levels
    Slot D, Chip 8: . BBBB.B.B ........  . BBBB...B ........  Cal Clk Levels
    Slot D, Chip 9: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD3 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: B BBBBBBBB B....BBB  B BBBBBBBB B....BBB  Cal Clk Levels
    Slot D, Chip 8: . BBBBBBBB ........  B ....B.B. BBBBBBBB  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
  Check POD4 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . BBBBB..B ........  . BBBB.... ........  Cal Clk Levels
    Slot D, Chip 8: B BBBBBBBB B....BBB  . ........ .BBBB...  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
> Slot D: Comparator Test Failed!
  Check POD1 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . .BB..... ........  B B.BBBBBB BBBBBBBB  Cal Clk Levels
    Slot D, Chip 8: B ........ .BBBBBB.  . .....B.. .BBBBBBB  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD2 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . ........ .BBBB...  B BBBBBBBB B....BBB  Cal Clk Levels
    Slot D, Chip 8: B BBBBBBBB B...BBBB  B BBBBBBBB B...BBBB  Cal Clk Levels
    Slot D, Chip 9: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD3 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . BBBBB..B ........  . BBB..... ........  Cal Clk Levels
    Slot D, Chip 8: . ........ .BBBB...  B BBBBBBBB B....BBB  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
  Check POD4 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: B ........ .BBBB...  . ........ .BBBB...  Cal Clk Levels
    Slot D, Chip 8: . .B...... ........  B BBBBBBBB BBBBBBBB  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
> Slot D: Comparator Test Failed!
  Check POD1 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . BBBBBBBB .....B.B  . ........ BBBBBBB.  Cal Clk Levels
    Slot D, Chip 8: . .BB..... ........  . .B...... ........  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD2 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . ........ .BBBB...  B BBBBBBBB B....BBB  Cal Clk Levels
    Slot D, Chip 8: B BBBBBBBB B...BBBB  B BBBBBBBB B...BBBB  Cal Clk Levels
    Slot D, Chip 9: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD3 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: B B...BBBB BBBBBBBB  B B..BBBBB BBBBBBBB  Cal Clk Levels
    Slot D, Chip 8: B BBBBBBBB B...BBBB  . ........ .BBB....  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
  Check POD4 Thresholds:
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 8: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk No Act.
    Slot D, Chip 9: . BBBBBBBB B....BBB  B BBBBBBBB B....B.B  Cal Clk Levels
    Slot D, Chip 8: . BBBBBBBB B.......  B B...BBB. .BBBBBBB  Cal Clk Levels
    Slot D, Chip 9: B BBBBBBBB BBBBBBBB  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot D, Chip 8: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
> Slot D: Comparator Test Failed!

After running this test, I did another 'x modtests' -- and the "vramSerDataTest" failed for the first time since plugging in the module, which makes me concerned that any corrosion internal to the board is getting worse. I subsequently stopped the test sequence and then re-ran the vramSerDataTest another 100 times -- it passed 100 out of 100, so maybe it's a fluke.

Next is the results of two runs of x zoomChipSelTest with debugLevel set to 9 and resultLevel set to 9.

Code: [Select]

  Slot D: Filling FISOs for Pods #1 with zeroes...
    Slot D: Checking FISO #4...
    Slot D: Checking FISO #3...
    Slot D: Checking FISO #2...
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
     Slot D: FISO #2 failed.
    Slot D: Checking FISO #1...
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
     Slot D: FISO #1 failed.
    Slot D: Checking FISO #0...
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
     Slot D: FISO #0 failed.
  Slot D: Filling FISOs for Pods #2 with zeroes...
    Slot D: Checking FISO #4...
    Slot D: Checking FISO #3...
    Slot D: Checking FISO #2...
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
      Actual = 0xffff, Expected = 0xdaff
     Slot D: FISO #2 failed.
    Slot D: Checking FISO #1...
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
     Slot D: FISO #1 failed.
    Slot D: Checking FISO #0...
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
     Slot D: FISO #0 failed.
  Slot D: Filling FISOs for Pods #3 with zeroes...
    Slot D: Checking FISO #4...
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
     Slot D: FISO #4 failed.
    Slot D: Checking FISO #3...
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
     Slot D: FISO #3 failed.
    Slot D: Checking FISO #2...
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
     Slot D: FISO #2 failed.
    Slot D: Checking FISO #1...
    Slot D: Checking FISO #0...
  Slot D: Filling FISOs for Pods #4 with zeroes...
    Slot D: Checking FISO #4...
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
      Actual = 0x888, Expected = 0xadda
     Slot D: FISO #4 failed.
    Slot D: Checking FISO #3...
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
      Actual = 0x808, Expected = 0xad5a
     Slot D: FISO #3 failed.
    Slot D: Checking FISO #2...
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
      Actual = 0x8808, Expected = 0xff5a
     Slot D: FISO #2 failed.
    Slot D: Checking FISO #1...
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
     Slot D: FISO #1 failed.
    Slot D: Checking FISO #0...
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
     Slot D: FISO #0 failed.
> Slot D: Zoom Acquisition Chip Select Test Failed!
  Slot D: Filling FISOs for Pods #1 with zeroes...
    Slot D: Checking FISO #4...
    Slot D: Checking FISO #3...
    Slot D: Checking FISO #2...
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
      Actual = 0xffff, Expected = 0xadff
     Slot D: FISO #2 failed.
    Slot D: Checking FISO #1...
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
     Slot D: FISO #1 failed.
    Slot D: Checking FISO #0...
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
     Slot D: FISO #0 failed.
  Slot D: Filling FISOs for Pods #2 with zeroes...
    Slot D: Checking FISO #4...
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
      Actual = 0x888, Expected = 0xffff
     Slot D: FISO #4 failed.
    Slot D: Checking FISO #3...
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
      Actual = 0x808, Expected = 0xffff
     Slot D: FISO #3 failed.
    Slot D: Checking FISO #2...
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
      Actual = 0x8808, Expected = 0xdaff
     Slot D: FISO #2 failed.
    Slot D: Checking FISO #1...
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
     Slot D: FISO #1 failed.
    Slot D: Checking FISO #0...
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
      Actual = 0x808, Expected = 0x5aad
     Slot D: FISO #0 failed.
  Slot D: Filling FISOs for Pods #3 with zeroes...
    Slot D: Checking FISO #4...
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
     Slot D: FISO #4 failed.
    Slot D: Checking FISO #3...
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
      Actual = 0xffff, Expected = 0x5aad
     Slot D: FISO #3 failed.
    Slot D: Checking FISO #2...
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
      Actual = 0xffff, Expected = 0xffad
     Slot D: FISO #2 failed.
    Slot D: Checking FISO #1...
    Slot D: Checking FISO #0...
  Slot D: Filling FISOs for Pods #4 with zeroes...
    Slot D: Checking FISO #4...
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
      Actual = 0xffff, Expected = 0xadda
     Slot D: FISO #4 failed.
    Slot D: Checking FISO #3...
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
      Actual = 0xffff, Expected = 0xad5a
     Slot D: FISO #3 failed.
    Slot D: Checking FISO #2...
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
      Actual = 0xffff, Expected = 0xff5a
     Slot D: FISO #2 failed.
    Slot D: Checking FISO #1...
    Slot D: Checking FISO #0...
> Slot D: Zoom Acquisition Chip Select Test Failed!

[mbalmer]

I have text files comprising a sequence of 50 successive runs of x cmpTest and 20 runs of x zoomChipSelTest. If you'd like me to post them, I will toss them onto my Google Drive. I'll hit the post character limit easily otherwise. 

[MarkL]

No need to post the other 50 runs.  Let's start with this.

Does this board work normally?  Does changing the logic thresholds work as expected?  I'm asking becasue we could be looking at a failure of the self-test mode and not an operational failure.

Whichever the case, I think the first suspects are whatever's in common to ALL channels.  There's just too many errors.

During cmpTest, the DAC output values are moved around to create different logic levels and clock patterns for Chip 8 and Chip 9, and those levels are captured in acquisition memory.  pv checks each input bit against what should have been received and that's what those errors are all about.  ("B" == bad bit).

One interesting clue is that all bits on all pods are always bad when there should be no activity or transitions on the ASIC input lines (Cal Clk No Act.).

The posts above from June 2020 show what should be happening on one of the comparators (U69).  A quick probe on U69 with cmpTest looping could be enlightening.

Since the DAC (U34) is in common to all comparators, there could be a problem with the DAC reference voltages (U44 and U45), or maybe the DAC CPU interface (data lines/chip select) have a problem.

There's also a test enable pin that's common to all the comparators (pin 18).  I would make sure that it's showing sane TTL voltage levels and is going low when cmpTest is running.  If it's flapping around because it's trace is severed, that might explain this.

I think we should set aside the zoomChipSelTest for the moment.  As pointed out before, I think the comparator failures reported first are probably the cause.

[mbalmer]

Does this board work normally?  Does changing the logic thresholds work as expected?  I'm asking becasue we could be looking at a failure of the self-test mode and not an operational failure.

Whichever the case, I think the first suspects are whatever's in common to ALL channels.  There's just too many errors.

During cmpTest, the DAC output values are moved around to create different logic levels and clock patterns for Chip 8 and Chip 9, and those levels are captured in acquisition memory.  pv checks each input bit against what should have been received and that's what those errors are all about.  ("B" == bad bit).

One interesting clue is that all bits on all pods are always bad when there should be no activity or transitions on the ASIC input lines (Cal Clk No Act.).

The posts above from June 2020 show what should be happening on one of the comparators (U69).  A quick probe on U69 with cmpTest looping could be enlightening.

Since the DAC (U34) is in common to all comparators, there could be a problem with the DAC reference voltages (U44 and U45), or maybe the DAC CPU interface (data lines/chip select) have a problem.

There's also a test enable pin that's common to all the comparators (pin 18).  I would make sure that it's showing sane TTL voltage levels and is going low when cmpTest is running.  If it's flapping around because it's trace is severed, that might explain this.

I think we should set aside the zoomChipSelTest for the moment.  As pointed out before, I think the comparator failures reported first are probably the cause.

I will have the opportunity to test some of this out tomorrow. Being a band director is sometimes rather involved, time-wise.

[mbalmer]

Just to brink MarkL up to speed on my TERRIBLE luck with these things:

I have five boards. Each module fails a different set of tests, although there are some that are disturbingly common.

Board A fails the comparators and ZoomChipSel tests.

Board B fails comparators and calibration.

Board C fails VRAM (both serial and parallel) and comparators. 

Board D fails VRAM, all Zoom chip tests, and one of the inter-board tests.

Board E fails every test except for chip registers, comparators, backplane clocks, the ICR test, and the arming test.

So it seems that while the comparators test is a point of failure, it’s definitely not the only one.

I received a sixth and seventh 16717A board today. I plugged what we'll call Board F into the mainframe as the only board installed, and immediately received an error upon bootup that said that the module failed its' high-speed system clock test, and it claimed that the frame may need service.

However, the board passes all tests with x modtests from within pv.

I then re-ran all tests by setting d=9 (but not r=9).

As it ran the test, once it reached the vramCellTest, it flooded the screen with information, mostly clock numbers and hexadecimal codes which I'm guessing relate to specific VRAM cells, the data sent and the data read back. It sat there counting down clock codes from over 2,000,000 for a good solid 30 minutes before it moved on to the next test. After letting it run, all the tests passed on debug level 9.

I pulled Board F out, and replaced it with Board G. The frame powered up and got to the Workspace system without throwing any errors, so I logged into it via telnet and then ran a suite of module tests through pv.

Sure enough, it fails three tests: cmpTest, calTest, and zoomAcqTest.

This one fails cmpTest in an unusual way, however:

 

Code: [Select]

  Check POD1 Thresholds:
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 9: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Activity
  Check POD2 Thresholds:
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 9: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Activity
  Check POD3 Thresholds:
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Activity
    Slot E, Chip 8: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD4 Thresholds:
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Activity
    Slot E, Chip 8: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
> Slot E: Comparator Test Failed!

I'm about ready to give up on this boat anchor. This is potentially only one out of seven boards I've received that has successfully passed all of its tests, and even then, the mainframe doesn't fully like Board F since it throws a boot-time error about the system high-speed clock.

[mbalmer]

Spoken in the manner of Jean-Luc Picard:

Repairs log, HP 16702B, calendar date September 19, 2022.

Board F is, thus far, the only logic analysis board to fully work out-of-the-box and not have issues. I have taken a hot-air gun to the board to gently remove the plastic runners, and they came off remarkably easily. The adhesive strips were supple and not crumbly, and clearly were in good shape. My estimation is that if there is any corrosion on this board, it's minimal at best.

Board F was plugged into Slot C and retested, and the system did not complain about a high-speed clock malfunction.

A 16755D module arrived yesterday and I have not had the opportunity to plug it in and look at it. It may be functional enough that I can use it as a second set of channels.

I've acquired a small set of the Mechano SMD lead grabbers that I can use to get in tightly to the pins on one of the boards. I will be using them, along with a board extension I designed to fully extend the board under test out of the mainframe so that signals can be safely probed on both sides of the unit without risking shorting it out against the case. This extension should theoretically also work for any 165xx, 167xx, or 169xx-series modules as well, so if anyone decides they'd like one, I can offer up the gerbers once I dial in the exact measurements (I was a little bit too wide from side-to-side, but everything else looks good.

Probing and investigation continues.

[MarkL]

On that card with the sparse failures, I would check that the comparators are receiving the test clock.  The failed bits are all in the positions where there should be a clock detected (when the DAC is set to 0V for that pod).  All the other bits I think are showing not failed because they are not having any transitions (no clock), which is correct.

Start cmpTest running in a loop and take a look at the ECL clock before it goes into an RC network near the comparators.  See comparator_ecl_clock.png below for the probe point.

You can also verify the enable signal for the clock signal near U74, test_clk_enable.png.

Below is also a scope capture where you can see the clock on Ch1 (yellow) and the enable on Ch3 (blue).  Most of the time the test runs 4ms or so, but it can stretch longer (probably dependent on other housekeeping the system is busy doing since this is not a real-time test).

Also below is a script I've been working on to loop the pv tests.  You can give it a try and let me know if it works ok for you.  I call it "pvl", short for "pv loop".  It's a little bit hack-y in places, but the idea was not to pile on a bunch of prerequisites to run it.  It only needs the stock shell.

The enable signal runs down the edge of the board to the FPGA near the edge connector and also passes directly under one of the runners.  If the enable signal is not showing up, I would check the continuity of that trace from start via to end via. And my general recommendation is to do such a check on ALL the traces under or near the runners.  The corrosion can get under the solder mask and break the trace and it's not always visible.

Code: [Select]

#!/bin/sh
# /bin/sh is the POSIX shell on HP-UX.  man sh-posix

# This script sends commands to pv to run a module test on one or
# more slots.

# shortened prog name
#
myname="${0##*/}"


# -------------------- Parse args --------------------
#
opt_w=30

usage()
{
  echo "
usage: $0 [OPTIONS] [<SLOTS> [<TEST>]]

  SLOTS  is a group of letters indicating which slots to test.  Multiple
         slots running the same test can be used to compare a broken and
         working modules.  If the word \"help\" is given, the general help
         will be printed which may have extra information at the bottom
         because of the non-default mode levels being set.

  TEST   is the test to be run.  If TEST is not given, the list
         of possible tests will be printed for the given slot(s).

  If no arguments are given, pv will be run in interactive mode
  with the result, debug, and mode levels set.

Options:
  -wN  run the idle loop N times between commands (60 is approx 1 sec).
       Idle loop defaults to $opt_w if this option is not specified.

Examples:
  Run cmpTest on slots e and b continuously with a pause after each
  set of test:
 
    $0 eb cmpTest
" 1>&2

  exit 1

  #  -sN  sleep for N seconds between commands (overrides -w)
}

while getopts "w:?" opt
do
  case "$opt"
  in
    w) [ "$OPTARG" ] && opt_w="$OPTARG" ;;
    s) opt_s="$OPTARG" ;;
    '?') usage ;;
  esac
done
shift $(($OPTIND-1))

slots=$1
test=$2


# -------------------- Funcs --------------------

waste_time() {
  # Do a slow-ish operation $1 number of times.  This is in place of a
  # fractional sleep command, which HP-UX doesn't have without getting
  # into compiling a program.
  #
  # 30 iterations is about 0.5 seconds on a 16702B.
  #
  local i=$1
  while [ $i -gt 0 ]; do
    ls / > /dev/null
    let i=i-1
  done
}

pv_cmd() {
  # Send command $1 to pv and wait for the marker on the output.
  # Running a system command from pv causes it to flush stdout.
  #
  print -p "$1"
  print -p "! echo %%MARKER%%"
  while read -p; do
    [[ "$REPLY" = *%%MARKER%%* ]] && break
    echo "$REPLY"
  done
}

# -------------------- Main --------------------

# Turn on all pv debug options via env variables (undocumented feature).
# Upping the mode level causes additional hints to appear at the bottom
# of the help output.  A result level greater than 9 shows additional
# information, despite that help says 9 is the max needed.
#
export PVRESULTLEVEL=10
export PVDEBUGLEVEL=9
export PVMODELEVEL=1

# There is odd behavior from pv that prevents calling it from being
# generalized.  It may be closing and re-opening some of the file
# descriptors, but I'm not sure.


(( $# == 0 )) && { pv; exit; }
[ "$slots" = "help" ] && { pv -c help < /dev/null; exit; }

slot_list="$(echo $slots | sed 's/\(.\)/\1 /g')"

if [ -z "$test" ]; then
  cmd_list="$(echo $slots | sed 's/\(.\)/s \1,t label,/g' | tr , '\012')"
  out=$(echo "$cmd_list" | pv)
  echo "$out"
  exit 0
fi

pv 2>&1 |&
PVPROC=$!

while true; do
  for slot in $slot_list; do
    pv_cmd "s $slot\nx $test"
  done
  [ "$opt_w" ] && waste_time $opt_w
done


# The above loop will never exit, but if this script is modified
# for anything else, one of the below can be used to quit pv.
#
#   print -p "q"
#   wait
#
# -or-
#
#   kill $PVPROC
#   wait

exit 0


[mbalmer]

On that card with the sparse failures, I would check that the comparators are receiving the test clock.  The failed bits are all in the positions where there should be a clock detected (when the DAC is set to 0V for that pod).  All the other bits I think are showing not failed because they are not having any transitions (no clock), which is correct.

Start cmpTest running in a loop and take a look at the ECL clock before it goes into an RC network near the comparators.  See comparator_ecl_clock.png below for the probe point.

You can also verify the enable signal for the clock signal near U74, test_clk_enable.png.

Below is also a scope capture where you can see the clock on Ch1 (yellow) and the enable on Ch3 (blue).  Most of the time the test runs 4ms or so, but it can stretch longer (probably dependent on other housekeeping the system is busy doing since this is not a real-time test).

Also below is a script I've been working on to loop the pv tests.  You can give it a try and let me know if it works ok for you.  I call it "pvl", short for "pv loop".  It's a little bit hack-y in places, but the idea was not to pile on a bunch of prerequisites to run it.  It only needs the stock shell.

The enable signal runs down the edge of the board to the FPGA near the edge connector and also passes directly under one of the runners.  If the enable signal is not showing up, I would check the continuity of that trace from start via to end via. And my general recommendation is to do such a check on ALL the traces under or near the runners.  The corrosion can get under the solder mask and break the trace and it's not always visible.

I should have the chance to test this out here in the next week or so. Once I get past our marching contest season, my weekends clear out rather dramatically and I start to feel less like I'm being hounded to accomplish everything all at once.