Dave, I know I'm supposed to use BNC connectors, it's just that I don't have them.
This is actually my first oscilloscope so I just have the out-of-box parts and whatever I can check out from school.
I'll do a full how-to when this is peer reviewed, I think, but here's what I found.
At first I thought it would be trivial, it was pretty clear that this was a Butterworth filter from the frequency vs. voltage plots that have been posted, so there should be some row of inductors and capacitors. I found one, but it was a filter for the AD8370 power. I figured that out by tracing it to the pin and then checking the datasheet. Then as I
read the datasheet (should have thought of this FIRST) I ran into something interesting. In the app notes they specifically mention using this as a variable gain amplifier for low-resolution AD converters.. like an 8 bit AD converter. This is on page 19 of 28, and right at the top of the page is a Butterworth filter with a Cauer topology. This is in case the AD converter can't handle high frequencies.. or in case you want to use the same design for two differently priced oscilloscopes.
NO WONDER the part is rubbed out!
VHI is pin 8 and VLO is pin 9, this is the amplified output signal. There are two gold pads on the PCB that are labelled as test points on the DS1102E. I noticed attenuation here. Then, looking through the comments on Dave's teardown blog I saw that someone guessed the part number for the smaller, 8-pin part. It wasn't the right part number, but it fit the bill exactly (this is not an Analog Devices part, though, because it matches
none of their ADC drivers,
not a TI part either). The only thing that was off was that the VHI and VLO pins were reversed, so I just assume this is the AD converter buffer. The datasheet reference circuit for the AD9288-40 ADC uses an AD8138. This doesn't come in a TSSOP-8 package. In fact, I can't find any op-amp with that configuration in a TSSOP-8 from Analog Devices.
Looking at the board, I noticed that the signal split in two directions at the test point. One way went to the buffer and the other went through a capacitor..
Re-organize a bit, and it's clear that there is a high-pass filter straight between signal HI and signal LO. This is why some of my measurements were out of phase, even though both channels were measuring the same signal.
(values based on in-circuit measurements by a cheap LC meter, not in stone)
The via hanging off of the inductor has a direct path to the shift register, I'm not sure why. It's not grounded, and I don't know why it would be tied to a logic high anywhere, not sure what's going on. There's a via hanging off of the inductor at the top too, I couldn't find where it goes.
So my "hack" was just to remove the capacitor on the VoutHI pin. I just now took all of the measurements as I was making the drawings, and I took out the other cap on the VoutLO pin. There doesn't seem to be any additional change, I believe that the AD8370 output is isolated from ground, which means when you sever just one capacitor the whole filter is rendered ineffective. So all that has to be done is cut the path to the filter somehow. I removed a capacitor with my soldering iron, you could also just slice the trace (but that's a little dirty).
In closing, your mileage may vary, at your own risk, warranty void and null, etc etc.
Here are some more good pictures, showing how the FFT smooths out with a mod and how a square wave shapes up. Flip between them quickly and the difference is very noticeable. Again, channel 1 (yellow) is unmodified, channel 2 (blue) has the filter removed.