At my last job,we tried to do this with an analog storage Oscilloscope,with the same problem of high level noise on the line in the "silence" period.
We really only wanted to see if the signal was there & not mangled in any way.
RS485 as we used it, was as a 2 wire system with neither wire at earth potential.
We ended up cheating,as the line went into a Max485,so we just looked at the signal after conversion to PIC compatible levels.
Have you tried using both channels in the quasi-differential connection?
Most people on this forum sneer at it,due to poor common mode rejection,but it does work OK in many cases.We didn't try it because we had the opportunity to "cheat".
The fault we were chasing turned out to be internal,& nothing to do with the data link!
VK6ZGO