Many modern DSOs have very low noise ADCs which produce almost a flat line especially with the 20MHz bandwidth on.
Marketing loves the term "low noise"; it is nice and ambiguous when they do not give an actual number in the specifications which can be confirmed by the user. None of the modern "low noise" DSOs I have tested had less noise than old analog or digital oscilloscopes although I am sure some exist. This might be attributed to integrated CMOS amplifiers which can have truly horrible input noise but interleaving and pipelining the ADCs does not help.
If they did have low noise, then averaging a clean triangle wave as I described should return the stair-step pattern of the ADCs differential non-linearity and this is one way to measure it.
This might be easy enough to check; is the record length halved when high resolution mode is used? What about in average mode?
That doesn't have to be true. Hi-res can also be done as a post-processing step when the signal is displayed. For example: on the R&S RTM3004 the memory depth doesn't half when hi-res is selected.
If high resolution processing is done during decimation, then the processing record takes twice as much memory if 16 bit values are returned. The record length may or may not be halved depending on if it is based on 8 bit or 16 bit samples.
This is a marketing issue. Which is the better of three DSOs with the same *amount* (cost) of sample memory when one always has the full (8 bit) record length, one has half the record length when high resolution or averaging mode is used, and one always has half the record length because it always processes 16 bit samples?
If high resolution processing is done after decimation in the same way, then the resulting record will be a fraction of its original length depending on the decimation ratio. Maybe your R&S RTM3004 is applying a complete FIR filter and not decimating? That is usually not feasible during decimation which is why boxcar averaging is used instead. Boxcar averaging is of course a (poor) FIR filter but a very simple to implement one which only requires adds and optional shifts.