If it has to do with peak detection which I think is the most likely option, then it will be easy to determine; check to see if the number of horizontal display points is halved when peak detection is used.
It always returns 1200 pts, also in peak det. mode.
But it cannot return a 1200 point peak detected record in 1200 8-bit words; it would take 2400 words. I assume the FPGA always returns a 1200 word record but in peak detect mode, the record is composed to 600 pairs and in normal mode, it is 1200 separate points. At least that is how other DSOs handle it. By returning a record which is twice as long as necessary in normal mode, peak detection mode does not visibly halve the horizontal resolution.
There is no reason the FPGA could not return different record lengths for different modes but it is an added complication which is not needed for a questionable increase in performance.
Offtopic, what I personally find strange is, why it uses a vertical resolution of only 25 lsb's per division.
For comparison, the DS6000 uses 32 lsb's per vertical division and thus uses the full dynamic range of the ADC for the display.
Lots of DSOs work this way. An integer mapping of LSBs to display pixels prevents unneeded processing and aliasing. Further having the digitizer range exceed the displayable screen size prevents clipping during some types of processing although I am unclear if the Rigol can take advantage of this.
I have seen and mostly used DSOs which use 25, 40, 50, 100, and 102.4 points per vertical division. Resolution in excess of that necessary to display 8-bit data operates on 16 bit data when averaging, high resolution, or DPO modes are used