No, I've tried setting the timebase to extreme values and it doesn't change at all. It really does look like the Math function is working with too little vertical resolution on the source waveforms. At the sensitivity I'm using here they look pretty pixilated as well, though the input signal is fine.
Waveforms vertical resolution come from 8 bit ADC. ADC full scale is around 10 vertical division on display.
One ADC step is 2 display pixel. Now your scale is then multiplied by 5 (your channels are 50V div and Math is 10V div (50/10=5). So, your Math trace steps on the display are now 10 pixel height)
This same happen if you capture signal, stop scope and zoom in vertically) It do not use interpolation between math result data points or other tricks for produce more nice looking "art" image.
Now also original traces CH3 and CH4 looks smooth. These waves update rate is much higher than Math trace and there is several waveforms overlayed in every TFT frame. If you change Acquistion mode from fast (default) manually to Slow then these all waveforms are updated around same speed, one wfm one TFT frame. Very specially it can see if you change display mode to Dots and then change between fast and slow acquisition mode.
What ever you use: Trace average or ERES result is always 8bit but method is different and for different needs. (and example some manufacturer advertise 12 bit after box car averaging. Result is 8 bit and nothing more.)
Also if use display mode Vector it is wise some times to look with interpolation Sin(x)/x on and off. (in Vector display mode. Dots mode do not use interpolation at all.
But then. Also there is possible something wrong in Math if Math sources a A and B are from different ADC groups. (CH1 and 2 is group one and CH3 and 4 is group 2 ... because there is 2 ADC's)
Math function FFT have display mode "Exclusive" and this display mode turn source display off.
I hope Siglent add this display function available also with other Math functions.
Now example with CH1 - CH2 only thing what user can do is dim A and B traces to minimum but even then they are very visible.
But, because Math is not only function where example I miss feature that I can shut off channel trace display but only display. Example some times if I need use one channel just as Ext Trig and I do not want this signal is always visible. If Siglent develop this (and with good ergonomy-usability) but it need experienced and intelligent UI designer, not only programmer, then it also can use with some Math functions.