You sampled the signal at 500mV/div, then zoomed in all the way to 20mV/div, what did you expect?
It's a 8bit ADC so this is completely normal, you have only 0-255, 256 different values.
500mv*8div = 4000mV -> 4000/256 = 15.6mV vertical resolution.
So of course it looks like that.
When you later sampled the signal at 20mV/div, the story changes, now it makes:
20mv*8div = 160mV -> 160/256 = 0.62mV vertical resolution.
But as you can see when zooming in, at 500mV the resolution is actually 20mV, so it's storing a bit more than 8 divisions:
20mv*256 = 5120mV -> 5120/500 = 10.24 vertical divisions.