Try an (exponential) moving average filter. Not just an normal averaging filter, because that also
decimates. Which you might not want to.
And while you're at it, left align the adc (there a option bit for that), since you need at least 10 bits for the adc. That automatically rounds to 16 bit, so why not use those 6 bits?
When left aligning the range will go up to 0 to 65535, but the resolution of your adc won't!
A filter now also does
oversampling, how negligible that may be here, but it means that is can also display 100.5 adc point, as in 6432 left aligned.
Which could be the result of your filter.
Don't try floating points, keep everything in integer math and divisions in multiples of 2. If you can.
I'm 100% sure someone has already done this before on an arduino atmega. But never use
float on an 8 bit avr.
But in the end 0 to 255 might work just fine for your joystick anyways. So lowering the hardware resolution could suffice as quick fix.
Here you go. https://github.com/AsheeshR/Microsmooth see ema_filter