Unfortunately you can get there from overlapping bounces on both A and B, which *CAN* look like the quadrature signal you are expecting from movement,
I might have read the datasheet wrong, but it shows several full complete transitions and there are gaps between the contacts aligning which limits the plausible bounces to single pins.
I can suffer a bounce, say for example I get:
0111 - which I take as clockwise.
If a pin bounces it could go to:
1100 - which is invalid
or
1101 - which is invalid
A double bounce I suppose could go:
0111 - Valid CW
1101
0111 - Valid - ERROR double bounce.
Maybe my vTaskDelay( 1 milisecond ) is helping as well here. I was going to replace it with a yeild. I don't really like hot looping because in other hardware it's very expensive and frowned upon to do so unless absolutely necessary.
EDIT: On rotary encoders as a user. Yes they seem to be one of the first things to go on items like HiFis and similar devices. My Sony Amp circa 1995 lost it's volume control a long while ago. Have to use the remote. The EQ preset encoder jumps all over the place. I expect the volume control has "anti-bounce" coding or hardware and that's why it's mode of failure was just that it started ignoring more and more inputs until it just stopped altogether, it no long sees any valid transitions!
But if you use a pot and an ADC it too is subject to wear and tear causing bouncing due to "scratchy tracks" syndrome.
Hall effect encoders seem the only way. Might see how expensive they are. I did try and find a hall effect analogue joystick for a different project and found they tend to be expesnive.