I did some work on this subject in the hope of getting JYETech to fix the rotary encoder servicing on my DSO150 kit scope, and having done the work I decided to post the info on Github in case it might be of use to anyone:
https://github.com/gbhug5a/Rotary-Encoder-Servicing-Routines/There's a PDF file there that describes in detail how the routines work. JYE is currently polling using the lookup table state machine method, and it isn't working well. There is no hardware debouncing. I realize that most people use that method, but it appears the lookup table is not able to distinguish between a legitimate state succession and noise which produces the same pattern. So on the DSO150, I may get no tick, or a tick in the wrong direction, or multiple ticks, or even the right tick, all from just one detent change. Basically, it's just no damn good.
My routines depend on the assumption that the two switches will not be bouncing at the same time. When one switch changes state and bounces, the other switch is stable. Of course that may not be true for rubbish encoders, or for any encoder that's spun fast enough, but testing my routines on an inexpensive Bourns PEC encoder showed that they work very well no matter how fast I tried to turn the knob. And I suspect that in practice the assumption is valid most of the time.
The basic idea is to avoid having any of the bouncing generate an interrupt, or cause a poll to think a tick has occurred. So in the case of the pin-interrupt method, you only enable one interrupt at a time, and when that interrupt occurs, you immediately switch the interrupt over to the other pin, which is stable. So all the bouncing on the first pin has no effect because the interrupt is no longer enabled on that pin. As a result, going through a full cycle, I have to service exactly four interrupts, one at each actual state change, and bouncing has no effect. The polling routine operates in a similar manner by setting a target state, and when that state is reached, changing the target to the opposite state - which is impossible to match so long as the stable pin remains stable at the old target state.
The routines produce one tick per detent, so no dividing down is needed. Also, a pin-triggered method typically has a problem because there is a delay between the interrupt trigger and the reading of the port. Bouncing during that delay can produce a bad read, but my routine fixes that.
I tested these routines on a TI MSP430G2231 at 1 MHz, and for polling the frequency was 488 polls per second. I've included the assembler source code for those tests, and executable .HEX files.
If anyone is interested, it's all in the PDF, including all the logical steps in understandable form, not obscure code. Then the assembler source shows the operations in detail.
The pin-triggered servicing routine is 146 bytes long, and the polling routine is 96 bytes. In both cases there is some setup code in the main routine. Maybe at some point someone will implement these routines for Arduino and test them.