Your debouncing circuit for the rotary contacts looks suspicious. A mechanical encoder uses a sectored contact disk and wiping contacts. When the contact is over an insulating sector it physically cannot bounce closed and when it is over a conducting sector, it skips, bouncing open. It is therefore desirable to have a significantly asymmetric debouncing response, quick to go low and slow to go high. I therefore believe it would be advantageous to swap the 1K and 10K resistors for CHA and CHB.
I also strongly suspect that the button debounce circuit will need different time constants to the CHA and CHB encoder debounce circuits. The button debounce needs near symmetric press and release time constants and to transition within a few tens of ms of actuation, certainly in less than 100 ms, whereas the encoder debounce is constrained by the maximum speed of rotation, and number of pulses per turn, and must never exceed half the interval between edges.
See
http://www.ganssle.com/debouncing-pt2.htm (part 1 is also worth a read)
Interrupts triggered by a mechanical event are a Bad Thing, unless you suppress the interrupt source for a period after each interrupt, as you risk an
Interrupt storm as the switches wear and their contact bounce increases.
Timer based interrupt driven polling is far more likely to give predictable results.