I FOUND THE PROBLEM! FINALLY!
and it is not the code, the problem was that I made a small encoder module because last time I just soldered pin headers to the encoder the encoder pins just broke, and somehow there was not a good soldering connection (which is weird because by eye it looks perfect) so I have just re-soldered it and and it works great, ISR or just polling. "never assume anything in electronics" - I assumed the problem was the code and not the encoder, silly me.
Thanks @rstofer, I connected a resistor and a led to see if the led would light up the way it should (00, 01, 11, 11) and I noticed only 1 led was on, the other one just didn't light up, measured continuity and found the problem. 2minutes fix, 6 hours+ troubleshooting a code that works.