The old panel (the 800x600 one) supports both 8-bit and 6-bit color modes, selectable with a pin, whereas the new one only supports 6-bit color. Since changing from one mode to another changes the order of the bits in the LVDS lines (and also which lanes carry which signals) maybe the display is getting the data in the wrong order making it do funky stuff.
Also, unrelated to the problem, did you just plug the new display in place of the old one and it worked or did you have to do something to adapt it? Because I'm looking to do something similar on a laptop I own and I'm not sure if I can just swap in a higher resolution LCD and expect it to work