As far as the jitter at delay goes, it would seem like the the architecture of the acquisition and memory systems might come into question.
On my TDS784D, I can plug in any delay value and see no jitter whatsoever. It became scary with really long delays so I turned on FM to be sure, so I could see no "jitter" at zero delay and increasing "jitter" as the delay was increased.
On the other hand, the problem could simply be incorrect indexing when mapping acquisition memory to display memory, and adding the intensity grading process on top of that.
If it is simple, how fast is Rigol at turning around solutions? If they are as fast as they are with revising security measures, we're all set