Jitter is not so likely, but (semi) static differences in baudrates are unfortunately quite common.
For 8-bit data the maximums absolute error between the baudrates of receiver and transmitter is around 4.5% Start bit + 8 data + stop bit = 10 bits, and at the end you must have accumulated less then half a bit time of error.
And that error budget gets divided and eaten up in various ways.
* Laziness or incompetence for using the right crystal (Especially arduino is notorious for this. 16MHz is quite bad for most baudrates).
* Drift when using an internal RC oscillator clock. (Mostly drifts with temperature)
* Fractional baud rate generators (These introduce jitter in bit timing but also average out the error, and the total result is mostly quite acceptable)
Also, some manufacturers of equipment do about everything to reduce BOM cost. 20+ years ago I bought a PCI card with 2 serial ports, and it's baudrates were about 2.4% off. That's acceptable if the receiving end is perfect, but if the receiving end deviates by the same amount in the other direction then it does not work at all anymore.
Also, it does not matter whether you have a high or a low baudrate. If it's over 5% off between transmitter and receiver, then it simply does not work anymore. At lower baudrates you do have to set a bigger divisor in your uC, and this allows for a bit higher resolution in setting the baudrate, but it's not always enough. Atmel used to print big tables with usable baudrates in their datasheets (for example ATMEGA328) Fractional baudrate generators are much better in this regard. Resolution is high enough that any crystal can be used up to quite high baudrates, but there is a limit.
More modern communication standards use bit stuffing to guarantee that there is always a signal transition after a certain amount of bits, and this is combined with hardware to re-synchronize on the bits.