Exchanging the leads should not produce much of a difference in thermal emf, as the leads should be almost identical, as should be the binding posts / banana jacks. The leads and jacks however form one or more thermal junctions, e.g. in the case of my 34401A: Cu jacks -> CuBe banana spring -> CuTe or brass banana body -> Cu wire. With higher-end meters, you would probably get something like Cu internal wiring -> CuTe binding post -> Cu spade -> Cu external wire -- better but still not perfect.
More to the point, if you exchanged the two leads, you'd have to wait around 5 minutes for the thermal transients to stabilize, as shown in the graph I posted earlier. In those 5 minutes, the meter/vref may drift slightly, corrupting your measurement. Or you may affect the natural airflow around the instrument.
As a related example, my meter offset takes 3 to 4 hours to stabilize from a cold turn-on. Then, I can see a slight jump when opening the window. This is with a copper wire shorting the inputs, so these are probably internal thermal offsets which do not fully cancel out and which are outside the autozero loop - front/rear switch, protection components maybe. They can be compensated through calibration of course, but there is a limit to this - maybe this is why the 34401A is only specified (24h) at 30 ppm of the 100mV range, as opposed to 6 ppm of the 1V range. I'd bet it's not the amplifier, it's the parasitic emfs.