For low Ohm values, i.e. low test current and low test voltages,you need an additinal function, called Offset Compensation. This removes thermo voltages, as you encounter.
Either you need a more sophisticated instrument, or you may also put a switch into the current line of your 4W setup, and null the reading, when switching the current off towards the DUT.
There are several videos, how to implement such a switch box.
Frank