I would expect that to be fairly normal. The bias current of the input amplifier (which is really a leakage current) has nowhere to go, and slowly charges up the input capacitance. It is a real voltage, not an amplifier offset, so auto-zeroing wouldn't help. If you connected the probes to a perfect voltmeter, you would see the voltage on the probes slowly charging up.
I have a keithley 2100 that has a configuration option for the millivolt range to be > 100 GOhm or 10 MOhm. The default is 10 MOhm, but when switched to high impedance mode it charges up exactly like you describe.