The offset of less than 1 µV would be acceptable for most cases, especially higher impedance use. Beside the photovoltaic, relatively fast part, there may be another slow thermal part, as the LED side will heat up the switch.
However there could still be charge injection from the internal FET. It is no per se clear that charge injection is less than with a normal MOSFET or JFET. It is only the normal capacitance around the switch that is reduced, not that actual redistribution of the gate charge.