I don't think an LM324 is a good choice for this application. I made one once with a *324, and ended up having to use a negative supply and bias an input to correct for offset voltage.
My uneducated guess is that the input offset voltage of the 324, in both stages (buffer then driver) is throwing the spanner in the works for you.
The offset voltage is potentially anything up to 9mV, and the output can only get down to 5mV above the gnd/negative (it's waaay worse at the top of the range too several volts under the positive supply).
Aside: I recently (as in, today) finished one based on an LM358 (hmmm, now I think about it, it should be working the same as the *324 since the specs are about the same, maybe I'm misremembering how bad the 324 one was), which works a bit better, but in hindsight I should have given it a negative supply and the ability to compensate. Also, I have resolved that next time I build one, I'm putting some smarts in it with a microcontroller so I can "cal" the damn thing in software, freaking trimmers.