If you want to calibrate gain and offset too, then let AREF be the ADC reading when the pin is connected to VREF, and AGND when the ADC pin is connected to ground. Then, the ADC measurement A is described by
A = (AREF - AGND) R0 / (R0+R) + AGND
which we can easily solve for R:
R = R0 (AREF - A) / (A - AGND)
or for R0:
R0 = R (A - AGND) / (AREF - A)
You'll need three measurements: AREF by measuring VREF, AGND by measuring GND, and one to measure the fixed resistor, by measuring a known precise resistor; either R or R0, depending on how you label the unknown and fixed resistors. Here, R is between ADC and VREF, and R0 is between ADC and GND.
Ok, I think this gets me to where I need to be. Just to summarize for my own benefit:
I should be able to measure
AREF by leaving the unknown resistor out of the circuit - I currently have the soldered-in-place resistor,
R as a 'pullup' to V
REF, and without
R0 in place, the measured value should be effectively V
REF. I should also be able to measure
AGND by inserting a short in place of the unknown resistor,
R0. And finally, by inserting a known high precision reference resistor as
R0, (probably with a similar nominal value as the known resistor), I should get the value for
A. I can then use the now known values of
A,
AREF,
AGND and
R0, to determine the actual value of
R. Once I have all of those values, I can then replace
R0 at will and use the previously determined values to calculate the value of
R0 given a newly measured value of
A.
I may find after a certain amount of testing that
AREF and
AGND are close enough to full range and zero that I should just ignore those values, since the ADC should be full scale at V
REF, and zero at GND. In which case calibration does get quite a bit simpler.
Thanks to everyone, this makes a lot more sense now. I had figured out some of the underlying math, but I think I was stuck on how to measure
R in place, and how to deal with gain and offset errors if enough exists to matter.
Oh, and to clarify my earlier statement about "nonlinear", I was simply referring to the fact that the solution to this problem isn't as simple as using slope/offset, i.e. the mathematical function of a line. I should have thought to better clarify what I meant.