Just one little addendum to the statistics discussion -- I decided to dig a little deeper into tolerance intervals (which, for example, given a bunch of multimeter observations, lets you answer a question like "what interval will contain 99.7% of other units in the population, i.e., future multimeters).
I followed the instructions
here. It turns out you have to provide
two things: a percentage of population units that you want your interval to capture (I chose 3 sigma, i.e., ~99.7%), and the confidence level with which you want your interval to be correct (in the style of confidence intervals, I chose 99%). So, in short, I'm answering the question: "Based on the sample data, what interval will 99.7% of future meters fall within? I want my interval to be sufficiently large in in 99% of parallel universes."
The answer is (AFAICT) 4.99975 ± 0.00054 (
unreadable spreadsheet), which is a significantly larger range than the naive 3 sigma method, even though this interval provides "only" 99% confidence. To get 99.7% confidence would widen the interval even further (and require access to chi-squared tables that I didn't immediately have to hand!)
To preempt a couple of questions:
- So why can't I use the 3 sigma rule? I touched on this before, but that rule is only legit when you have population means and standard deviations to hand. When you have only sample means and standard deviations, those are only estimates of the population statistics, and that's where the chi-squared distribution comes in.
- How do you make sense of, or choose, the two parameters? Unfortunately that's a question of interpretation that the science of statistics is not particularly concerned with answering. All that statistics has to say is that it is true that in 99% of possible samplings/parallel universes, ~99.7% of meters in the population fall within an interval computed in the same fashion as above. In the other 1% of parallel universes, you'll get an unlucky sampling that is way out of whack and you'll have no idea that that's even happened. Deal with it. How you choose to make decisions based on that information is entirely up to you.