USA CAl Club: Round 3 suggestions

[TiN]

In return, those that are able can verify the reference with their equipment and provide feedback and additional confidence in the reference.

For me, that was the whole point why I decided to donate FX (with help of Jason's resistors, and LTZ) to the club, so I can see how my design survives multiple shippings/abuse from normal users. Eventually next year I'll have it back and compare its performance to original data.

[bitseeker]

I see some possibilities here:

1. Since the name, USA Cal Club, is quite broad being limited only by geography, there could be different groups coordinated by potentially different people to satisfy different cal interests: core voltage/resistance, extended voltage/resistance, time, etc.

2. Rename this club to something more specific to better reflect its purpose and scope (e.g., USA Volt Cal Club. Hey, that's V_CC ). Of course, anyone is free to make other purpose-driven clubs.

3. Update this club's user guide to explain what this club is for and set appropriate expectations. This would be beneficial in general.

[GigaJoe]

true ...  my 10Mhz OCXO+battery idea quietly die, due to null response and GPSDO $50+
and 4 X ad587 in small +45C enclosure, produced around 2ppm \ year (worst case).  but ... we are north and no one close to verify.
nulling - pieces of cake, ideally with 10G voltmeter , but if a source has a low impedance,  even $20 multi-meter works ... if it had micro volt resolution ...

[The Soulman]

nulling - pieces of cake, ideally with 10G voltmeter , but if a source has a low impedance,  even $20 multi-meter works ... if it had micro volt resolution ...

High input impedance isn't absolutely necessary with these low differential voltages current is really limited.
Yeah some $20 meter work nicely, but they're typically limited to 100 microvolt resolution (set to millivolt range).
But some to 10 microvolt, that already gives 1ppm resolution when comparing two 10 Volt references, good enough for normal people.
Beyond that is noise and thermal emf mayhem, fun to look at when designing and building v-refs, but for transferring an absolute value not that necessary.

btw, The cheapest data logging bench dmm with microvolt resolution I could find was the Vici vc8145

[rhb]

Shipping is now $30/person vs the original $7/person.

That is not true. The current kit fits perfectly in a USPS medium flat rate box. The cost is $14.35 for 2-Day priority mail anywhere in the US.

My $30 figure is based on the recipient paying for shipping both directions.  It seems quite unfair to me to ask the person handing the shipping to also pay for it.  Just checking and shipping seems quite  a lot to ask, even of a volunteer.

So I urge everyone who can afford it to reimburse @vindoline when they return the kit.  Even if he doesn't ask.

At the moment we have a "one size fits all" as a consequence of TiN's generosity.  I think there is great merit in, as a first step, arranging for a two track system, PX/FX and PX only, that provides regular measurement of the PX standards by those with the best capability.  For that we need to build a couple more PX references.  In the process I suggest  packaging  all the references uniformly using a rubber seal die cast Hammond case of suitable size.

The two additional PXs and three  150K:100K:10K:1K voltage dividers made from precision resistors would be a modest expansion of the current kit. The additional 150k  accommodates the 3478A ranges using a 7 V reference.  It would permit performing calibrations on other ranges on both the 344401A and 3478A class DMMs.

One of my goals is an automated  full bench cal.  That's very bench specific and starts getting rather expensive and heavy when you start switching RF signals.  That's not something a Cal Club can provide.  Some precision attenuators and RF sources are reasonable, as is sharing switching system designs and programs.  But it makes no sense to ship precision attenuators and RF sources to someone who has no RF gear.  All that does is delay access for those who do need it.

All of this takes a long time to accomplish which is why I raised the subject now.  Building two more PXs,  repackaging the current one and building 3 precision voltage dividers in the next year seems an achievable goal.  The PX kits would essentially be higher grade versions of Doug Malone's excellent DMMCheck with more detailed calibration histories at 2-3 month intervals.

While not capable of logging, a 20 uA (50k/V) full scale analog meter will certainly  provide uV null detection.  Those used to be commonly available, but even the standard 50 uA (20K/V) VOM would work.

[hwj-d]

rhb:

What would you suggest about type of resistors, housing and bindingposts it should be for a divider?

[rhb]

I'm expecting input from TiN and others on how we should do that.  At this stage I'm just attempting to articulate requirements.  Not how we get there.  I only have 3478A and 34401A DMMs, so I can't speak to what Keithley and Fluke owners need.

The only comment about how that I have at present is that things should be as robust as possible so that minimal maintenance is needed over a 10-15 year lifespan.

The USA Cal Club: Round 2 waiting list is rather long.  So I wanted to address how to shorten that for Round 3.

I have started a separate thread on the more difficult topic of doing full bench calibrations here:

https://www.eevblog.com/forum/testgear/automating-full-bench-calibrations/msg2590392/#msg2590392

I was able to sort out the general design for switching DC,  low frequency AC and RF over the last couple of days.  The spec on the 44421A card relays is a thermal EMF of under 1 uV.  While that won't meet TiN's requirements, for a general bench it is entirely adequate.  And the 44421/2A cards have provision for reporting the board temperature which would allow improving the accuracy.  The sensor is intended for thermocouples and only implemented on the 44422A terminal block.  But the jumper provisions on the relay card would easily allow placing a sensor on the relay card itself.

Full bench calibrations are a very different subject from the problem of providing access to basic references which is what the Cal Club is all about.  Closely related in that you need the latter to implement the former.  But still very different.

[TiN]

No need to wait for my input, as I'm sadly do not see need of additional complications / components to existing Cal Club R2 necessary. What is needed instead - unified data storage, analysis toolkit and error budget analysis, established by members, so everyone onboard know what is the state of things/condition of references at each given time. Otherwise it's just random numbers on pretty displays without meaning, and spending even $1 for shipping hardware is not worth the effort. So instead of getting everything even more complex, somebody with web-programming skills could help a LOT to a club to make a simple site, that consolidate results, plot some pretty graphs, and have bill-board information where references are, who is next, who is on the list, status of reference, last calibration data, etc. 

Even if set of dividers and additional PX references (still have no clue why need 2 additional PX, triplicating need of their calibration work, instead of using same 1 PX + 1 FX combo?) included - it's not like that magically reduce waiting time for participants. Now participant would need measure not just one or two references but four of them, so how that make things faster? Add dividers, now we measure 4 references 5 times each (as for 100mV,1V,10V,100V,1kV range?). I thought Reg was recently saying he had lot of stress (which we want to avoid!) due to lack of time to get everything logging/working right time even just for 1 measurement and two simple instruments, without trying anything on ppm level? Bit lost here, sorry. 

We are not inventing anything new, BIPM/Metrology labs doing "Calibration clubs" and rounds with references for decades already, so Cal Club idea is just replicating that on much smaller/cheaper level. And if one think waiting time for few months is long, well.. one round between NMIs usually is 2-3 year feat  . Again if there is particular need to calibrate cheap meter like 3478A or 34401A, just shipping those to calibration lab is cheaper and faster, than trying that DIY solution.
I admire and love idea being to calibrate stuff in-house, but it's investment of time and money far above what most participants are willing to manage, as a hobby. Ask me how I know that... 

[maxwell3e10]

I agree that record keeping is the biggest part that is missing so far. As any metrologist knows (which I am not), careful measurements are useless without accurate records. It should be the expectation that members analyse their own data. Keeping a web log would be the best. There was some discussion back on the other thread about what software to use. If there is no dedicated site, then Google sheets would be fine, anything is better than nothing.

[TiN]

maxwell3e10
I'm afraid most hobbyists not doing it (data analysis) because it's the most difficult part of any measurement. I still wrapping mind around some concepts and calculations. 
We all love to poke buttons and see some warm fuzzy 10.000000 V readings on fancy DMM, but that all useless without complete measurement with error factors accounted for.
If you open any BIPM comparison result report, you'll see one paragraph about measurement itself, and dozen PAGES for error calculation and uncertainty math.

[rhb]

The reason for my stress was the length of the waiting list.  If it were shorter I would have been comfortable taking more time .

Having 3 PX references and 3 voltage dividers shortens the waiting list by dividing it in half.  We also gain more data on the effects of transport on  references.  From what I have read so far, that is poorly understood.  With the addition of transit temperature and humidity logging we might learn something.

I expect to spend many hours analyzing the data I collected.  I am back at home but still recovering from surgery, so that will not begin for a while yet.  A friend who is a professor of economics and statistics at Drexel has all my data.  I will upload the PX and ratio data to xdevs.com later today.

My first order questions are:

Is flicker noise zero mean?

Is shot noise zero mean?

I do not recall any mention of this in anything I have read so far on either topic.   And found nothing in my library during a search earlier this week.  So my question to my friend was how does one prove mathematically from the data that a noise process is zero mean in the case of an arbitrary noise process.

If someone has references which discuss either, please post them. 

The only way to get hobbyists to perform PhD level data analysis is to package the requisite mathematics in a piece of software.  I spent my career doing that sort of thing.  I would not trust anything other than bespoke software for such work.  My friend published quite a few papers on gross mathematical  errors in Excel.  It's unlikely that any spreadsheet program would be any better.  Most of my time on such work is spent crafting test cases to verify that the software is correct. The second largest effort is determining what the appropriate calculations are.  Actually writing the software takes the least time. 

[SilverSolder]

What math errors are there in Excel (that are not related to its max 15 significant digit maximum)?

[maxwell3e10]

Just like on the equipment thread there are endless arguments about low-end scopes, one can have endless arguments about software.
There are many choices of free software for data analysis. I personally use Genplot: www.genplot.com.  For something like Allan deviation there are also free analysis tools: www.alamath.com. A lot of other things is just simple record keeping. We are not talking about proper metrology type A and B uncertainty evaluation here. At a basic level, any software can calculate average and standard deviation.

[Grandchuck]

My thoughts:

1/ The person in the center of the star (vindoline currently) should not bear any shipping costs.
2/ Round 2 should finish with no changes (unless necessary due to failures, etc.).
3/ Round 3 could be diversified according to interests and accuracy needed.

[rhb]

I've never read Bruce's papers on Excel.  We're both pretty intense math wonks who met in Bill Guy's  2 semester Integral Transforms course at UT Austin 30+ years ago.  So we only chatted on the phone very briefly about the issues before moving on to more interesting stuff.  He was fed up with reading papers with major errors resulting from the author's use of Excel and went on a campaign to embarrass people out of using Excel as the basis for professional papers in economics and statistics.

I'll ask him for copies. They were pretty serious issues.  These were published in refereed journals.  So *not* minor errors.

I am interested in getting all the information out of the data possible.  I've been very frustrated by plots without data I could analyze independently.  I'm fairly certain I can correct aging errors for a period equal to the length of prior data.  But have been unable to get access to the multiyear data I need to demonstrate doing that.  I'm a retired research scientist, so not a typical hobbyist (except around here where members with PhDs are amazingly common).  But I am pursuing metrology lab level analysis even if I don't have access to metrology lab grade references.

After all, I could reference my entire lab to any approximately correct standard for the volt, ohm, second, etc and all my repair and design calculations would come out fine as long as the implied definitions for each instrument were the same.

As the title states, my proposal is for Round 3, though we might shorten Round 2 if we get the additional PX references ready before then.  Those at the end of the Round 2 list won't get the kit until next year.

Have Fun!
Reg

[hwj-d]

I'm expecting input from TiN and others on how we should do that.  At this stage I'm just attempting to articulate requirements.  Not how we get there. 
...

I see.

[dacman]

In Excel 2003 Microsoft switched to 2-pass statistical functions.  1-pass functions, which some calculators use, are sometimes not as accurate.  (HP calculators that store arrays use 2-pass functions.)  For standard deviation, for the 1-pass function, each number gets squared.  It can take 30 digits to hold the result of a 15 digit number that is squared.  If the deviation is beyond about the seventh digit then the deviation will not be seen by the 1-pass function.  (A deviation in the 15th digit will be seen by the two-pass function.)  If the interim results of the 1-pass function were used to 30 digits, then it would give the same results as the two-pass function.  One workaround was to use offsets.  For example, if a 10 MHz oscillator was being analyzed for stability, 10 MHz could be subtracted from each measurement.  Earlier versions of Excel would then give accurate results if the offsets were analyzed.

[SilverSolder]

Perhaps it is not completely crazy to use Excel to analyze results, it could even be done with some macros or VB routines that everyone shares, to guarantee consistent / comparable results?

[CatalinaWOW]

I am often an advocate for Excel as it is almost universally available in corporate environments, and generally works well enough.  It does have real limitations on large data sets, so I really recommend listening to and evaluating other options.

[rhb]

Octave or R are both free and much better suited to the task.  R is a professional statistician's tool.

Properly analyzing such data is far more work than most would imagine.

Each DMM has a temperature and humidity dependence which must be removed from the data as does each reference.  There are two of everything in a single measurement series.  Two shot noise series, two and possibly as many as four 1/f noise series, thermal EMFs, etc.  I don't even have a full list yet.

I *think* I have enough data to solve for all the terms, but I might not.  I was not prepared for getting the kit and had to make things up on  the fly.  My data might not be useful for anything other than saying that the 10 V range on both my 34401As still meets spec.

I think I might get away with being underdetermined on the 1/f processes, but for most of the stuff you have to be evendetermined to separate the DMM reference from the transfer reference.

[rhb]

Here is a list of Bruce's papers on Excel.  I asked if I could post copies and was told that would lead to trouble with the publishers. 

I don't like spreadsheets because they are very difficult to check and very prone to user error.

 H.-J. Sun, Kaoru Fukuda and B. D. McCullough
"Inaccurate Regression Coefficients in Microsoft Excel 2003: An Investigation of Volpi's Zero Bug"
Computational Statistics 32(4), 1411-1421, 2017

B. D. McCullough and David A. Heiser
"On the Accuracy of Statistical Procedures in Microsoft Excel 2007"
Computational Statistics and Data Analysis 52(10), 4570-4578, 2008

B. D. McCullough
"Microsoft Excel's 'Not the Wichmann-Hill' Random Number Generators"
Computational Statistics and Data Analysis 52(10), 4587-4593, 2008

B. D. McCullough
"The Unreliability of Excel's Statistical Procedures,"
Foresight: The International Journal of Applied Forecasting 3 , 44-45, 2006

B. D. McCullough and Berry Wilson
"On the Accuracy of Statistical Procedures in Microsoft Excel 2003,"
Computational Statistics and Data Analysis 49(4), 1244-1252, 2005

B. D. McCullough
"Fixing Statistical Errors in Spreadsheet Software: The Cases of Gnumeric and Excel"
CSDA Statistical Software Newsletter (2004)
www.csdassn.org/software_reports.cfm

B. D. McCullough and Berry Wilson
"On the Accuracy of Statistical Procedures in Microsoft Excel 2000 and Excel XP,"
Computational Statistics and Data Analysis 40(4), 713-721, 2002

B. D. McCullough
"Does Microsoft Fix Errors in Excel?"
in Proceedings of the 2001 Joint Statistical Meetings, [CD-ROM]
Alexandria, VA: American Statistical Association

B. D. McCullough and Berry Wilson
"On the Accuracy of Statistical Procedures in Microsoft EXCEL 97,"
Computational Statistics and Data Analysis 31(1), 27-37, 1999

[bitseeker]

Thanks for the references, Reg.

Now, for the hard question: How many of these have already been fixed and in which versions?

[SilverSolder]

Thank you @rhb for opening my eyes to the many shortcomings of Excel for statistical work, which I was not aware of at all.

It seems clear that for statistical analyses beyond basic descriptive statistics, Excel is not the best choice.

What Excel has going for it,  is the fact that it is far more widely used than all the professional level tools put together, so does not have much of a learning curve for most people. It also appears to be true that as long as you don't venture too far away from high school level statistics, you'll probably be fine with respect to the quality of the results.

I guess the real question is if calculating the noise of a reference or a DMM requires anything more sophisticated than mean and standard deviation, necessitating the use of more advanced tools?

Even the more advanced metrology topics like Allen deviations etc. could be implemented as a formula or even a VBA function, avoiding the use of any built in functions?

Signed:  Result oriented of Wichita, Kansas

[rhb]

I'm just getting warmed up on analyzing my data.  Give me a month or two to peel away the layers of the onion.  So far all I have done is prove that the noise is Gaussian.

My goal is to quantify and predict reference aging and hysteresis.  I've got a *lot* of work to do to get to that level of error.

I did a spreadsheet to model a business turnaround using OpenOffice.  I found the spreadsheet UI pretty scary from a correctness perspective.  It's extremely difficult to verify that all the fields in a complex spreadsheet are being calculated correctly.  However, that was also my first and only attempt to use a spreadsheet.  Normally I use C, FORTRAN, awk, etc.  So an experienced user might do better.

However, I still would not want to board an airplane designed in a spreadsheet.

[Conrad Hoffman]

Ha! You probably already have. I use Excel almost every day and have never had an issue with non-statistical functions, even out to many significant figures. I will say the graphing is lousy for many of the things I do, and gnuplot usually comes to the rescue there. Without Excel and programming languages I'd be a ditch digger, since I can't remember which variable is what for more than seven seconds. I use a LOT of comments and notes in anything I do. I'm also not above throwing in an extra equation or two, to calculate something using a different method, to see if at agrees with my main program.