Author Topic: Uncertainty Budgets and Decision Rules  (Read 16498 times)

0 Members and 3 Guests are viewing this topic.

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Uncertainty Budgets and Decision Rules
« on: August 09, 2021, 09:05:29 am »
As I have been spending a fair bit of my work time on these above subjects. Would people here like to know more about it? I'm no expert but I am happy to show people the stuff I know and hopefully it will help the next person. I often see people here going on about the 1yr manufactuer's spec of their bit of test gear or even quoting the 24h spec. But the measurement is much more than that.

The new 17025 documents are putting more pressure on labs to have a decision rule and that is a good thing as there are lots of labs stating stuff is passed when actually given the barn door of an uncertainty they have it could well be undetermined. It is quite funny looking at mechanical calibration certs where the Unc is wider than the spec but even funnier is that labs had an Unc for length but they never had one for flatness or parallelism of that gauge, now they have to and it is causing lots of cuffufle. The alternative is labs just don't say it conforms to a spec but I get a feeling that won't slide for much longer as customers assume compliance of a gauge that has been calibrated.
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 
The following users thanked this post: try, e61_phil, bck, MiDi, Anders Petersson

Offline e61_phil

  • Frequent Contributor
  • **
  • Posts: 963
  • Country: de
Re: Uncertainty Budgets and Decision Rules
« Reply #1 on: August 09, 2021, 11:31:11 am »
Hi,

I think that could be a real eye opener for many people here. Many people are thinking they can measure with lower uncertainties than most of the cal labs, just because they own a 3458A or something like that ;)

Best regards
Philipp
 
The following users thanked this post: try, bck

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 20750
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Uncertainty Budgets and Decision Rules
« Reply #2 on: August 09, 2021, 12:25:09 pm »
That would be valuable particularly if we can point people to it, thus avoiding hasty half-correct statements.

Since there's no way you would be comprehensive, a "30000 ft" overview plus pointers to more detailed info might be appropriate.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline FransW

  • Frequent Contributor
  • **
  • Posts: 270
  • Country: nl
Re: Uncertainty Budgets and Decision Rules
« Reply #3 on: August 09, 2021, 02:04:19 pm »
Generally a minimum awareness of absent knowledge is necessary or required to appreciate and understand the limits of a piece of test equipement used.

It certainly pays to elevate a basic knowledge of physics to appreciate the limits of the individual in question, the test equipment in question and the DUT in question.

Working in a certified lab with accreditation (Laboratory quality management system review) should prevent under-achievement.

ISO / IEC 17025:It is an international laboratory management standard that guarantees the accuracy of laboratory data results.

ILAC-MRA:ILAC (The International Laboratory Accreditation Cooperation) is a global association of accredited laboratories, inspection bodies, proficiency testing provider bodies and reference material producer, whose members include accreditation bodies and stakeholder organizations around the world. Accreditation bodies around the world are peer-assessed to be capable, signing agreements to enhance acceptance of cross-border products and services.
PE1CCN, Systems Engineering, HP, Philips, TEK, BRYMAN, Fluke, Keithley
 

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #4 on: August 09, 2021, 02:17:26 pm »
Measurement uncertainty is just defining the statistical probability of how good your measurement capability is to a 95% probability where k=2. UKAS has published some interesting documents talking about the subjects.

M3003-Expression-of-Uncertainty-and-Confidence-in-Measurement-Edition-4-October-2019.pdf & LAB-48-Decision-Rules-Edition-3-June-2020.pdf

They are fairly accessible and show the importance of looking at Uncertainty and how the lab that calibrated your equipment influences your measurements and if anyone uses your measurements how that can affect others. Further away from government standards the worse it gets.

It takes in lots of elements but once you get your head around it, it is not as complicated as people fear.

Now there are some very good write-ups on it on websites such as ISO Budgets - 7 Steps to Calculate Measurement Uncertainty

Just so people are aware I am a Lab Manager for an ISO 17025 UKAS lab in the UK and I have had to pick up the pieces left by colleagues who have left the company so my know-how is from learning as I go. I am no expert in the matter, I'm just working it out for myself at times but we are a small lab and don't do a lot of UKAS work our bread and butter in is onsite work but many of our customers expect us to be UKAS accredited to do the work we do even if it is not to UKAS. I have just spent the past few months working towards getting 17025 in the mechanical lab and I much prefer electronics much easier.

What I am thinking is to show you an example of a budget for a digital multimeter I have here at home (121GW) and explaining where all the bits I have, go. Therefore allowing other people to build up a budget for their needs.
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 
The following users thanked this post: alm, FransW

Offline FransW

  • Frequent Contributor
  • **
  • Posts: 270
  • Country: nl
Re: Uncertainty Budgets and Decision Rules
« Reply #5 on: August 09, 2021, 02:41:10 pm »
It is not only:
"Measurement uncertainty is just defining the statistical probability of how good your measurement capability is to a 95% probability where k=2. UKAS has published some interesting documents talking about the subjects."

But including the probalitity in perspective of physics today.
The impossible is done immediately, wonders take a bit more time and on request there is magic. For example quantum uncertainty and entanglement.
PE1CCN, Systems Engineering, HP, Philips, TEK, BRYMAN, Fluke, Keithley
 

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #6 on: August 09, 2021, 02:55:43 pm »
It is not only:
"Measurement uncertainty is just defining the statistical probability of how good your measurement capability is to a 95% probability where k=2. UKAS has published some interesting documents talking about the subjects."

But including the probalitity in perspective of physics today.
The impossible is done immediately, wonders take a bit more time and on request there is magic. For example quantum uncertainty and entanglement.

No need to get into quantums and it's entanglements we all know the answer is 42 but we just don't know the question. If pedantry is going to be the theme then I will just wander off now.
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 

Offline FransW

  • Frequent Contributor
  • **
  • Posts: 270
  • Country: nl
Re: Uncertainty Budgets and Decision Rules
« Reply #7 on: August 09, 2021, 03:17:52 pm »
That is not the point I am trying to make.\My point is that the assumption of in-depth knowledge is often over-sahdowed by lack of sufficient knowledge.
PE1CCN, Systems Engineering, HP, Philips, TEK, BRYMAN, Fluke, Keithley
 

Offline bdunham7

  • Super Contributor
  • ***
  • Posts: 8010
  • Country: us
Re: Uncertainty Budgets and Decision Rules
« Reply #8 on: August 09, 2021, 04:14:23 pm »
I would be interested in a discussion of the matter because while I don't fully understand all the issues, I think there is a wide range of misunderstanding about what you do and don't get from a calibration lab.  And quite frankly, while perhaps UKAS is better at this, I see a lot of what I consider obfuscation and misdirection in the policies of a lot of so-called cal labs here in the US.  That may seem harsh, but in my view the common no-data cal cert is just a way for underperforming cal labs to collect money for work they are unqualified for from customers that often don't care.  I would like to hear from a non-OEM cal lab about how they actually deal with the exact topics you listed, as well as to what standards a device like a DMM should actually be calibrated to.  For example, if I send one of my 6.5 digit DMMs for calibration, what I want is before/after data and all adjustments made to better than the 24-hour specifications with equipment demonstrated to have an accuracy uncertainty better than a 5:1 TUR relative to those 24-hour specs or alternatively, at least 3:1 TUR with sufficient guardbanding used.  And I'd like that in a 23C +/1 1C environment with a 72 hour acclimatization and a 2-hour warmup.  A typical sales rep from a typical cal lab will flubber around for half an hour about their various services and accreditations without managing to actually address any of the issues.

Here is something from a long-ago thread about a dodgy eBay voltage reference.  The dodgy reference isn't the issue I'm interested in, this cal certificate is.  Comments?




« Last Edit: August 09, 2021, 08:52:15 pm by bdunham7 »
A 3.5 digit 4.5 digit 5 digit 5.5 digit 6.5 digit 7.5 digit DMM is good enough for most people.
 

Offline try

  • Regular Contributor
  • *
  • Posts: 112
  • Country: de
  • Metrology from waste
Re: Uncertainty Budgets and Decision Rules
« Reply #9 on: August 09, 2021, 04:28:31 pm »
Hello mendip_discovery,

maybe your contribution could make some folks rethink their quantitative postings that are flooded with insignificant digits. ::)
 
The following users thanked this post: e61_phil, mendip_discovery

Offline try

  • Regular Contributor
  • *
  • Posts: 112
  • Country: de
  • Metrology from waste
Re: Uncertainty Budgets and Decision Rules
« Reply #10 on: August 09, 2021, 05:04:44 pm »
Hello bdunham7,


Here is something from a long-ago thread about a dodgy eBay voltage reference.  The dodgy reference isn't the issue I'm interested in, this cal certificate is.  Comments?

DISCLAIMER:
please be aware that I don't work in a cal lab and that I am not certified in any way!

1. Nearly 54 ppm measurement uncertainty with k=2 is not something I would pay any dime for.

2. Maker and models of the calibration instruments used are omitted - obviously for a reason. They only show their IDs. :--
3. M. Smith did not sign this certificate.
4. Furthermore there is no description of the measurement setup.
A good lab will provide you with a picture that shows you even the wiring setup during measurement and the id of the cable used.
5. There is no reference to the id of the document containing the calibration procedure.

Let me guess, the ebay voltage reference was sold with an uncertainty of +- 2ppm slightly ignoring the 54 ppm shown above?  :palm:

I can't see anything documenting NIST traceability:

https://www.foxvalleymetrology.com/fvm-advantage/quality/accreditations/

The next lab from where I live is Perschmann, located in Braunschweig, where the PTB (German primary lab) is residing as well.

You can download their accreditation if you want to:

https://www.perschmann-calibration.de/wp-content/uploads/Akkreditierung_Anlage_DAkkS_DIN_EN_ISO-IEC_170252018_D-K-15089-01-00_2019_E2_12.01.2021.pdf

Look at page 9 for the voltage figures.
I did not find any similar NIST related document with Foxy Valley.  8)









« Last Edit: August 09, 2021, 05:34:16 pm by try »
 

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #11 on: August 09, 2021, 05:54:32 pm »
I see a lot of what I consider obfuscation and misdirection in the policies of a lot of so-called cal labs here in the US.  That may seem harsh, but in my view the common no-data cal cert is just a way for underperforming cal labs to collect money for work they are unqualified for from customers that often don't care.

This I admit is common, most customers don't know and don't look at a cert. They just accept it as a necessary evil that has to be done. You do get some that look and you get the odd question or two. I have just had to explain to a test lab what Uncertainty is and isn't. He assumed it was the specification. I think its partly down to information about it being kept to the nerds and not spoken about in public. Most customers just want to know is it in spec.

I would like to hear from a non-OEM cal lab about how they actually deal with the exact topics you listed, as well as to what standards a device like a DMM should actually be calibrated to.  For example, if I send one of my 6.5 digit DMMs for calibration, what I want is before/after data and all adjustments made to better than the 24-hour specifications with equipment demonstrated to have an accuracy better than a 5:1 TUR relative to those 24-hour specs or alternatively, at least 3:1 TUR with sufficient guardbanding used.  And I'd like that in a 23C +/1 1C environment with a 72 hour acclimatization and a 2-hour warmup.  A typical sales rep from a typical cal lab will flubber around for half an hour about their various services and accreditations without managing to actually address any of the issues.

The whole idea of ratio should be downgraded as this is an old school method of thought and often lead people along the line of assuming as this is more accurate its better and it often doesn't take other things into account and in many cases is just not possible if you do a proper budget. I remember chatting to a friendly UKAS auditor about this and they did say it was a very strong thought process in America.

The Certificate should make it clear the method used to do the calibration. For example "The voltage source was left in a temperature controlled environment and allowed to stabilise for a minimum of 24 hours, then the equipment was turned on and allowed to warm up for 1hr prior to the calibration where it was measured using digital multimeter" this is so that if it was sent to another lab then the method could be reproduced. Because you may have compared it to a known voltage source using a DMM etc etc.

Our lab is 22C +/-2 and for electrical that is perfectly fine and its common for UK elec labs to run at this temp because of the yanks and the cheapness on the aircon, that was the story told me, I think the 22C comes via Fluke. Otherwise, it would be 20C like we do in the mech labs.

From what I have seen it is the 1yr spec that labs aim for as the 24hr spec is almost what I would call repeatability in the absence of a repeatability study. Think of it this way, unless you get it calibrated every 24hours then that spec is worthless.

If no adjustments have been made then the cert should say that its not been adjusted. If it has then you need to show before and after adjustment results. Because you need to work out the risk its had to your prior measurements let's say it's 0.5V out of spec, you would need to check that you hadn't had anything passed or failed because of that error then maybe look at recalling those jobs to rectify your measurements. Yes, it is a big can of worms if that happens.


Here is something from a long-ago thread about a dodgy eBay voltage reference.  The dodgy reference isn't the issue I'm interested in, this cal certificate is.  Comments?



That is an interesting cert in that they don't make it clear if it was compared to a reference standard though there are hints about it but in a confusing manner. They say before and after adjustment but the results are the same so no adjustment has been made. The Temp looks a little chilly and hum is nice and dry not sure if that is a little too dry, also it doesn't state the lab tolerance on temp, it might drift by 5C over 24hrs so the kit could have been in 14C and only 1hr before taken up to 19C

Upper and lower specifications are confusing as hell as they have the wrong specifications there. I would be aiming for a measurement in the 6ppm for the general measurement. Also, they should measure the other outputs. Now I would assume there is a separate Unc for the 24hr constant measurements as that would differ to a single measurement.

Now the interesting thing is they state compliance and yet the Unc would say they can't make that statement in today's rules with decision rule. See attached, which is going on a 6ppm Tolerance.
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #12 on: August 09, 2021, 06:04:25 pm »
DISCLAIMER:
please be aware that I don't work in a cal lab and that I am not certified in any way!

Wuss. But it only takes a moment to see issues, but to an average customer, there is plenty to say its ok.

1. Nearly 54 ppm measurement uncertainty with k=2 is not something I would pay any dime for.
2. Maker and models of the calibration instruments used are omitted - obviously for a reason. They only show their IDs. :--
3. M. Smith did not sign this certificate.
4. Furthermore there is no description of the measurement setup.
A good lab will provide you with a picture that shows you even the wiring setup during measurement and the id of the cable used.
5. There is no reference to the id of the document containing the calibration procedure.

1. Yup that is shocking.
2. Not a requirement, I don't show the kit used. I do need to record what kit was used but the customer doesn't need to know. But they do need to know the method, aka measured using a DMM.
3. Yeah not a good sign.
4. Again not a requirement to show the way it was connected, the uncertainty will be based upon their procedure and leads used. Another lab might have a lower Unc due to a better setup. I have gone to PTFE coated and twisted wires to reduce my Unc.
5. Yes it does, FVE-000 and FVE-011.


Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 

Offline bdunham7

  • Super Contributor
  • ***
  • Posts: 8010
  • Country: us
Re: Uncertainty Budgets and Decision Rules
« Reply #13 on: August 09, 2021, 06:37:24 pm »
The whole idea of ratio should be downgraded as this is an old school method of thought and often lead people along the line of assuming as this is more accurate its better and it often doesn't take other things into account and in many cases is just not possible if you do a proper budget. I remember chatting to a friendly UKAS auditor about this and they did say it was a very strong thought process in America.

What would you replace TUR/guardbands with?  It's not just a school of thought, it is both manufacturer and official policy still, AFAIK.  I happen to agree with the concept that more accurate is better and I understand that it can be hard on the 'budget'--both the uncertainty budget and the bank budget.

Quote
From what I have seen it is the 1yr spec that labs aim for as the 24hr spec is almost what I would call repeatability in the absence of a repeatability study. Think of it this way, unless you get it calibrated every 24hours then that spec is worthless.

Yes, a 1 year cal is often done to 1 year specs.  The problem with that is the 1-year specs include a years worth of drift and typically +/- 5C worth of tempco.  So you have 24-hour +/-1C specs vs 1-year +/-5C.  At the factory, the meter will presumably be adjusted to the 24-hour specs in a +/-1C environment and the reason the OEM doesn't just list that as the specification for a year is that they are including the additional drift and tempco as well as a confidence interval.   It isn't about any guarantee that the meter will retain its 24-hour specs--for that  you would indeed have to calibrate every 24 hours.  It's about assuring that the meter will remain within the 1-year specs for a year and from 18C to 28C.  The reason that the 'typical' performance of a good DMM is much better than the specified performance is simply that it is necessary to insure that k=2 or 2.58 as the case may be.  If you test at at just one controlled temperature and accept any value that falls within the 1-year specs, you have not assured that the meter will remain accurate for the calibration period or over the specified temperature range nor do you have any sort of confidence interval, no matter how accurate your references are.  Somebody correct me if I'm wrong about any of that. 
A 3.5 digit 4.5 digit 5 digit 5.5 digit 6.5 digit 7.5 digit DMM is good enough for most people.
 

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #14 on: August 09, 2021, 06:58:22 pm »
Yes, a 1 year cal is often done to 1 year specs.  The problem with that is the 1-year specs include a years worth of drift and typically +/- 5C worth of tempco.  So you have 24-hour +/-1C specs vs 1-year +/-5C.  At the factory, the meter will presumably be adjusted to the 24-hour specs in a +/-1C environment and the reason the OEM doesn't just list that as the specification for a year is that they are including the additional drift and tempco as well as a confidence interval.   It isn't about any guarantee that the meter will retain its 24-hour specs--for that  you would indeed have to calibrate every 24 hours.  It's about assuring that the meter will remain within the 1-year specs for a year and from 18C to 28C.  The reason that the 'typical' performance of a good DMM is much better than the specified performance is simply that it is necessary to insure that k=2 or 2.58 as the case may be. 

Yes, the 1yr is what you can safely assume it will be within a year to year. The 24 hrs is what that claim it will do over 24hrs. The idea I have seen is that the 24hr performance is what you would use if you were building a budget for its ability to measure a for a whole 24hours. Let's say you wanted to measure the output of 10V source over 24hours taking readings every 10mins, you would take that 2ppm spec and add that to your Unc because over 24 hours it would be expected to drift within that spec. I would say any measurement Unc for a 24hour measurement would be higher than a normal quoted Unc as it has extra issues. Most devices even at 24hr will have a spec including the +/-5C drift in temp as the insides will wander as well as the external, thermal effects and all that.

If you test at at just one controlled temperature and accept any value that falls within the 1-year specs, you have not assured that the meter will remain accurate for the calibration period or over the specified temperature range nor do you have any sort of confidence interval, no matter how accurate your references are.  Somebody correct me if I'm wrong about any of that.

It is saying on that day in these conditions and with my Unc that it measured this when that was applied. Often certs will say this does not imply long term stability. Its a check that the meter is still in spec. You need to look at it over the years to be confident it won't drift out.
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 

Offline bdunham7

  • Super Contributor
  • ***
  • Posts: 8010
  • Country: us
Re: Uncertainty Budgets and Decision Rules
« Reply #15 on: August 09, 2021, 07:24:19 pm »
Its a check that the meter is still in spec. You need to look at it over the years to be confident it won't drift out.

Indeed, most calibration is just a check to make sure the device is still in spec.  However, although years of data may be nice to have, it is the manufacturers design and specifications that provide the assurance that drift and tempco will not cause it to go out of spec within the relevant time/temp/etc limits, to a certain level of confidence.  And that is predicated at least partially on the meter being dialed in right at the center at the beginning of the calibration interval.  Observation and testing over time may give you the ability to assign an uncertainty lower than the OEM original specs--which is how most labs achieve their uncertainty numbers, right?

Quote
2. Not a requirement, I don't show the kit used. I do need to record what kit was used but the customer doesn't need to know. But they do need to know the method, aka measured using a DMM.

It may not be an ISO 17025 or whatever requirement, but this is what I was referring to by obfuscation.  Why would you not want your customers to know this?  And what would your best guess be as to what was used in this case?
A 3.5 digit 4.5 digit 5 digit 5.5 digit 6.5 digit 7.5 digit DMM is good enough for most people.
 

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #16 on: August 09, 2021, 07:41:03 pm »
Its a check that the meter is still in spec. You need to look at it over the years to be confident it won't drift out.

Indeed, most calibration is just a check to make sure the device is still in spec.  However, although years of data may be nice to have, it is the manufacturers design and specifications that provide the assurance that drift and tempco will not cause it to go out of spec within the relevant time/temp/etc limits, to a certain level of confidence.  And that is predicated at least partially on the meter being dialed in right at the center at the beginning of the calibration interval.  Observation and testing over time may give you the ability to assign an uncertainty lower than the OEM original specs--which is how most labs achieve their uncertainty numbers, right?

Yup that is how you do it.

Quote
2. Not a requirement, I don't show the kit used. I do need to record what kit was used but the customer doesn't need to know. But they do need to know the method, aka measured using a DMM.
It may not be an ISO 17025 or whatever requirement, but this is what I was referring to by obfuscation.  Why would you not want your customers to know this?  And what would your best guess be as to what was used in this case?

It is not required and not even a recommendation. It shouldn't matter to the customer what I used, they just need to know the method and that I have a good Unc. I think part of the thinking behind it is to prevent armchair experts that go pah they use a puny meter, not knowing that lab has had it for 10 years and it is rock steady etc etc. also what do you do if you are using a custom bit of kit that you have proven works, for example, an inhouse resistance standard.
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 
The following users thanked this post: bck

Offline bdunham7

  • Super Contributor
  • ***
  • Posts: 8010
  • Country: us
Re: Uncertainty Budgets and Decision Rules
« Reply #17 on: August 09, 2021, 07:59:59 pm »
It is not required and not even a recommendation. It shouldn't matter to the customer what I used, they just need to know the method and that I have a good Unc. I think part of the thinking behind it is to prevent armchair experts that go pah they use a puny meter, not knowing that lab has had it for 10 years and it is rock steady etc etc. also what do you do if you are using a custom bit of kit that you have proven works, for example, an inhouse resistance standard.

Well, I hope this remains a friendly conversation, but as a customer if you don't think I'm worthy of this information or if you think it is OK to not reveal it because I might not interpret it correctly, we aren't off to a good start.  This just reinforces things I've said previously about cal labs.  If you have a homemade contraption that works, show it to us and show us the data that proves it works--and let the customer decide.  If the customer wants their stuff calibrated only by techs wearing a suit, tie and leather shoes, that is their call.  In my case I would just want it to be accurate, no matter how it gets there.

Here's an example from a presumably reputable (not known to me personally) independent cal lab.

https://www.esscolab.com/uploads/files/8508A_Example_Cert_w_Data.pdf
A 3.5 digit 4.5 digit 5 digit 5.5 digit 6.5 digit 7.5 digit DMM is good enough for most people.
 

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #18 on: August 09, 2021, 08:01:07 pm »
My first step towards a budget would be to get more than 1 calibration on your device so you can prove to yourself that it is within spec and that over a period of time it isn't drifting between calibrations.

Below is a picture of the kind of thing I do this. I have just chosen the 50V DC linearity as an example but when you build a complete budget you would have to cover the whole of the DC, AC etc I think this is good practice as it makes you look at the results and then you can start to work out the mistakes if there is any on the cert. As you can see my 121GW isn't too brilliant. I think I might take it into work and do a good job on it and even run through the adjustment procedure to get it back inside the limits. The good thing is in 1 year the drift between calibrations isn't too bad. So it is repeatedly inaccurate.

But this won't affect my ability to calibrate it as I don't plan to measure using the negative side of the meter so it's within specification, just.


But over time if it's well behaved you can get a graph like this and then you can start to think about reducing the Spec to bring your Unc down.

« Last Edit: August 09, 2021, 08:03:43 pm by mendip_discovery »
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 
The following users thanked this post: bck

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #19 on: August 09, 2021, 08:14:53 pm »
It is not required and not even a recommendation. It shouldn't matter to the customer what I used, they just need to know the method and that I have a good Unc. I think part of the thinking behind it is to prevent armchair experts that go pah they use a puny meter, not knowing that lab has had it for 10 years and it is rock steady etc etc. also what do you do if you are using a custom bit of kit that you have proven works, for example, an inhouse resistance standard.

Well, I hope this remains a friendly conversation,
:box:  :D

But as a customer if you don't think I'm worthy of this information or if you think it is OK to not reveal it because I might not interpret it correctly, we aren't off to a good start.  This just reinforces things I've said previously about cal labs.  If you have a homemade contraption that works, show it to us and show us the data that proves it works--and let the customer decide.  If the customer wants their stuff calibrated only by techs wearing a suit, tie and leather shoes, that is their call.  In my case I would just want it to be accurate, no matter how it gets there.

I have a few customers who like to know the kit when it is due for cal etc. But there are only a few. Stating a recalibration date for my kit on a cert is just daft. I do regular checks crosschecks and depending on those results I may extend or reduce the calibration frequency and therefore it's my risk to manage.

I don't need to list the kit as UKAS as the accrediting body is happy with the work I am doing. Why should you think otherwise? It's a bit like saying to the people at NPL I am not sure you know what you are doing is right.

But you are as a customer or even potential customer welcome to come and audit us and see the lab, the equipment, the records etc. Under ISO9001 its expected for you to audit your suppliers especially if the work you do for them is classed as a high risk with regards to failures of products.
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 

Offline bdunham7

  • Super Contributor
  • ***
  • Posts: 8010
  • Country: us
Re: Uncertainty Budgets and Decision Rules
« Reply #20 on: August 09, 2021, 08:23:52 pm »
My first step towards a budget would be to get more than 1 calibration on your device so you can prove to yourself that it is within spec and that over a period of time it isn't drifting between calibrations.

Well, even with 1 cal you still have the factory cal to rely on, so that's 2 right?  If your gear is characterized spot on to a certain uncertainty  :) , you should be able to calculate the upper and lower bounds of drift, right?

Quote
then you can start to work out the mistakes if there is any on the cert. As you can see my 121GW isn't too brilliant. I think I might take it into work and do a good job on it and even run through the adjustment procedure to get it back inside the limits. The good thing is in 1 year the drift between calibrations isn't too bad. So it is repeatedly inaccurate.

First, if you haven't recorded an accurate temperature, how can you know how much of what little drift there is might be due to tempco?  Perhaps I harp on tempco a lot, but I've observed it to be the single largest cause of deviation in the mid-grade bench DMMs that I work on and play with.  Second, isn't the 121GW  0.05% + 5 counts on that range (not 0.01%)?  So your meter appears to still be in spec?
A 3.5 digit 4.5 digit 5 digit 5.5 digit 6.5 digit 7.5 digit DMM is good enough for most people.
 

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #21 on: August 09, 2021, 09:04:52 pm »
My first step towards a budget would be to get more than 1 calibration on your device so you can prove to yourself that it is within spec and that over a period of time it isn't drifting between calibrations.

Well, even with 1 cal you still have the factory cal to rely on, so that's 2 right?  If your gear is characterized spot on to a certain uncertainty  :) , you should be able to calculate the upper and lower bounds of drift, right?

But I don't know the acceptance limit for the manufacturer as I didn't get a cert as such, I have done it from Nominal for the first year. This is why kids its important to get a cert with your toys and if you buy used its really nice if they can give you the old certs.

First, if you haven't recorded an accurate temperature, how can you know how much of what little drift there is might be due to tempco?  Perhaps I harp on tempco a lot, but I've observed it to be the single largest cause of deviation in the mid-grade bench DMMs that I work on and play with.  Second, isn't the 121GW  0.05% + 5 counts on that range (not 0.01%)?  So your meter appears to still be in spec?

On the worksheet, I recorded the temp and hum for both tests and I see in the manual there is a temp coefficient I can apply to the budget. I am going to do the same as Keysight gear and assume a 5C swing.

Ooh the Spec has changed, it was 0.01% in the Sep 19 edition. Them crafty Aussies.

When you say accurate temperature, you want to know I had to do an Unc budget for the thermometer used in the lab so that I could use that error in the error I use for a coefficient calculation.
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 

Offline bdunham7

  • Super Contributor
  • ***
  • Posts: 8010
  • Country: us
Re: Uncertainty Budgets and Decision Rules
« Reply #22 on: August 09, 2021, 09:38:23 pm »
But I don't know the acceptance limit for the manufacturer as I didn't get a cert as such, I have done it from Nominal for the first year. This is why kids its important to get a cert with your toys and if you buy used its really nice if they can give you the old certs.

Good point--I would think that anyone seriously concerned about accuracy would get a cert with purchase, but in fact even my 8846A did not come with any data.  They do now.  I would presume at least the 24-hours specs for an acceptance limit for those manufacturers that list a 24-hour spec.

Quote
Ooh the Spec has changed, it was 0.01% in the Sep 19 edition. Them crafty Aussies.

That must be a misprint, the original manual from 2018 and the Kickstarter page also show 0.05%.  0.01% would probably make it the most accurate handheld DMM in existence by a wide margin.

https://www.eevblog.com/files/EEVblog-121GW-Manual.pdf

Quote
When you say accurate temperature, you want to know I had to do an Unc budget for the thermometer used in the lab so that I could use that error in the error I use for a coefficient calculation.

Perhaps, but at some point if you can demonstrate that the effect of an error is much less than other uncertainties, I would think you could drop it.  If your thermometer is accurate to 0.5C, I wouldn't worry about it unless you are required to.  After all, you can start throwing in electric field, gravitational pull, sound level and background radiation if you like--all of which theoretically could have an effect--but at some point they don't matter enough to worry about. 
A 3.5 digit 4.5 digit 5 digit 5.5 digit 6.5 digit 7.5 digit DMM is good enough for most people.
 

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #23 on: August 09, 2021, 09:51:24 pm »
Next, I would look at the repeatability of the device and build up some standard deviations for the measurement. I did this with the 5V source I have here and took a reading every 5 mins. Would prefer it to be at 50V but this is just an example so meh.



E8    =COUNT(E2:E6)
E9    =AVERAGE(E2:E6)
E10  =STDEV(E2:E6)
E11  =(E10/SQRT(E8))
E12  =(E11/E9)*1000000
E13  =(E11/E9)*100

The limiting factor is the resolution and that does give you a bit of a hit with regards to your imported ppm.
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 

Offline mendip_discoveryTopic starter

  • Frequent Contributor
  • **
  • Posts: 998
  • Country: gb
Re: Uncertainty Budgets and Decision Rules
« Reply #24 on: August 09, 2021, 10:06:06 pm »
As I am someone who likes to show my working out I often will have a tab just for the Contributions, lots of labs write it straight into the Budget but I like to do the maths then at least when I get it wrong I can correct it and the budget gets filled in as well.



As you can see it is fairly basic and I haven't had to take in the magnetic flux caused by bdunham7 spinning in his sleep. You can add more, if you can warrant it. I get the info from the specs etc then I convert it to the same resolution as I will be working in ppm and mV otherwise the % and ppm stuff can get confusing. I think the only slightly complicated one is the Temp Coefficient, and that is from the manual and with most good multimeters they are happy with a +/-5C but I don't have that so I did the % at 18C and at 26C and minus one from the other getting me the error I have from 22+/-4 if you can control your space even better then you can get this down more. Just be wary of saying you are at +/-0.5C as I bet you your thermometer doesn't have that good of an uncertainty and yes I have had to calculate that before.

If you are doing pressure gauges you have to take into account local air pressure, and local gravity. Thread gauges, rake and compression. Let us just say electrical/digital is much nicer.
Motorcyclist, Nerd, and I work in a Calibration Lab :-)
--
So everyone is clear, Calibration = Taking Measurement against a known source, Verification = Checking Calibration against Specification, Adjustment = Adjusting the unit to be within specifications.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf