USA Cal Club Round 3

[RandallMcRee]

The kit has arrived in North Carolina and I'm logging away! 

One of the banana jacks had a strain relief fatality. The connection is redundant so it isn't an immediate problem, but since I am not very far down the volt nut rabbit hole I thought I'd check in before trying to fix it. Can I just strip back the wire, trim its partner to match, and re-seat the terminals? If I need to apply special anti-corrosion schmoo or perform ppm-preserving rituals I want to make sure to do it right.

Fatigue on the banana jacks? Not surprising!
Any voodoo is in the *lack* of voodoo. That is to say, as far as I know, we just rely on the cable being pure copper and the jack being gold-plated along with some good mechanical connection. So, yes, go ahead and fix as you proposed. You do not need to apply any de-oxit/cramolin so long as you do not smear the cable with your hair grease!
(That was a joke)

Thanks,
Randall

[SilverSolder]

I've gotten to the point of looking at the data from the experiment. 

This being 2023, I asked ChatGPT for some ideas on how to analyze the numbers - here's what GPT-4 suggests:

Analyzing your data can be done in several ways. For multi-dimensional data, you can use a pairplot or scatterplot matrix, which will give you a two-dimensional scatterplot for each pair of your four variables (voltmeters). This way, you can visualize any pairwise relationships in your data.

Here are some techniques you could use:

1. Pairplot or Scatterplot Matrix: This is a great way to visualize multidimensional data. Each scatterplot in the matrix represents the relationship between two of your variables, and the diagonal of the matrix often contains histograms or density plots of each variable. In Python, you can easily create this kind of plot using the seaborn library's pairplot function.

2. Correlation Matrix: A correlation matrix would show how correlated each pair of variables is. This would be a way to quantify the relationship that you see in the scatterplot matrix. Correlation values range from -1 (perfect negative correlation) to 1 (perfect positive correlation), with 0 indicating no correlation.

3. Time-Series Plots: You can create separate time-series plots for each of the DMMs. Plotting the readings over time would help identify any drift or other time-related patterns in the readings.

4. Box Plots: These can provide a summary of the central tendency, dispersion and skewness in your data, and can also help to identify outliers.

5. Histograms and Density Plots: These could be useful to visualize the distribution of the readings from each DMM.

6. Statistical Tests: Depending on your goals, there are a variety of statistical tests you could use to compare the measurements from the different DMMs. For example, a paired t-test or a one-way ANOVA could be used to test if there is a statistically significant difference in the means of the measurements from the different DMMs.

7. Regression Analysis: If you logged the temperature and humidity, you could also perform a multivariate regression analysis to see how these factors relate to the measurements from the DMMs.

8. Allan Deviation: As discussed earlier, Allan deviation could be used to assess the stability of the DMMs over time.

Remember that visualizations can often help to reveal patterns or relationships in your data that aren't immediately obvious from the raw numbers, so it's a good idea to try several different kinds of plots and analyses. Once you have a sense of what's going on in your data, you can then use more focused statistical tests or models to quantify those patterns or test specific hypotheses.



So the question is -  how good of an engineer is ChatGPT?  Is there a better way to do this?

[rhb]

I've gotten to the point of looking at the data from the experiment. 

This being 2023, I asked ChatGPT for some ideas on how to analyze the numbers - here's what GPT-4 suggests:

Analyzing your data can be done in several ways. For multi-dimensional data, you can use a pairplot or scatterplot matrix, which will give you a two-dimensional scatterplot for each pair of your four variables (voltmeters). This way, you can visualize any pairwise relationships in your data.

Here are some techniques you could use:

1. Pairplot or Scatterplot Matrix: This is a great way to visualize multidimensional data. Each scatterplot in the matrix represents the relationship between two of your variables, and the diagonal of the matrix often contains histograms or density plots of each variable. In Python, you can easily create this kind of plot using the seaborn library's pairplot function.

2. Correlation Matrix: A correlation matrix would show how correlated each pair of variables is. This would be a way to quantify the relationship that you see in the scatterplot matrix. Correlation values range from -1 (perfect negative correlation) to 1 (perfect positive correlation), with 0 indicating no correlation.

3. Time-Series Plots: You can create separate time-series plots for each of the DMMs. Plotting the readings over time would help identify any drift or other time-related patterns in the readings.

4. Box Plots: These can provide a summary of the central tendency, dispersion and skewness in your data, and can also help to identify outliers.

5. Histograms and Density Plots: These could be useful to visualize the distribution of the readings from each DMM.

6. Statistical Tests: Depending on your goals, there are a variety of statistical tests you could use to compare the measurements from the different DMMs. For example, a paired t-test or a one-way ANOVA could be used to test if there is a statistically significant difference in the means of the measurements from the different DMMs.

7. Regression Analysis: If you logged the temperature and humidity, you could also perform a multivariate regression analysis to see how these factors relate to the measurements from the DMMs.

8. Allan Deviation: As discussed earlier, Allan deviation could be used to assess the stability of the DMMs over time.

Remember that visualizations can often help to reveal patterns or relationships in your data that aren't immediately obvious from the raw numbers, so it's a good idea to try several different kinds of plots and analyses. Once you have a sense of what's going on in your data, you can then use more focused statistical tests or models to quantify those patterns or test specific hypotheses.



So the question is -  how good of an engineer is ChatGPT?  Is there a better way to do this?

All of those are sound basic tests.  I did this in the context of running an "orphan home for lost problems" at  large oil companies including  super-majors.  Those  are all basic, *essential* things to be done *before* you proceed with more detailed and focused methods.  In my case, those were design and write the code, apply it to the data and then proceed to the next such step in the process.  The suggestions are all part of  the "exploratory data analysis phase" that should be applied to *all* data as the first step.

ChatGPT makes up data, so it's not reliable as a lawyer recently discovered.  He asked for some additional citations which he included in his brief.  He got a stern rebuke when the judge discovered the citations were fabricated by chatGPT. 

Gnuplot is the best tool for doing this.  I don't think it does Alan deviation or ANOVA,  but the Marquard-Levinburg implementation is the best L2 solver I've used.

Have Fun!
Reg

[Conrad Hoffman]

I wouldn't trust ChatGPT as far as I could throw it. OTOH, Gnuplot is great, though it has a significant learning curve to take full advantage of it. I don't use it often enough to be really good, but it will do things that are clumsy or near impossible in Excel. An old program that I loved was Javelin. It was designed for time-based data, which is just what you need for keeping track of standards over time. It would probably still run in one of the emulators, but I don't even have a drive for the disks and I think its time has passed. I think that's what I used for several years worth of data when I did the Mini-Metrology stuff.

[CatalinaWOW]

I think ChatGPT is much like the internet in general.  A great resource for finding new ideas and information.  But never trusted and the results never used without auditing and reality checks.

[mycroft]

I found NIST's Engineering Statistics Handbook at http://www.itl.nist.gov/div898/handbook/index.htm an interesting resource.

[SilverSolder]

Perhaps a reasonable approach is to do the first cut in Excel, to get an idea of what's going on, and decide on more advanced methods depending on what turns up?

[alm]

I'd use whatever tool you feel most comfortable with. Excel should be totally sufficient. Excel becomes annoying once things become complicated because formulas are hidden across multiple columns and worksheets, and it's also not very convenient if you want to run the same analysis on multiple sets of data unless you want to go into macros.

[jjoonathan]

Yep, the tool you know has a big advantage over the tool you have to learn!

I have some preliminary results. Just the voltage measurements so far; resistances are WIP. If I assume FX is ideal, it looks like the DMM is just outside its 24 hour spec but still within the 90 day spec -- which is just dandy given that it's been 5 years and a disassembly between then and now. I don't see a tolerance band for that on the datasheet 

Something is a bit funky with the PX measurements -- it seems to have 10ppm periodic negative excursions. I'm looking into it, so far I've ruled out autozero and a lack of cable twist but I haven't done a proper job of RF isolation yet. I may be biased, but that's my chief suspect atm, speculation about other possibilities is welcome and appreciated


EDIT: see post below for final results

[Andreas]

Hmm,

looks like EMI.

check WIFI (in one case it was my printer although I never use it)
check mobiles
check USB cables near the setup (in my case it was all 15 seconds from a unused USB RS232 interface plugged in the computer).

https://www.eevblog.com/forum/metrology/lowest-drift-lowest-noise-voltage-reference/msg3738565/#msg3738565

with best regards

Andreas

[jjoonathan]

Yep, sure enough! You can spot the exact moment I closed the shield box, lol.

[SilverSolder]

I ended up with a zen 'Keep It Simple' approach, doing the analysis using Access and Excel to put the different data series together. 

The meters need calibration, but their performance in terms of drift doesn't seem too bad.

[jjoonathan]

Ok! I've taken all the measurements I plan to take, uploaded the data + plots to github, and the kit is on its way back to Randall.

FX	PX	R

I repaired the FX banana connector and added some strain relief. It turns out that the heat-shrink was just the right diameter for a good press-fit with the plastic housing, so it now has that going for it! See attached.

My measurements of the xDevs 10k standard were anomalously low compared to my other resistance measurements (even the other 10ks) and the xdevs measurement. I repeated this on different days and with different 4-wire attachment styles, so I don't think that's the problem. I look forward to seeing what the next person measures 

[SilverSolder]

So, I decided to have a go at using AI to analyze the test data.  I used the "GPT-4 Code Interpreter" model from OpenAI (which is able to create and use Python programs on the fly to help analyze and plot data).  I uploaded a CSV file with the measurements, and asked the AI to walk me through an analysis of the data.

It would be too much to include all the details and plots that were generated, suffice it to say, the results were nothing short of astounding...  Jaw dropping.

Here's a summary of what was done:


SilverSolder (speaking to GPT-4):  Can you summarize the analysis we have done on this dataset so far?


GPT-4:  Below is a summary report of the analysis we performed so far on the dataset containing the measurements from four Digital Multimeters (DMMs).

1. Data Preparation and Filtering

• Action: Removed extraneous rows and outliers, focusing on stable measurements.
  
  
• Interpretation: Ensured clean and consistent data for analysis, allowing for accurate assessments of stability, accuracy, and consistency.
  

2. Allan Deviation Analysis

• Action: Calculated overlapping Allan deviation to assess stability across different time scales.
  
  
• Interpretation: Observed U-shaped curves, indicating white noise at short averaging times, optimal stability at the bottom of the U, and potential drift or bias instability at longer times. This information is crucial for understanding the noise characteristics of the DMMs.
  

3. Drift Analysis

• Action: Calculated the slope of the drift in voltage measurements for each DMM, expressed in ppm/h.
  
  
• Interpretation: Identified slight downward drift in all DMMs, highlighting potential long-term instability or bias that could affect measurement accuracy over extended periods.
  

4. Frequency Domain Analysis

• Action: Performed Fourier Transform to identify frequency components.
  
  
• Interpretation: No dominant frequency components observed, and noise level below 10^{-4}, indicating stable and noise-free measurements. This analysis further affirmed the quality of the DMMs.
  

5. Comparative Statistics

• Action: Calculated mean, standard deviation, variance, etc., for each DMM.
  
  
• Interpretation: DMM1 showed the highest consistency; slight differences in mean values across DMMs. These statistics offered a quick overview of the performance characteristics of each DMM.
  

6. Bland-Altman Plot

• Action: Plotted differences vs. averages to assess agreement between DMM pairs.
  
  
• Interpretation: Provided insights into biases, limits of agreement, and systematic differences between DMMs. This analysis was valuable in understanding how the DMMs compared to one another and identifying any consistent biases.
  

7. Cross-Correlation Analysis

• Action: Calculated cross-correlation to study time-lagged relationships between the DMMs.
  
  
• Interpretation: Peak at zero lag indicated synchronization; falling off and rising again reflected complex relationships, possibly due to noise or cyclic variations. This revealed intricate dynamics and relationships between the signals across different time-lags.
  

Conclusion

The comprehensive analysis provided insights into the stability, accuracy, and consistency of the measurements across the four DMMs. Key findings include stable performance with slight downward drift, high consistency in DMM1, and complex time-lagged relationships between DMM pairs.

The methodologies applied, including Allan deviation, frequency analysis, Bland-Altman plots, and cross-correlation, allowed for a multifaceted exploration of the data. The findings can inform calibration practices, quality control, instrument selection, and further investigations into underlying factors affecting the measurements.

The insights gathered from this analysis can be vital for professionals working with precise measurements, ensuring that the instruments used are reliable and accurate.

[SilverSolder]

Here is an AI analysis of the DMMs' sensitivity to temperature and humidity.  It seems that most of the drift can be explained by these factors, not surprisingly.

Graph and regression analysis done by GPT-4 Code Interpreter:




Summary of regression analysis:
All DMMs exhibit a strong relationship with humidity, with varying degrees of influence.
The relationship with temperature is more varied, with some DMMs showing a significant effect and others not.
The models explain a substantial portion of the variance in voltage readings, especially for DMM2 and DMM3.

[mendip_discovery]

That is interesting as spec sheets rarly show much in the way of issues around RH unless it's at the extremes.

The lab I work in we monitor and record humidity and I try to keep it at a sensible level though it's never been an issue with the assessors.

[maxwell3e10]

It would be interesting to see raw data that you used for ChatGPT analysis.  Correlation analysis is always tricky. The plot you posted above doesn't show humidity variation.

[SilverSolder]

It would be interesting to see raw data that you used for ChatGPT analysis.  Correlation analysis is always tricky. The plot you posted above doesn't show humidity variation.

Sure, attached here!

Of course, it could all be coincidence...   correlation != causation, as we all know.


Here's a couple of example of graphs from the AI:

[maxwell3e10]

It seems humidity change is mostly monotonic, so it looks correlated with any drift in DMM signals.

[Andreas]

Hello,

humidity (%rH) depends also on temperature.

And since humidity changes usually need about 1 week settling time to do component changes (by swelling of the epoxy PCB or component packages) the correlation is clearly temperature instead of humidity.
And there is still drift over time and self heating of the instrument.

with best regards

Andreas

[mendip_discovery]

Hello,

humidity (%rH) depends also on temperature.

And since humidity changes usually need about 1 week settling time to do component changes (by swelling of the epoxy PCB or component packages) the correlation is clearly temperature instead of humidity.
And there is still drift over time and self heating of the instrument.

with best regards

Andreas

My thought is that the kit is getting humidity issues in transport and the powering on for tests is releasing that moisture but with the ambient %RH having an effect on how well the moisture evaporates.

[SilverSolder]

Hello,

humidity (%rH) depends also on temperature.

And since humidity changes usually need about 1 week settling time to do component changes (by swelling of the epoxy PCB or component packages) the correlation is clearly temperature instead of humidity.
And there is still drift over time and self heating of the instrument.

with best regards

Andreas

Why would that not show up in the correlation analysis?

[SilverSolder]

Hello,

humidity (%rH) depends also on temperature.

And since humidity changes usually need about 1 week settling time to do component changes (by swelling of the epoxy PCB or component packages) the correlation is clearly temperature instead of humidity.
And there is still drift over time and self heating of the instrument.

with best regards

Andreas

My thought is that the kit is getting humidity issues in transport and the powering on for tests is releasing that moisture but with the ambient %RH having an effect on how well the moisture evaporates.

The DMMs had also been turned off for several months before this test, it seems reasonable to suppose that they were "letting off steam" as well.

Interesting phenomenon, for sure!

[Andreas]

Why would that not show up in the correlation analysis?

Hello,

You would need to PT1-filter the rH% by a time constant of several days to see any correlation.
The measurement "over night" is simply too short for that.

with best regards

Andreas

[SilverSolder]

Why would that not show up in the correlation analysis?

Hello,

You would need to PT1-filter the rH% by a time constant of several days to see any correlation.
The measurement "over night" is simply too short for that.

with best regards

Andreas

Mathematically, there clearly is a correlation, though?  On four different DMMs measuring at the same time, in parallel?  I'm not sure what you mean...  are you saying the correlation that is visible is just a mathematical artifact and should be discounted?