AI parsing data sheets

[AndyC_772]

Has anyone had any success getting any real help from AI to read, parse, and answer questions about parts from their data sheets?

Given how relatively proficient ChatGPT seems to be in coding, I had reasonably high expectations that it might also be able to answer questions about components. In this instance, I found two ADCs from competing manufacturers, which have completely different part numbers but actually look as though one may have been designed as a pin compatible alternative to the other. They both come in the same package, have similar headline specs, and although many pins have different names (eg. "VCC" vs "AVDD"), their functions seem similar. Certainly there are too many similarities to be a coincidence.

This seemed like a good opportunity to get some help from AI. Ask it what the functional differences are, and whether they are in fact likely to be compatible.

Of course I can't trust the answers without checking, but if it notices that (say) one has a particular feature that the other lacks, that at least is something I can quickly look for in both data sheets. I often seem to find myself trying to compare and contrast similar parts, so any help in spotting differences is welcome.

To say I was unimpressed would be an understatement. ChatGPT (-o1 preview) failed to correctly identify the number of inputs or even the number of ADC cores in each part. Features clearly common to both were missed in one device, and highlighted as a point of difference. Each time I said "no, try again" it would apologise, correct its mistake, then make a bunch of new mistakes next time around. Stupid, obvious errors, like "pin X is called NNNN", where it just isn't.

In the end it was completely useless, and I gave up - but I do think this is an area where AI might one day help and save time.

Has anyone found an AI tool that does a better job of "understanding" data sheets and answering questions about them?

[tom66]

A while ago I attempted to use it to find pin-compatible alternative voltage regulators, but it just barfed some made up part numbers at me.  Unfortunately, at the edge of its training data, it's quite likely a lot of the output is garbage.  I can imagine an LLM would be quite good at this if trained specifically on electronics engineering datasets.

[kripton2035]

this one : https://docs.flux.ai/
claims to have been trained to do such a job of reading datasheets, but I've never been able to make it work correctly.

[tggzzz]

Has anyone had any success getting any real help from AI to read, parse, and answer questions about parts from their data sheets?

Realise that LLMs are a mechanism for producing grammatically correct sentences of plausible words. They have zero understanding of the words an concepts.

There's a name for humans that manage to do that: bullshitters.

Is is any surprise that managers think LLMs work well?

[SiliconWizard]

But the marketing is apparently really good.

[amyk]

Given that https://www.eevblog.com/forum/chatgptai/ai-generated-lies-in-datasheet-search-results/ has been all I've seen AI do with datasheets, I'm not surprised.

[Berni]

this one : https://docs.flux.ai/
claims to have been trained to do such a job of reading datasheets, but I've never been able to make it work correctly.

They are likely using OpenAIs GPT models via the API or something similar.

Just blindly asking these general purpose LLM AIs about very specific things like chip partnumbers or specs is a bad idea because when they talk about a more niche topic they hallucinate like there is no tomorrow, making up stuff as they go, rather than saying that they don't know. To make things worse it puts the made up information into nice confident sentences that makes it look like it knows what it is talking about.

Feeding it a PDF of a datasheet likely works fairly well since todays LLMs have enough context length to hold even a fairly sizable PDF in memory. Tho most of the time Ctrl+F will get your answer sooner in datasheets, while you also have to understand that the AI doesn't read a PDF like a human, so depending on the internal PDF structure it might not be able to see some diagrams or see them wrong.

[ebastler]

Realise that LLMs are a mechanism for producing grammatically correct sentences of plausible words. They have zero understanding of the words an concepts.

That's a description on the same level as "Computers are really simple and really dumb. All they do is manipulate 0s and 1s." It's true on some level, but it leads you to totally underestimate the capabilities that can be obtained by piling on a few layers of abstraction and complexity.

(That computer quote was actually pretty commonly heard when computers became more visible in the 1970s. Typically used by people who were slightly nervous about them and had a very limited idea what they were and how they worked.)

[tggzzz]

Realise that LLMs are a mechanism for producing grammatically correct sentences of plausible words. They have zero understanding of the words an concepts.

That's a description on the same level as "Computers are really simple and really dumb. All they do is manipulate 0s and 1s." It's true on some level, but it leads you to totally underestimate the capabilities that can be obtained by piling on a few layers of abstraction and complexity.

(That computer quote was actually pretty commonly heard when computers became more visible in the 1970s. Typically used by people who were slightly nervous about them and had a very limited idea what they were and how they worked.)

With computers you could always work out why they produced the result they did[1], and the limits within the results were valid. With LLMs that is, ahem, "an active research topic" with no resolution in sight.

If you read comp.risks (which everybody should), real world examples abound, from sentencing/custody decisions in the US, to things which appear to "work" for spurious "reasons" https://catless.ncl.ac.uk/Risks/32/80/#subj4.1
  "Some AIs were found to be picking up on the *text font* that certain
  hospitals used to label the scans. As a result, fonts from hospitals with
  more serious caseloads became predictors of covid risk."

That's a recent version of an old old problem. Igor Aleksander's 1983 WISARD, effectively the forerunner of today's LLMs, demonstrated a key property of modern LLMs: you didn't and couldn't predict/understand the result it would produce.  WISARD correctly distinguished between cars and tanks in the lab, but failed dismally when taken to Luneberger Heath in north Germany. Eventually they worked out the training set was tanks under grey skys and car adverts under sunny skies.

As for the "edge of the envelope" problem, consider there are documented examples where changing one pixel in a photo of a road "stop" sign, caused the ML system to change to interpreting it as a "40MPH" sign. The corollary is that if a bugfix is issued for that, there is no way of knowing whether it will cause a "slow" sign to be misinterpreted as a "50 mph".


[1] that's less true now because they have piled on so many layers of abstraction and complexity that not many people comprehend them all. Look at all the synchronous and asynchronous protocols layered on top of each other in a typical enterprise/telecom system. Too many practitioners think FSM== parsers, and believe the salesmen's (implicit) claims that their software avoids the Byzantine Generals problem. Except that either they don't know that problem or have forgotten it and don't relate it to their distributed systems.

[ebastler]

With computers you could always work out why they produced the result they did[1], and the limits within the results were valid. With LLMs that is, ahem, "an active research topic" with no resolution in sight.

That is a very different concern from your prior over-simplification that "LLMs just produce grammatically correct sentences with plausible words." Which is all I had commented on.

If you read comp.risks (which everybody should), real world examples abound, from sentencing/custody decisions in the US, to things which appear to "work" for spurious "reasons" https://catless.ncl.ac.uk/Risks/32/80/#subj4.1
  "Some AIs were found to be picking up on the *text font* that certain
  hospitals used to label the scans. As a result, fonts from hospitals with
  more serious caseloads became predictors of covid risk."

That particular anecdote is a quote from a "quote" from a Technology Review article. The Driggs paper which Technology Review refers to does not seem to deal with LLMs at all, and it certainly does not contain the word "font". Which confirms my bias that Technology Review is just a collection of grammatically correct sentences with plausible words... 

[AndyC_772]

Realise that LLMs are a mechanism for producing grammatically correct sentences of plausible words. They have zero understanding of the words an concepts.

I'm going to go out on a limb here and suggest you've not spent any time with a recent paid-for ChatGPT interface.

If you get the chance - and I strongly recommend subscribing for a month just so you can - ask it to explain a technical topic with which you're already somewhat familiar. Ask it to write some code in your preferred language to carry out a useful function, or upload code of your own and ask it to check for errors. I think you'll very soon realise it's far more capable than just a package that strings together plausible sounding words on a statistically likely basis.

The latest -o1 preview build is specifically crafted to follow chains of logical reasoning. It even describes each step as it works toward composing a response.

[tom66]

That's a recent version of an old old problem. Igor Aleksander's 1983 WISARD, effectively the forerunner of today's LLMs, demonstrated a key property of modern LLMs: you didn't and couldn't predict/understand the result it would produce.  WISARD correctly distinguished between cars and tanks in the lab, but failed dismally when taken to Luneberger Heath in north Germany. Eventually they worked out the training set was tanks under grey skys and car adverts under sunny skies.

"AI was a bit shit in the 80s, so of course in 2024 it will still be shit."

Yes, there are problems with LLMs & NN image processing, but that does not mean that they are useless.  For your tank example it's a well known problem in model training where the model is overconstrained and the training data is too small.  ChatGPT has been trained on some 40 terabytes of input data with man years of human filtering applied afterwards.

And, anecdotally, I've found that over the past two years it's become increasingly difficult to get ChatGPT to output bullshit.  It either says "I don't know the answer" or it answers. 

[Berni]

Even the GPT4o you get from a free ChatGPT account is impressively smart.

Yes deep down these kind of AIs are indeed glorified statistics. The actual technology here behind this AI is in figuring out a computationally more efficient way to calculate these statistics since there is no way you can store the probability of a word for all possible combinations of 10000+ words. So a giant neural network is trained to output something as close as possible to the real word probability.

The intelligence part of LLMs is an emergent behavior (like the patterns in flocks of birds) where the simple actions come together to express a complex behavior. The neural net inside the LLM learns to encode various high level concepts inside the network because that is the only way it can squeeze them into the network, there are not enough neurons to store it brute force as a giant 'lookup table'.

You can make the same point with a biological brain. It is just a pile of neurons that fire off signals depending on their inputs, there is no real intelligence inside a neuron. not even in groups of neurons. But when you put a huge number of them together and teach them on how to react to certain stimulus, the whole thing starts acting as something that appears to be intelligent. If the pile of biological neurons is trained badly it also performs badly.

And yes an AI like this is only as good as its training data. For common things where they could scrape a lot of training data from the internet,books,etc.., there the AI performs remarkably well. For things where it did not have enough training data it will perform poorly. You need a LOT of data for a good training dataset. However even more importantly you need high quality training data, so that there are enough properly varied examples to isolate the concept of interest and there are no large biases. There also has to be intentional examples of garbage in the training data so that it can learn to ignore the garbage.

Fact of the matter is that LLM AIs have enabled computers to do tasks they could never do before. If you train an AI for a task properly and only use it for that task, they perform great. The problem is that this new AI stuff has picked up a massive amount of hype around it, so now everyone is forcefully shoehorning the new fangled AI tech into absolutely everything, regardless if it makes sense to do so.

[daqq]

I tried it on a chinese datasheet for a LED driver - I was actually pleasantly surprised by the results. Wouldn't be very useful for an MCU probably, but it worked great for the SM16306SJ:

https://chatgpt.com/share/66e9b300-d3e0-8002-bf35-3a4c456b2a20

[tggzzz]

That's a recent version of an old old problem. Igor Aleksander's 1983 WISARD, effectively the forerunner of today's LLMs, demonstrated a key property of modern LLMs: you didn't and couldn't predict/understand the result it would produce.  WISARD correctly distinguished between cars and tanks in the lab, but failed dismally when taken to Luneberger Heath in north Germany. Eventually they worked out the training set was tanks under grey skys and car adverts under sunny skies.

"AI was a bit shit in the 80s, so of course in 2024 it will still be shit."

Er, no.

AI in the 1980s was a set of rules, based on knowledge elicited from experts. It was then easy to see which rules caused the results, and why.

WISARD was not like that; it was unique.

In the 2020s ML is shit for different reasons, as foreshadowed by WISARD. In particular it is impossible to know the chain that caused the results, because there are no explicit rules.

Yes, there are problems with LLMs & NN image processing, but that does not mean that they are useless.  For your tank example it's a well known problem in model training where the model is overconstrained and the training data is too small.  ChatGPT has been trained on some 40 terabytes of input data with man years of human filtering applied afterwards.

That merely makes the problems more subtle.

Never forget Tony Hoare's wise words from his 1980 Turing Award Lecture[1]; Communications of the ACM 24 (2), (February 1981): pp. 75-83.
There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

And, anecdotally, I've found that over the past two years it's become increasingly difficult to get ChatGPT to output bullshit.  It either says "I don't know the answer" or it answers.

Are you one of those people that believes a piece of software is works because it passes the unit tests?

[tggzzz]

Even the GPT4o you get from a free ChatGPT account is impressively smart.

Yes deep down these kind of AIs are indeed glorified statistics. The actual technology here behind this AI is in figuring out a computationally more efficient way to calculate these statistics since there is no way you can store the probability of a word for all possible combinations of 10000+ words. So a giant neural network is trained to output something as close as possible to the real word probability.

The intelligence part of LLMs is an emergent behavior (like the patterns in flocks of birds) where the simple actions come together to express a complex behavior. The neural net inside the LLM learns to encode various high level concepts inside the network because that is the only way it can squeeze them into the network, there are not enough neurons to store it brute force as a giant 'lookup table'.

You can make the same point with a biological brain. It is just a pile of neurons that fire off signals depending on their inputs, there is no real intelligence inside a neuron. not even in groups of neurons. But when you put a huge number of them together and teach them on how to react to certain stimulus, the whole thing starts acting as something that appears to be intelligent. If the pile of biological neurons is trained badly it also performs badly.

And yes an AI like this is only as good as its training data. For common things where they could scrape a lot of training data from the internet,books,etc.., there the AI performs remarkably well. For things where it did not have enough training data it will perform poorly. You need a LOT of data for a good training dataset. However even more importantly you need high quality training data, so that there are enough properly varied examples to isolate the concept of interest and there are no large biases. There also has to be intentional examples of garbage in the training data so that it can learn to ignore the garbage.

Fact of the matter is that LLM AIs have enabled computers to do tasks they could never do before. If you train an AI for a task properly and only use it for that task, they perform great. The problem is that this new AI stuff has picked up a massive amount of hype around it, so now everyone is forcefully shoehorning the new fangled AI tech into absolutely everything, regardless if it makes sense to do so.

The point about the similarity between wetware and LLMs is undeniable. After all, wetware was the inspiration for LLMs, starting with the earliest "perceptron" experiments in the 50s and 60s.

The output of brains and LLMs is indeed a consequence of "emergent behaviour". That makes it impossible to determine what the response will be and why. As an engineer (not a psychologist, not a politician, etc) I regard that as a key failing.

Bad training data is a key problem. People are already worrying about what happens when a lot of the content on the web is semi-accurate LLM output, and that is used as training data for the next round of LLMs. Experiments have demonstrated that is a realistic problem.

Saying "it works if you have good training data" is merely opium that I would expect salesmen to spout. Similarly, fracking salesmen say that "fracking is safe when done properly", which is obviously a meaningless truism, and ignores the probability of it not being done properly.

Some training data has, by definition, to be polluted. That doesn't stop users from claiming "the machine says so therefore it is right" (note the present tense). Classic example is the systems in the US that determine the risk associated by an accused person or a convicted felon and therefore whether they should be incarcerated before/after trial. Since black people have a higher probability of being incarcerated, the LLMs can only perpetuate that injustice.

[SiliconWizard]

In other words, throwing stuff at the wall until it sticks, and calling that engineering.