Pulling my hair out. Circuit boards stop working once shipped to client and more

[Psi]

Also its common for even genuine chips to come in a few different looking versions depending on the factory they were manufactured in.

[OwO]

 
No triaging work has been done so far, and already starting to blame parts.
This is not how you do troubleshooting, you start from the symptoms and work backwards from there. The OP has not even said HOW the devices are malfunctioning.

Example troubleshooting steps:
Device not showing up on WiFi? => Attach the debugger and verify the MCU is running. MCU not running? => Attach debugger again and see where the execution is at, or even reset MCU from the debugger and step through the code. Debugger does not see MCU? Check all supply rails. Unsolder MCU, solder onto an arduino board to check if the MCU is functional.

NOT
Device not showing up on WiFi? => Is my LDO blown? Is my programmer broken? The MCU is counterfeit!

Please if you aren't going to actually troubleshoot it, at least describe the symptoms so we can have a better guess.

[Jackster]

I am not blaming parts or anyone. Just pointing out something that looks odd to me. I am not making any presumptions here.
The text was barley visible when I got the tray.

[mcinque]

No triaging work has been done so far

I think it's correct, debugging is essential.
You must absolutely understand before WHERE the boards failed.
Only after discovering a problem you can solve it.

[Jackster]

Just soldered on a new ATMega328p that I ordered last night.
Does not burn the boot loader.

Thinking the ATMega that I had already had a boot loader or by pure chance it burnt on that board.

[Psi]

Have you checked the fuse bits to confirm settings are correct?
- Clock source (often there are different settings for <8Mhz and >8Mhz xtal.  If you use the <8Mhz setting for 16mhz clock it works just not reliably.
- Startup delay (set max delay if unsure)
- Brownout setting.

[mcinque]

I have the boards that have failed from when clients used them. More than half worked as expected. Some did fail testing.

Sho you had perfectly working boards and now some of them aren't working.
So why don't you trace what's failed in that boards? It's the MCU? It's one of the power rails? The crystal? What else? That is essential! Knowing that would be a huge help.

Please, do it and report it here.

[Jackster]

I have the boards that have failed from when clients used them. More than half worked as expected. Some did fail testing.

Sho you had perfectly working boards and now some of them aren't working.
So why don't you trace what's failed in that boards? It's the MCU? It's one of the power rails? The crystal? What else? That is essential! Knowing that would be a huge help.

Please, do it and report it here.

I am not sure what was wrong with them other than the firmware stops running as it should be.
Power is fine, sensor input is fine.

The only thing it could be is the MCU or WiFi board.

[mikeselectricstuff]

Only read the first post and skimmed the rest but my guess would be oscillator startup issues - assuming you use a crystal or ceramic resonator. This would be consistent with inability to program and some units working and some not.
There can be a number of causes, bad PCB layout being one, in particular capacitance across the device, as well as non-optimal load capacitance.
A quick fix can often be to add some resistance across the resonator ( 1-10M), or if possible select a different oscillator power - I don't recall the AVR options offhand but many MCUs have drive-level options to trade off power draw vs startup time.

If you have a dead board in front of you, poke the oscillator pins and see if it starts.

[mcinque]

I am not sure what was wrong with them other than the firmware stops running as it should be.
Power is fine, sensor input is fine.
The only thing it could be is the MCU or WiFi board.

As I (and the wise Mike) said: check the crystal/oscillator. Put an oscilloscope to see if it's working. Keep in mind correct probe range/capacitance while probing.
Also, "power is fine": did you check it with a DMM or with an oscilloscope?

[mikeselectricstuff]

I am not sure what was wrong with them other than the firmware stops running as it should be.
Power is fine, sensor input is fine.
The only thing it could be is the MCU or WiFi board.

As I (and the wise Mike) said: check the crystal/oscillator. Put an oscilloscope to see if it's working. Keep in mind correct probe range/capacitance while probing.
Also, "power is fine": did you check it with a DMM or with an oscilloscope?

Don't forget that the scope probe itself can start it oscillating. If possible use a x100 probe. Or better look at an output pin that is being toggled by software after startup

[Jackster]

I don't have an oscilloscope. Just multi meter.

[mcinque]

I bet that with an oscilloscope you could find the issue in minutes. Since you're doing circuits for work, consider purchasing one.

[Jackster]

I bet that with an oscilloscope you could find the issue in minutes. Since you're doing circuits for work, consider purchasing one.

Will a £50 USB one be enough?
https://www.amazon.co.uk/HANTEK-Portable-Based-Digital-Oscilloscope/dp/B00EEX1W5G/ref=sr_1_28?keywords=oscilloscope&qid=1559823810&s=gateway&sr=8-28

[mikeselectricstuff]

I don't have an oscilloscope. Just multi meter.

Buy one.
Without it you are pissing away your time.

[mcinque]

Will a £50 USB one be enough?

An USB like that has a very very low bandwidth; you need at least double bandwitdh of your clock.
However a proper real DSO with proper knobs on the chassis is much more confortable to use.

Like Mike said, without one you're really wasting your time.

[tggzzz]

I don't have an oscilloscope. Just multi meter.

That was the equipment I used as an amateur when I was still at school. It got me a long way, but I knew when it wasn't sufficient and used a scope where necessary.

Quite frankly I'm surprised you do still have paying clients.

[tggzzz]

you need at least double bandwitdh of your clock.

False. All that matters is the transition time; the period/frequency is irrelevant.

For an outline of the theory and some measurements, see https://entertaininghacks.wordpress.com/2018/05/08/digital-signal-integrity-and-bandwidth-signals-risetime-is-important-period-is-irrelevant/

[tszaboo]

You dont have power supply bypassing. Whatever you put there is not enough, too far and so on. You dont even have a ceramic cap next to that FTDI chip. I'm not surprised it is not working.

[Jackster]

So something like the Hantek DSO5102P?

Even so, would it even be worth it if I need to hire someone to remake the circuit anyway...
And it would take a while to learn how to use the thing..

[daqq]

One of the issues we've had were crystal load capacitors - in the prototype everything worked. After a batch was made, quite a lot didn't work. This was because the load capacitance was above the one specified by the processor (STM32L1). The oscillator simply wouldn't start. We had to replace all of the capacitors.

[Jackster]

So I managed to get some time in the office today and got one of the bad boards and started removing components and testing.

After a few, I removed the FTDI and it burned the boot loader.
Put all the other components back other than the FTDI and again it burned.

I then put the same FTDI chip back and it did not burn. Removed it and it burned.
I then got a known working FTDI chip from RS and it burnt the boot loader.

I need to double check all of this in the morning so this is more of a note for myself.
The LCSC ones are dated year 17 and the RS ones are dates 18.

Will take a working board and remove the working FTDI chip and replace with one of the new FTDI chips and see if it burns the boot loader in the morning.

[mcinque]

If I may, I will invest time in understanding WHAT is failed instead trying to fix something that you don't know exactly.

HantekDSO5102P is fine, you could choose also a Siglent SDS1052DL or a used classic Rigol DS1052E.
I know that you should learn how to use it properly but it's something mandatory and not so difficoult (if I can use it , anyone else can do). You can't live without it.

Once you have an oscilloscope, you can't (and you wouldn't) get back!

Imagine this: without that instrument you are blind.

[Jackster]

If I may, I will invest time in understanding WHAT is failed instead trying to fix something that you don't know exactly.

HantekDSO5102P is fine, you could choose also a Siglent SDS1052DL or a used classic Rigol DS1052E.
I know that you should learn how to use it properly but it's something mandatory and not so difficoult (if I can use it , anyone else can do). You can't live without it.

Once you have an oscilloscope, you can't (and you wouldn't) get back!

Imagine this: without that instrument you are blind.

IDK what I am looking for though with it.

[Ian.M]

So I managed to get some time in the office today and got one of the bad boards and started removing components and testing.

After a few, I removed the FTDI and it burned the boot loader.
Put all the other components back other than the FTDI and again it burned.

I then put the same FTDI chip back and it did not burn. Removed it and it burned.
I then got a known working FTDI chip from RS and it burnt the boot loader.

I need to double check all of this in the morning so this is more of a note for myself.
The LCSC ones are dated year 17 and the RS ones are dates 18.

Will take a working board and remove the working FTDI chip and replace with one of the new FTDI chips and see if it burns the boot loader in the morning.

Possible fake FTDI chips?  Hack a bad board to loop the FDTI's RX and TX pins, open the USB serial port with a terminal program and see if it responds

Code: [Select]

NON GENUINE DEVICE FOUND!character by character as you type instead of the loopback echoing what you type.  (see: https://www.eevblog.com/forum/microcontrollers/ftdi-gate-2-0/ )