I2C slave device does not respond with an ACK

[tilblackout]

Hello everyone, I am currently debugging an I2C device. There are two devices on the I2C bus: one is an ADXL367 MEMS accelerometer, and I am able to read and write its registers normally using the I2C driver I wrote.

At the same time, there is also a battery gauge on this I2C bus. The software driver is similar, only the device address and registers are different. I want to obtain the battery level, but the I2C communication only succeeds about 20%-30% of the time. By observing the waveform, it can be seen that the issue is caused by the device not responding with an ACK.



What could be the possible cause of this issue?

[mikerj]

There appears to be a lot of glitches on the data line, is this cross talk from the clock?

[tilblackout]

There appears to be a lot of glitches on the data line, is this cross talk from the clock?

Does this have a significant impact on the I2C communication?
I've tried slowing down the I2C clock, and the issue is still the same.

[selcuk]

They are independent of the I2C clock speed. I2C clock may disturb I2C data line when they are routed paralel with long traces on board. Or using wires and twisting clock and data together may cause such issues as well. If the glitches are caused by this, the rise and fall times of the clock plays a more important role than the clock speed. So changing clock speed doesn't help.

https://www.eevblog.com/forum/beginners/why-do-these-modules-refuse-to-work-together/msg5414684/#msg5414684

[RoGeorge]

Battery management chips usually use the SMB protocol, which is not identical with I2C.  There are some exceptions about the ACK from the SMB (e.g. delayed ACK), and those are specified in the protocol.  Search for SMB protocol specifications pdf, those are open and free.

Usually the I2C interface from casual microcontrollers can be used to implement SMB, but sometimes it is not possible.  For example, if you look at various Arduino SMB battery readers, you'll notice they are all using a bitbang I2C library instead of the existing hardware I2C drivers.  Vaguely remember something about delayed ACK (for SMB only), which can not be done using the I2C hardware, but I'm not very sure if this is the reason in your case.  Might be.

Another thing might be that some proprietary battery management chips do not implement the SMB protocol 100% as in the specifications (particularly when you try to talk with batteries dedicated for other closed source/proprietarydevices, like for example a HP laptop battery, or a Dyson pack for power tools, etc.).  Some of these proprietary battery controllers will expect passwords, or proprietary sequence of commands in order to work reliably.

[tooki]

The fact that it succeeds some of the time makes me think that a protocol incompatibility is not the issue. I agree with mikerj and selcuk that the waveform looks quite bad.

The numerous negative-going spikes are potential sources of problems, but it’s also odd that the voltage when pulled low is inconsistent.

OP, please share the part number of all devices on the bus. Also share how they are connected, including PCB layout. And also what the supply voltage is, and what values of pull-up resistors are used (and where they are).

[mtwieg]

It's incredibly difficult to tell what's going on from these photos. Could you capture SCL and SDA simultaneously on two channels, or at least export the waveforms using the scope's software so the waveforms have consistent scaling?

The waveforms don't look terrible, but it's very difficult to really tell because I can't align the SCL/SDA traces.

Battery management chips usually use the SMB protocol, which is not identical with I2C.  There are some exceptions about the ACK from the SMB (e.g. delayed ACK), and those are specified in the protocol.  Search for SMB protocol specifications pdf, those are open and free.

I've never heard of "delayed ACK" wrt I2C/SMBus/PMBus. I just searched the latest SMBus spec (3.2) and couldn't find anything.

The numerous negative-going spikes are potential sources of problems

How so?

but it’s also odd that the voltage when pulled low is inconsistent.

It's not odd at all to see two different low levels on each line during a transaction. This is due to the master and slave having different pull-down strength, and it's actually convenient because it provides a way do tell who is actually in control of the signals. If there are three different levels, that may indicate both are pulling low at the same time (may happen on SCL during clock stretching).

For example, looking at SDA the we can see the first byte is 0xAA (slave address 0x55 followed by a write bit). The following SDA bit is low, but at a slightly higher voltage than the preceding low bits, which makes sense because that's where we expect the slave to take over SDA to issue an ACK.

The next low pulse on SCL is noteworthy, as its voltage is different from preceding low SCL pulses. In fact it transitions between two different voltages, both of which are different from the preceding pulses. What's likely happening is both master and slave are pulling SCL down, and then the master lets go, and then the slave lets go a bit later, allowing SCL to go high again.

Also the next SDA high pulse is very brief. Impossible to tell from these waveforms if it would be ignored, or interpreted as a repeated start condition.

In the following frame, the master sends 0x2C. The slave then appears to stretch the clock again, but for a longer time than during the address frame. During this time, SDA idles high, and then the ACK comes from the slave (again, with a different low level on SDA).

At this point my eyes are getting very tired of squinting at these waveforms... I'm betting your issue is due to your own MCU and/or firmware not handling clock stretching correctly. Clock stretching is often not repeatable by nature, which may explain why your results are not repeatable.

In order to really help I really will need more clear waveforms, with SCL and SDA properly aligned.

[RoGeorge]

Battery management chips usually use the SMB protocol, which is not identical with I2C.  There are some exceptions about the ACK from the SMB (e.g. delayed ACK), and those are specified in the protocol.  Search for SMB protocol specifications pdf, those are open and free.

I've never heard of "delayed ACK" wrt I2C/SMBus/PMBus. I just searched the latest SMBus spec (3.2) and couldn't find anything.

If that's not the proper term, then it's my wrong wording, sorry. 

Don't recall the exact details, so I've wrote "vaguely recall" in my previous post, but I've bumped into similar behavior with the one described by the OP, while I was trying to reverse engineer a backup battery for HP servers some months ago.  The issue was about the differences between I2C and SMBus that are hinted by this this phrase:

In addition, there are some protocol differences with regard to use of Acknowledge and No Acknowledge conditions that are rarely encountered.

Quote from https://www.analog.com/en/resources/design-notes/guide-to-comparing-ic-bus-to-the-smbus.html

In my case there were some delays from the slave combined with clock stretching, and the I2C hardware (master) was taking that as an error. 

After trying to read battery packs from a few more brands, my conclusion was that the state machine in the battery management chips can behave (apparently) unpredictable.  They were designed and tested to work properly only when interrogated with proprietary software/drivers and with proprietary sequence of SMBus commands.  When proprietary batteries are interrogate with generic SMBus software it's not always straightforward, talking about situations like these (not related with the OP questions):  https://www.karosium.com/2016/08/hacking-bq8030-with-sanyo-firmware.html

The next link might be related with the OP issues, tells in a couple of pages what to look for, including when mixing I2C with SMBus:  https://ez.analog.com/cfs-file/__key/communityserver-discussions-components-files/330/SMBusRulesForSuccessV14_5B00_1_5D00_.pdf

[Peabody]

If both slave devices, and even the MCU, have pullup resistors, perhaps the effective parallel resistance is too low, requiring too much sink current to bring the line low reliably.

[globoy]

I'm not sure if this is a result of the way you are probing the signals but I see some glitches in at least the SDA signal.  I've had problems with an audio codec (ES8388) where it would get confused by glitches on the signals.  The solution is to filter the signals going to the part.  Here's what I did to make a reliable interface (shamelessly copied from other designs using the part).

[Postal2]

.... There are some exceptions about the ACK from the SMB (e.g. delayed ACK), ....
....  Vaguely remember something about delayed ACK (for SMB only), ....

Samba (SMB) TCP delayed ack

But

System Management Bus
4.3.3.    Clock low extending

[glenenglish]

yeah dont worry about the glitches too much, you could try a couple a 22pF capacitors across the bus to see if its happier, in fact most mfrs recommend this .

So, the  I2C devices are generally fancy state machines and you can get them upset if you treat them wrong.

Atter the device fails to respond, try clocking through, with SDA high, maybe 64 SCLK pulses to throughly reset the state machine.

I saw this with quite a few maxim devices, and I now do this for all I2C peripherals on start up.

Yeah, SMB is a bit different.

[tilblackout]

The MCU chip I’m using, the I.MX RT1052, has a glitch filtering feature, so there's no need to add capacitors on the hardware side. (I think) Those glitches have already been filtered out.

The power chip indeed uses the SMB protocol, so there are some differences compared to the I2C protocol. I adjusted it to the minimum supported frequency of the power chip, which is 10kHz, and both modules can now communicate. I had tried 20kHz before, but it didn't seem to work. For now, this is how I resolved the issue.

Thank you all for your help.

[mtwieg]

The MCU chip I’m using, the I.MX RT1052, has a glitch filtering feature, so there's no need to add capacitors on the hardware side. Those glitches have already been filtered out.

This won't remove glitches on the bus, it will just make the master less affected by glitches (if the feature is configured properly). Slaves may still be affected though.

The power chip indeed uses the SMB protocol, so there are some differences compared to the I2C protocol. I adjusted it to the minimum supported frequency of the power chip, which is 10kHz, and both modules can now communicate. I had tried 20kHz before, but it didn't seem to work. For now, this is how I resolved the issue.

I guess increasing the clock period may help as long as the slaves' delays are much less than a clock period. But I wouldn't expect this to be a robust solution, as it's usually impossible to know what the worst-case clock-stretch will be for a given slave (per the protocol spec, slaves are allowed to stretch the clock up to 35ms).

[tilblackout]

Thank you, I agree with what you said. I should first address the hardware glitches. The battery manufacturer mentioned that the chip’s ACK might be a bit slow. I’m not sure how to extend the ACK time when using hardware I2C. As for using software I2C, I don’t think it’s a good option since it takes up a lot of CPU usage.

[glenenglish]

No need for software I2c to take up much CPU usage- just have an interrupt trigger at 2x (min)  the desired clock rate... and have inputs edge  interrupt triggered..

[mtwieg]

Thank you, I agree with what you said. I should first address the hardware glitches.

To be clear, I'm not saying you actually have a glitch issue. Hard to say with confidence without better waveforms.

The battery manufacturer mentioned that the chip’s ACK might be a bit slow. I’m not sure how to extend the ACK time when using hardware I2C. As for using software I2C, I don’t think it’s a good option since it takes up a lot of CPU usage.

I don't think anyone was suggesting you bit-bang the I2C protocol. The LPI2C peripheral on your MCU should handle clock stretching gracefully. But whether it works will also depend on your code (I2C is never entirely hands-free, unfortunately).

Again, I can't tell for certain whether what you're seeing from the slave is "normal" I2C behavior (clock stretching). Capturing SCL and SDA simultaneously would make it much easier.