A simple test would be to hook up a single LED with an appropriate current limiting resistor directly to the PCA9634 exactly as shown in “10. Application design-in information”, for LED0 - from the data sheet that you linked. This eliminates the mosfet and all of the additional drive circuitry. My guess is that you will see a very different brightness function.
There is some truth that the mosfet may distort the PWM waveform in some manner. Looking at the circuit, you can see that there is a gate resistor present before the mosfet. I was told to put this 100 ohm resistor there to prevent large current spikes on the mosfet driver. But thinking about it, the mosfet has some internal gate capacitance, and together with the resistor they both form an RC filter on the line. I've already checked the scope and the waveform looks quite different (much narrower and spiky) on the input than the one behind the resistor on the mosfet gate. I tried removing the gate resistor and feed the waveform from the function generator directly into the mosfet gate, and the only change this seems to have caused is more ringing on the gate terminal. LED brightness did not seem to change noticeably. I'll try to do a proper test with the output of the driver connected directly to an LED and a current limiting resistor.
Then if you look at the gamma correction function it should be clear why this isn't working.
No, you misunderstand. The gamma correction algorithm is present on the microcontroller, but I was doing the tests with a separate function generator that outputs a square waveform of given duty cycle with no gamma correction. The problem isn't gamma correction, but the fact that the change of brightness on the final step from 1/255 to 0/255 duty cycle is too big while the changes from 1/255 to 2/255, 2/255 to 3/255, 3/255 to 4/255, etc are barely noticeable. See
tooki's reply one post above yours. All 8 channels are dedicated to different and separate LED's.
Here is a 16 channel 12 bit controller that can operate at 1.5 kHz also from NXP: PCA9685 https://www.nxp.com/products/power-management/lighting-driver-and-controller-ics/ic-led-controllers/16-channel-12-bit-pwm-fm-plus-ic-bus-led-controller:PCA9685 This was just the first one I found, I'm sure there are others.
Yeah, I just found it myself. It's a 12-bit, but 16-channel PWM driver that apparently even has an external clock input pin. They don't have an 8 channel one though. In worst case I'll have to discard 8 channels, or as
StillTrying suggested, maybe use 8 channels for high current and the other 8 for low current? I'll have to study this possibility.
Thanks for all the suggestions so far, everyone. I wasn't expecting my thread to get this much attention.
I'm getting new helpful posts faster than I can process them. That sigma-delta modulation approach does look interesting. Although I'm unsure I'll be able to implement it on an existing microcontroller since it already has to process quite a lot of code.