First one thing I've noticed: With new regulator, both regulator and coil run much cooler than before. Previously I couldn't hold my finger next to them because of heat. Now they're just very hot, but not burning hot.
I did some scoping on the timing capacitor itself and its waveform is very noisy and jittery. According to the scope's frequency measurement, the frequency is always changing. It's usually around 48 kHz, but can go as high as 60 kHz sometimes. I'm attaching image with infinite persistence. Another image which I found interesting is when I probes the pin 2 and pin 3 at the same time. It seems that there are huge bursts of noise on pin 3 when pin 2 is high.
Should I try changing the capacitor? It seems like a polyester capacitor to me, but from what I heard they should be pretty good.
I can easily obtain ceramic multilayer capacitors of same capacitance and tolerance with NP0 dielectric and try with them, to see if it's the capacitor.
I looked at these waveforms some more to try to come up with some kind of plausible scenario that explains what I see.
I think that the bursts when pin 2 is high are just the result of common mode noise interaction between the two probes, I don't know if you had each probe individually grounded, but if you didn't, that could have enhanced this interaction. This assessment is partly based on the fact that the first capture doesn't show these noise bursts. In addition, the same noise burst can be seen on the pin 2 waveform on a scale matching the V/div setting for that channel.
The one thing I find troubling is the irregularity of the duty cycle, and I'm just going to write what I'm thinking as I try to analyze why.
The large spike at the beginning of the capacitor's discharge cycle coincides with pin 2 going low, but it's not always the same size. It's size seems to depend on the length of time pin 2 remains high. I believe this spike represents D12 turning on due to L2's collapsing field and as a result charging EC8 to correct the output voltage. But why isn't it consistent.
I'm going to leave that aside for a second and look at what seems right. Pin 2 is supposed to go high during the charge time of the capacitor and nowhere else. It goes high when the output voltage has dropped a little as sensed by the IC's pin 5. This seems to be happening as it should, at some point during the charge cycle of C6 the output voltage drops a little bit and Bam, pin 2 goes high. In addition pin2 is supposed to go low at the beginning of C6's discharge cycle, and nowhere else, and it does that correctly too.
Back to what doesn't seem right. When pin 2 goes high late during C6's charge cycle, it goes off where it should (at the beginning of the discharge cycle), but the spike marking D12's turn on is not there, and pin 2 comes back on early during the next charge cycle. Why?
The only explanation I can think of is that EC8 didn't get enough charge and a voltage drop was detected by pin 5 again. Why?
A couple things I can think of, L2's collapsing field wasn't enough to forward bias D12 and charge EC8. Alternatively, D12's forward voltage is abnormal or inconsistent and fails to conduct and charge EC8 properly sometimes.