Hi dzseki,
Just thinking out loud really.
You mention that the heatsink is not really adequate so considering that area:
Are you certain that you can't beef up the heatsink one way or another?
Look for an NMOSFET with a lower thermal resistance junction to case, as you mention.
Use the best new thermal grease.
Investigate thermal insulating washers with a low thermal resistance- I think aluminum oxide is the lowest, but ceramic is also pretty good.
In the same vein, if the transistors are T0220, consider a case clamp to lower thermal resistance to the heatsink or change to TO3.
You could consider making the heatsink live and thus eliminating the insulating washer completely. This should reduce the thermal resistance to the heatsink a fair amount, but don't know how practical this would be. You would have to be cautious about the extra capacitance and inductance this would introduce to the drain circuit too.
Fit a fan- this is not as radical as it sounds- you can use the small CPU type fans which are compact, but effective.
The MC34151/33151 driver chip has a fairly high drive current capability at 1A, but that is not an awful amount by modern standards, where some drivers can sink and source 4A to 7A. Modern MOSFETs with lower RDss, tend to have higher parasitic capacitances in general.
There is a spare section of the driver, so maybe you could parallel them up to get more drive. It would be best to fit a better driver though if you were seeking more drive current. But, the thing is that driving MOSFET gates is a complex task with quite a few variables to consider.
Fit a fan- this is not as radical as it sounds- you can use the small CPU type fans which are small, quiet, and compact, but effective.
And a real crazy idea- fit liquid cooling as used on some high end PCs.
I have looked at this approach for a few critical applications and it is possible to design quite a simple, low cost system with a bit of ingenuity. Last time I looked, off-the-shelf liquid cooling kits were costly.
You could consider an altogether more beefy NMOSFET- IXUS make some real beasts.
Check and possibly upgrade the critical capacitors may be worth considering too.
And that is about it.