Diecast (eddystone etc..) boxes tend to have smooth walls (otherwise they would get stuck in the mold tools), so shouldn't be an issue.
The main problem is that they tend to be thin walls, this limits the thermal resistance somewhat.
As for insulation - probably look at bergquist gap pad, I have had a lot of success with this stuff, and it's available in some pretty thick formulations which can help. The other thing I do now (for high dissipation) is not to bolt power devices down to heatsinks, but to arrange either a spring clip or a bar to press down on the device.