Datasheets for convinience:
IRF510:
https://www.vishay.com/docs/91015/sihf510.pdfIRF530:
https://www.vishay.com/docs/91019/91019.pdfI haven't looked properly but I notice that the IRF530 has 3x more gate charge and all the switching parameters are slower. This is expected given you get the improved thermal performance by having a bigger die. It may well be a suitable MOSFET but I wouldn't just throw it in without re-examining the gate drive and testing on a prototype. If you unintentionally slow the switching then you will increase dissipation and it may very well run hotter despite the factor of two in R
thJC. Essentially the real figure to consider for thermal performance is the change in sum R
thJC+R
thinterface+R
thheatsink which is around a change of 6.5 C/W to 5 C/W with the 100x100mm heatsink and a greased mica insulator. Don't pay too much attention to the headline power figures in the datasheet, they're not really representative of real-world use (as in you can't usually get anywhere close).
I suspect the MOSFET-killing on switch-off relates unintended linear operation dissipating hundreds of Watts for milliseconds and taking it outside it's safe operating area. That should be fixed by the undervoltage lockout circuit.
I personally wouldn't change it, mine seems pretty robust and thermal performance seems sufficient with the 100x100mm heatsink everyone else has used. During testing I had it running continuously into a 50 Ohm load (equivalent of a cold tip that never warms up) for about half an hour and the MOSFET was only warm - maybe 40-50C.
If you do want to change it the datasheets offer hints but as they're not really specified for use as an RF power amplifier you will probably have to do it by experiment. I would stay away from the IRF540, as a minimum you would need to seriously beef up the gate drive (9x more charge needed, so 9x the current to do it in the same time), but the rise and fall times might still catch you out. Paralleled smaller MOSFETs might be simpler.