Those chinese ready-made attenuators (have a bunch of N-connector type ones) are definitely quite small for the rated power, would guess the temperature rise would be extreme.
I also use them mainly as a protection of equipment when probing unknown stuff and am aware of the bad heatsinking. To be honest, I have never disassembled any of those I have, so I am not even sure, what is lurking inside. 2512 SMD resistors are certainly not good enough for a "10W" attenuator, regardless of heatsinking.
I can test them, but only with DC voltage, I do not have any suitable higher powered RF source.