I'd suggest posting the PCB layouts. Then people here can check it out, especially as regards things like the crystal section, missing or poorly placed decoupling capacitors (as already mentioned by another poster), and other things.
I went over my PCB design and evaluated it against various posted guidelines and two things are clear violations from what I can tell:
1) I used vias on my crystal signal lines, didn't cross the audio output though
2) I violated the length tolerance for the audio differential pair - not sure I can do better because of the switching jack
3) not clear: I have a ground plane under the crystal - lots of differences of opinion here
What I don't understand is why I need the combination of 3 things to fail:
1) unpowered speaker
2) using the external crystal
3) high volume PWM output - either through the pot or through the code
It must be all 3 happening for things to fail - I can see how 2) and 3) together might cause interference, but I don't see what role 1) would play here.