The A-Star board has a 4.7uF decoupling capacitor - would that make that much of a difference?
If you mean C3 4.7 μF and R2 (1 Ω), then they are probably a requirement of the specific LDO datasheet (I can't see the LDO part number), as it specifies (I suspect) a minimum ESR (equivalent series resistance, some types of capacitor can have a value, which is too low, to work well), for that capacitor, to ensure it damps down oscillations (for stability).
So, it shouldn't matter, as you are not using an LDO.
In your PCB layout. Are things like the crystal, its supporting (load) capacitors and the decoupling capacitors, very, very close to the IC pin(s), that need them?
Did you source all your parts from reliable sources?
I.e. Not from AliExpress/Ebay Chinese sellers, with x10 .. x50 parts for $0.25 total, for things that normally cost at least $2 each, everywhere else?
(Because of possible, fakes, rejected components, substandard-clones, reused components, faulty parts, etc).
Very, very close, on both the MCU and the amp. All my parts are from LCSC, the parts provider for JLCPCB.
I am now wondering specifically about the value 4.7uF because I can see it on the Nano schematic as well https://www.arduino.cc/en/uploads/Main/Arduino_Nano-Rev3.2-SCH.pdf
One last update: No more issues when I disable the crystal and use the internal oscillator instead (but of course the audio quality is now terrible). I am not sure what to make of that.
Should I have a ground plane behind a 20MHz oscillator and its copper traces?
Would the 4.7uF polarized capacitor have helped?
I'd suggest posting the PCB layouts. Then people here can check it out, especially as regards things like the crystal section, missing or poorly placed decoupling capacitors (as already mentioned by another poster), and other things.