As long as you use a modern switcher with a high switching frequency, I think you'd be fine, since the majority of the noise is most likely harmonics from the switching frequency, plus noise from the diode and MOSFET (both pretty high frequency). Your speakers make a fine low-pass filter, but I wouldn't be so comfortable with an ancient 30kHz switcher, which is too close to audio frequencies for my comfort.
A switcher + LDO actually pretty common, I believe. The efficiency of the LDO can actually be pretty good, since you can set the switcher to slightly above the dropout voltage. But I wouldn't expect the LDO to reject much high-frequency noise either, check the NMRR vs frequency graph in the datasheet. As a general rule, it's quite hard to get clean output from a switcher, which is why low power low noise loads like multimeters and low-power lab supplies are still linear. But it should be doable to make it good enough for audio (including the NMRR from the amplifier), since there's tons of audio equipment around with switching regulators, and even PWM amplifiers (class D?). Especially if you're not trying for studio quality.