It is probably not advisable to power the speaker via the button battery.
If you have to have some sound, you may consider piezo elements.
Why? I've got a dual common anode zener diode across the piezo transducer to protect against kickback.
There's nothing wrong with your circuit and yes zener diodes are a good idea, as a piezo can generate fairly high voltages, in response to mechanical shock. I think the reason for the confusion was because you didn't change the symbol in your circuit.
http://electronicsclub.info/circuitsymbols.htm#audioI don't have a problem with any of those statements, but I thought it wouldn't work that way in practice.
As it turns out, it does. I just tried it myself. I stand corrected. It works, at least on my 'scope.
It's effectively an H-bridge. When one pin is high and the other low, the voltage is positive and when the logic levels are reversed, the polarity goes negative. The piezo will be louder because the voltage swing is double and it's being powered continuously.