The audio signal is there anyway and the AM envelope should modulate the flux in the headphone magnets. Interesting - I've never thought about this...
Maybe the reason is the magnets are too slow
Yes , with sufficient amplification you could hear the signal directly, it's not that dissimilar to class-D "inductorless" headphone amplifiers.
In practice it won't work, as the amplified RF signal will get radiated from the speaker/headphone wiring back to the antenna, and cause the radio to oscillate.
Even with a superhet, amplifying the 455kHz signal enough to work the headphones/speaker will cause that signal to get coupled back into the radio too and make it unstable.
In the early days of radio they had "reflex receivers" where a small fraction of the RF was fed back to the input to make it marginally unstable, this greatly increased the RF gain.
Another aspect to consider is automatic gain control , incoming radio signals can vary over 1000:1 from different radio stations. You need a detector to be able to control the gain of the IF stage, this needs to make a DC voltage to change the bias on the transistors in the IF stage. Without AGC , the RF signal to the headphone speaker will either be saturated or too small to be audible much of the time.