You can obviously hear it, otherwise you need to see a doctor.
My colleagues ask me to turn it off when I'm not using it due to the noise. (2-3 meters away)
Although I don't think it's very annoying, you do get the feeling "ah much better" when the noise is gone.
And if you are alone in a silent office you may also be bothered by the hum, I find that more annoying, I can hear that hum as soon as I enter the room but when the fan is on the hum isn't obvious anymore.
That is also the reason why I disconnect the power line when I don't use it. So the standby button is more a reboot button for me.
Having some music playing helps to forget the noise.
P.S my monitor or tablet do not have a noisy fan, even my laptop isn't making a sound now, not even a power adapter hum, it's certainly possible to design it silent but I guess then it will cost more or maybe they didn't care about it very much 'good enough state of mind'