This is somewhat easier to understand in the world of RF where everything is in impedance matched lines. In that case, the real quantity of interest is "watts / Hz" -- how much noise power there is in a given bandwidth. If you take your circuit and send its output through a bandpass filter with a given bandwidth, then onto a power meter, the signal detected will depend on the bandwidth of the filter. You just multiply by the bandwidth BW to get watts: the signal from your meter. The tricky part comes when you want to find out what the voltage is. With a fixed impedance (say 50 ohm), watts = V^2/R. Thus, you can see that the voltage noise is simply:
Vn^2 = noise_power_density (W/Hz) * BW * R.
Now, we can't resist the temptation to take the square-root of this, and solve for the RMS voltage noise. Then we get this funny unit "volt / sqrt(Hz)" what that means is that you take that quantity, multiply by the square-root of the bandwidth to get the noise voltage. This means that if you double the bandwidth, you double the noise power, but the RMS noise voltage only goes up by sqrt(2). You need to quadruple the bandwidth to get the voltage to double.
The nice power analogy doesn't quite apply as cleanly in non-impedance matched systems, but the basic scaling remains. If you have a random noise voltage source, the RMS noise voltage is proportional to the square root of bandwidth.