On the same book Self gives some clues about why such capacitors are much larger than initially thought. It is related to avoiding/minimizing distortion caused by the caps. See chapter 2.
I don't have the book. but minimizing distortion by using electrolytic output caps without DC bias seems way out.
As Tation posted, on page 72 of the 2nd Edition, Self starts a detailed analysis of electrolytic capacitor distortion, based on measurements.
Some points of his, specifically for unbiased electrolytics:
1. Do not use electrolytics to set time constants, as in equalization networks.
2. As Tation stated, he recommends using larger capacitors than required for high-pass filtering, to reduce the voltage across the capacitor itself and thereby reduce its distortion.
3. Rule of thumb: keep the AC voltage (within the amplifier's bandwidth) below 80 mV (again, across the capacitor).
4. If no substantial DC blocking is required, a lower-voltage capacitor can be used (carefully) for a higher capacitance in a given space.
5. Assuming these small AC voltages across the capacitor, DC bias is not required.
He demonstrates these points with data taken with his trusty Audio Precision analyzer, connecting the generator through an unbiased polarized capacitor to a reasonable load resistor at the analyzer input.
He refers to a series of articles on "Capacitor Sound" in Wireless World From July 2002 to March 2003, specifically part 4 in November 2002 on the subject of DC bias.