On my scope I measure 18.6V peak on the speaker terminals. Converting to rms gives 18.6 * 0.707 = 13.15 volts rms. To get power I use (13.15^2)/8 = 21.15 watts.
That's correct. If you have Vpk amplitude, you can just divide square by double load:
18.6^2 / (8 * 2) = 345.96 / 16 = 21.6225 W
My speakers are 6 ohms. So if I measure 18.6 volts peak at 8 ohms, does the voltage stay the same when I connect 6 ohm speakers?
No, it depends on your amplifier output impedance. You can check it by measure output amplitude for two different known load resistors. More difference for resistance leads to better precision. But at the same time very high or very low load can be out of working range for your amplifier, in such case the result will be distorted. Resistors should NOT be inductive.
For example, for exactly the same input:
At 8 Ω load your amplifier produce 18.6 Vpk amplitude.
At 4 Ω load your amplifier produce 17 Vpk amplitude.
Then you calculate amplifier output impedance:
Z = (R1 - R1*(U1/U2)) / (U1/U2 - R1/R2) = (8 - 8*(18.6/17)) / ((18.6/17) - (8/4)) = -0.75296 / -0.90588 = 0.8311 Ω
When you know amplifier output impedance, you can calculate output voltage for specific load (simple voltage divider). And when you know voltage and load resistance you can calculate power for a new load: P = Upk^2 / (2 * Rload)
But since most of audio amplifiers usually have pretty low output impedance, the result for 6 Ω load should be close enough to 8 Ω load.