Do you have DC current running through the wiper? Even small amounts can significantly increase scratching noise. A simple fix is to use a DC block capacitor in series with the wiper, while making sure you have an alternative biasing path for the following stage.
Edit: It's possible to make a voltage controlled attenuator, but making one with low distortion and good tracking between channels can be pretty expensive and involved, so it depends on your requirements. LDRs are a disaster in this respect, I once tried to make a two-pole state variable filter with fancy NOS Vactec vactrols, and the two devices I used from the same batch had a 3:1 variation in resistance at the same current between each other. This would obviously not be an issue with mono sound, but they also change with temperature and light history. Some amplifier ICs have built-in VCAs for volume control, and you get dedicated volume control ICs from THAT, Alfa RPAR, Sound Semiconductor and a few others.