It's probably even more than that, your circuit just isn't well matched (i.e., probing it with a scope, you will find the probe has relatively low characteristic impedance at high frequencies; a constant-Z probe would show different results).
In general, wires and traces cluster in the 100 ohm range, characteristic impedance. Resistors near the same range (say 22 to 220 ohms) tend to have high cutoff frequencies, with values farther out having a ~proportionally lower cutoff. (Example: if a 1k resistor has a cutoff at 500MHz, a 1Meg resistor has a cutoff at 500kHz.)
So a somewhat lower impedance pot may be better suited to your application.
Other features to expect: single turn, vertical adjust types will have a single horseshoe of resistive material, and sit relatively close to the PCB ground plane. The length of the track affects frequency response (everything here is proportional to length, of course), so small SMD pots are better than large THT pots, and vertical is better than horizontal adjust.
Needless to say, multiturn pots are much poorer -- the track is a helix going around the shaft, so will have lots of inductance and weird "hook" effects (distributed capacitance); as will any wirewound type. These are best left for low frequency application, like adjusting DC control voltage to a VCA or such.
Tim