If you can find a sound card that doesn't filter out the high frequencies, that's a good option. You can get very low noise and very high dynamic range with standard drivers and a selection of programs. My current audio interface though, for example, has a pretty sharp rolloff at about 21.5kHz, so it wouldn't be suitable for your application, and cheaper options probably don't specify whether they do this or not - so the best value would probably be had in finding user frequency response or noise floor plots that go that high.
They don't make them really anymore (I think the soundcard based solutions have taken over), but if you're willing to consider used gear, look at Dynamic Signal Analyzers. Dave's mentioned them in a couple videos and there are a few options available, they are basically spectrum analyzers for the low end of the frequency spectrum, and were designed for vibration or acoustic analysis. There are newer ones, but not from many manufacturers and not at nearly the same value proposition as the older units.
If you find one with the frequency response, I do think the soundcard route is the best bet. Something like the QA401, which has a generator and is designed for audio work, may be ideal because they even use BNCs for a differential input and you wouldn't have to go swapping cables everywhere, but I don't know whether or not they filter the high end.