Instead of making a wall to create a separate room, make a cupboard/box for the rack instead.
Although server rack hardware noise profile tends to be dominated by high-frequency noise, start by eliminating low-frequency conducted noise.
You do this using something that builders call a floating slab: a thick layer of rockwool or similar insulating wool on top of the base concrete, with a reinforced slab of concrete on top of that. The only contact between the concrete slab and the concrete foundation is the insulating wool. Minimum thicknesses depend on the area, but knowledgeable building engineers can give you exact figures. Your box then builds on top of that floating slab.
(It is how in e.g. apartment buildings you can completely stop conducted noise through floors. With a correctly done floating slab, you can walk with wood clogs, and the downstair neighbour will hear absolutely nothing. It does add weight and cost, as typical floor slabs are at least 4" thick, also reducing room height by about 10".)
High-frequency noise can be absorbed by normal acoustic panels. Air cooling intake and exhaust are your next focus –– unless you build a rack out of a fridgerator with sufficient continuous cooling capability for your rack hardware, say used restaurant cooling cabinets or similar. Soft foam baffles and dust covers, and as large fan diameters as you can fit, can help a lot.
I've built silent PC enclosures (but latest one almost a decade ago now). The tricks there are similar: you use an outer box as the shell (this corresponds to your existing basement space), then an insulated inner heavy box for the actual hardware. You carefully design the airflow so that it has enough turbulence to ensure all surfaces and components are kept cool, and also to hinder sound propagation. Using convection (hot air rising upwards) can help, but can also create laminar flows with "backwater pools" outside the flow where the air does not mix and just heats up. Foam baffles work well, as they also absorb higher frequencies. Cheap soft closed- and open-cell foams work well, and you can use gasket silicone (non-acid curing; preferably platinum-curing ones, like you'd use with electronics) with such foams to "glue" e.g. case fans, stopping all conducted noise. Personally, I'm still looking for cheap very low durometer (i.e. soft, spongy) silicones a 'hobbyist' can use; the ones I have easy access to here are automotive (gasket silicones) or household (but these often have antifungals and tend to be acid-curing) silicones, with hobby shop casting and makeup/effect silicones either too expensive or sold in too large buckets.
That said, my own limit for furniture-hidden "silent" PC enclosures is around 100-200 watts of heat produced, and for anything substantially more, I'd actually go for compressor cooling –– the fridge approach. Running that does cost relatively more, depending on the hardware of course, and you need to use e.g. desiccants (larger silica packets you can dry out in an oven, for example) to keep the air inside dry and not condensing to the surfaces and ruining your electronics.