XH and PH are friction-locked connectors, where you have to overcome that friction to unplug. You may find a latched connector easier: you squeeze on the latch to disengage it and then pull out. You’re probably familiar with the Mini-Fit Jr as an example. (Though it’s much bigger than you want.)
There are numerous small connectors with real latches. I like the 3mm pitch Molex Micro-Fit for power applications, but there are also smaller ones, like JST XA (2.5mm), JST PA (2.0mm), JST PNI (2.0mm) TE/AMP Economy Power 2.5 (mm), TE/AMP Grace Inertia (2.5mm), TE/AMP High Performance Interconnect (2.0mm), Molex MicroClasp (2.0mm) and many more. There are even smaller ones, down to 1mm pitch, but crimping those is a pain in the butt.
Molex KK254 (and the countless clones thereof; 2.54mm) is another option, using a friction lock that is generally a bit easier to unplug than XH.
Go on Digi-Key and go into connectors > rectangular connectors > rectangular connector housings. You can filter by “fastening type” to find these things.