ESR is the real part of the impedance.
Simple*, make a synchronous RF detector circuit. Apply excitation (presumably, sine wave at some test frequency and amplitude), and measure the complementary parameter (i.e., if supplying current, measure the voltage; or vice versa; or for a general source impedance, measure the ratio of voltages = ratio of impedances).
*In the canonical RF sense. Implementation is left as an exercise for the student, etc. etc.
The usual approach, usually differs in these ways:
1. Excitation is square wave, so the frequency content includes harmonics.
2. Detection by magnitude, not in-phase only, so the impedance |Z| gets measured instead.
3. The measurement is done at some fixed frequency, usually ~100kHz.
Measuring |Z| is only okay when X_C and X_ESL << ESR.
Measuring with a square wave is only okay if this remains true for all the harmonics of significance. That is, for the Nth harmonic, X_ESL goes up as N, but the amplitude goes as 1/N, so it's a wash. As long as all those harmonics stay below the signal due to ESR, it's okay. Eventually, ESL will win out, but no square wave has infinite harmonics, there's always an upper cutoff (where harmonics transition from going as 1/N to 1/N^2 and more). Hopefully, the circuit is designed to filter the square wave a bit, to reject the ESL signal by some reasonable frequency.
Fixed frequency is a range problem. For the ESR signal to dominate, the frequency must be above 1 / (2*pi*ESR*C). Likewise, ESL must be less than ESR / (2*pi*BW), where BW is the signal chain's bandwidth.
Note that it doesn't matter, where in signal chain the filtering is applied. The signal source can be filtered, or the detector can be filtered, or both. The total is all that matters. Note that, for a sine wave source, BW is quite small (the width of the sine wave, more or less), which is why it's the gold standard.
So, all together -- for a typical use-case that's only concerned with electrolytic capacitors over 10uF or so, with modest lead lengths, the typical design is alright. It falls over when the capacitance is small, or the ESR is very small, or the ESL is exaggerated.
And, it's not too hard to make a proper one, following a canonical RF design -- you set up a local oscillator, a buffer amp and a balanced mixer, plus whatever coupling transformers or differential amps are needed to sense the signal and perform level-shifting and all that. The in-phase component corresponds to resistance, so just filter the mixer's output and read out the voltage.
You can even set up a quadrature oscillator (or phase shifter, or..) to measure reactance instead. Note you can only measure total reactance at the given frequency -- while it might be positive or negative (implying overall inductance or capacitance), you need multiple frequencies to disambiguate where inductors and capacitors are. But even if not going to that trouble, you can still draw the equivalent circuit (R in series or parallel with L or C) at that frequency, it just might not be applicable over a wide range of frequencies, that's all.
Tim