Anyway my thoughts so far are basically that in memory hash tables have had tremendous development in the last 30+ years
Very true.
Chiefly, the quality and speed of hash functions has vastly improved, with near-cryptographic quality available cheaply, meaning every bit of the hash is equally random, allowing simple power of 2 table sizes instead of prime number table sizes.
It describes a linearly growing hash table that works by appending new buckets on the end one at a time to avoid rebuilding. That seems interesting but how often can you linearly expand a data structure in memory but you couldn't just as well make it larger to start? So they add an extent allocator underneath, adding an extra level of indirection. That's fine but if you want to store really huge tables it's going to run out of space unless you make more layers of indirection at which point you actually have a B tree.
On a 32 bit machine, two levels of pointer arrays and 4K blocks for both pointer arrays and data gives 1024 pointers in the root block, 1,048,576 pointers in the 2nd layer, and 4 GB of data (hash table in this case) in the final layer. Ok, 4 GB minus epsilon, since 1025 of the 1,048,576 4K pages are used up by the pointer array blocks.
This is a pretty feasible solution on 32 bit machines, and one I've used myself to implement all potentially "large" data types on small machines: arrays, stretchy arrays, strings. It vastly simplifies your memory allocator / GC to only have to deal with blocks of size 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 (9 sizes). The last (or only) block at each pointer layer and the data layer can be smaller than the full 4K.
It's not even slow, because the top level index block and whichever 2nd level index blocks you're using at any given time will be in L1 cache. And with always a known fixed 2 levels (NOT the case with a B-tree), the lookup code is branch-free. It's a few instructions, but trivial compared to calculating the hash function.