@elektrolitr, I was hoping to find a power bank that already had both USB connectors in a nice case. But at this point I only know of one like that which is also "always on", and it's $30. If I can't do better than that, I'll have to resort to modules.
But I'm surprised that so far I have not even found one single-module solution that's a combination LIPO charger and boost converter with load sharing. It seems the module kings of the Far East could build a combination module like the one you referenced, but using the MCP73871, which has load sharing built in, instead of the TP4056, which doesn't. Or use the TP4056 but add the mosfet, diode and resistor needed for load sharing. I don't understand why there is no such module available on Ebay.
With respect to just using a battery, I too have done that for an IR remote control using an 8MHz Pro Mini. But that won't work for all projects. In addition to the processor speed issue, there are many peripherals such as sensors, SD cards, and so forth that need to operate at a specific voltage - some at 3.3V, some at 5V, but many may not work at 3.5-4.2V. So powering directly from a LIPO may work fine for some projects, but not for others.
Anyway, I still think the world needs an "Arduino powerbank" for a lot less than $30.