I don't have a problem with the linear regulator. Switching regulators tend not to be very efficient at very low currents. You don't have a big voltage drop here, and the regulator has a 1.6µA quiescent current, which is pretty good.
You can charge the battery when the switch is turned off, but may not be able to do so safely when it's turned on. You would be drawing current for the regulator and controller through the charger. If that current is too high, the charger will not be able to sense that the battery is fully charged, and may continue charging. That's not safe for lithium batteries, which cannot be trickle-charged. It depends on the current needed for the load, but in the worst case you would have to turn the switch off while charging the battery.
What's needed is a "load sharing" circuit, which both charges the battery and powers the load, independently, when USB is plugged in, so that battery charging can proceed properly to termination. You might see if Adafruit has a charger which uses the MCP73871, which includes the load sharing circuit. Info on load sharing can be found here:
http://ww1.microchip.com/downloads/en/AppNotes/01149c.pdf