I think your circuit is a very good start, but I have a couple comments.
First, I'd suggest you always show the body diode of the mosfets in your circuit. That will help you see what they do. In your circuit, you have M1 and M4 configured in opposite directions, and it's not clear why.
It seems to me that M4 is correct as shown. Even though the source is on the right, which is the opposite of the typical orientation for a P-channel used as a switch, when SW1 is open the body diode will block current from flowing from M1 back through the M4 body diode, and raising the M1 gate voltage, which would turn M1 off. So M4 looks right to me as you've drawn it.
But you have M1 oriented the opposite way, which means when SW1 is closed and M1 is off, current can still flow from M4 back through the M1 body diode to the battery, which you may not want. As long as the battery is less than a diode drop below the 5V V3, no current will flow. But as the battery discharges, the forward bias of the body diode will be exceeded, and current will flow.
Now if you reverse the orientation of M1, then even when it's off, current could flow through the body diode to the load. But that would only happen if Vbat is at least one diode drop *above* V3, which I assume will not be possible. So I'd suggest you try reversing the M1 orientation so that it matches M4. Mosfets generally conduct equally well in either direction, but the orientation of the body diode matters.
I don't think your choice of P-channel mosfets is right for these voltages. The Vgs threshold voltage can be as high as 4V, and that's the voltage at which the mosfet only *begins* to turn on. For these voltages you need a "logic level" mosfet - one with an IRFL part number, or something similar from another manufacturer. The Vgs threshold should be near 1V maximum so the mosfet will be fully on at 4.7V.
Other minor things: If you have R2 and R3, you don't need R1. And you don't need R4. I think R5 and R2 could be higher value, like 10K.