So with my current process, I'm at about an hour per a panel including cleanup. I have the gear to make double-sided panels of A4 size. Then it's cutting, drilling the holes and s.o. which adds another 20 minutes. I'd say that's pretty fast.
Still I don't think it's worth it, most of the time. Even if you earn exceptionally little, making a panel is about 25$ total in time and material. About the same that DirtyPCB charges. But with them, you get it without lifting a finger and with much better quality. Making your own boards means making a lot of mess and keeping supplies around the house. Cleaning up the former is annoying work. I like my day job much much better. So I'd charge much more for an hour of PCB making, than just what I make at my day job. Storing the supplies is also annoying and takes up considerable space, even further adding to the perceived costs of board making. And on top of that, you have to design the boards more carefully than you otherwise would. You can't use that fine a trace, that many vias and so on.
The only argument for make your own PCBs is: "But I get my boards faster that way". It's not a strong argument though, because let's face it:
You rarely have all the parts, you need at hand. You'll almost always have to wait for a mail order of something - you might as well spend that time waiting for components and ordered boards.
Bottom line: Only sadist or very very time pressed people manufacture PCBs at home.