Thanks all for your replies, I've put everything back on the breadboard again.
You could try a common mode choke (a few mH) directly after the USB connector. If that's not sufficient add a Schottky diode and a large electrolytic cap behind that. The diode keeps the filter and decoupling cap local.
I've tried a BAT42 schottky diode with a 470uF cap, but there was no noticeable difference.
Instead of joining the top end of R4 directly to VCC, which is coupling any power supply noise directly into the audio path with no PSRR at all, connect it via an RC filter.
With the phantom power as AndyC says, take R4 and split it into 2 resistors, 470R to Vcc with a 1k resistor to the mic, with a 47uf capacitor between the junction of the 2 resistors and ground. That should provide the decoupling you need.
Thanks Andy and Sean, this has helped a bit, I also added a 470uF to the LDO's output which also helped greatly.
Before on breadboard:
After 470uF:
After 470uF and RC filter on Mic:
And it drops even more if actually touch the phone:
It sounds better now, there is a tiny bit of noise but I can live with that.
Now the next problem is if you plug in the iPhone USB to the PC, it makes everything worst, haha -
I think I'll add the 470uF and Mic RC filter on the PCB and see if anything changes.
Edit - Here's the noise on the PCB now (without phone connected to PC), not as good as on the breadboard for some reason but still passable:
Have you looked at the noise level on the USB power supply? There are a few potential issues, in order of importance:
1) Your 1/2 Vcc divider (R1,R6,C2) provides only ~10dB attenuation at 1kHz
2) The ripple rejection of the LP2950 drops pretty rapidly above 1KHz
3) The LMV358 doesn't have particularly great power supply rejection.
I'd start by increasing R1, R6 and C2. Going up by a factor of 10 for all three parts will give around 50dB attenuation at 1k.
2) and 3) - Thanks, noted. With the LMV358, were you looking at the -kSVR or something else?
After I add the 470uF and Mic RC filter, I increased R1, R6 and C2 by 10x except for R6 which only did 5x and it's kind of just increased everything a bit -
You have about a half volt of allowable headroom on the LDO, so try a RC input filter on it, probably a 2R to 10R resistor and a 100 uf capacitor at the chip input. That will reduce supply noise a lot, and as a plus it provides inrush limiting on the USB line so you can use a 220uF capacitor to provide extra filtering.
After I add the 470uF and Mic RC filter, I tried 10R and 100uF at the LDO input but it didn't change it much, however stranglely enough if I try it on the output, it's reduced it by a small amount (compared to the after 470uF and Mic RC filter) though I get waves appearing now -
Can you show a layout? Do you have a ground plane or separate traces? A bad ground strategy could cause what you see.
Sure, the USB connector's isn't connected to ground, I have tried with a wire and it hasn't made any difference.