Optocouplers aren't a relevant solution for your problem. You have a motor that's drawing a lot of current because it's stationary and low-impedance; that current is overwhelming the power supply whose voltage sags, and the controller browns out because it has insufficient voltage. Once you reason that out, the solutions become clear. Either:
1) Use a schottky diode and a capacitor to power the control circuitry. The capacitor is placed directly at the power input to the control circuitry, to hold that voltage steady. The diode is placed so that if the motor/power supply voltage sags, the capacitor won't discharge backwards through the diode to feed the motor -- the schottky diode enforces that the capacitor's energy is reserved for the controller circuitry only. To determine the size of capacitor you need, measure A) how long the motor spike overwhelms the power supply for, B) how much current the control circuitry consumes, and C) how much voltage sag your controller circuitry can handle, and calculate A*B/C (with units) to figure out your capacitor value.
For example, a 20 ms current spike, 50 mA control circuitry with can withstand a drop of 1V (e.g. nominal 5V to brownout at 4V) gives 1000uF (click for Google calculator).2) Last resort, use a completely separate power supply for the motor. But you should never need to do this.
And/or as suggested above, implement soft-start and/or use a more powerful power supply. Or, combine all these methods: better power supply + soft start + schottky-enforced capacitor are all good practice to avoid this sort of issue. But regardless of all of the above, an optoisolator isn't relevant or useful for any of this.
evb149, using heavy wire to the motor isn't going to help, because that just helps the motor drag the supply voltage down.
Edit: Using an inductor achieves a similar goal as the schottky diode; i.e. preventing the capacitor from just being discharged by the motor -- but I'd opt for Schottky because it's easier/smaller/cheaper for a given level of performance.