Never mind. Found the problem, and I chuck it up to inexperience with motors and motor drivers.
What happened is that the big yellow fischertechnik motor needs a lot of current to start, and this causes the voltage to drop all over the system. The buck regulator needs more then 4V on the input to keep on working and it dropped to just under 4V. The MCU sees this as a power down and up again and resets. Receives the signal from the transmitter and tries to start the motor again, causing another reset, and so on.
The 1508S IC's did not like this and died. Tried it with a MX1508 today, and it survived, but as it can only handle peak current of 2.5A it warms up. The prototype works due to a slightly different power supply setup, and during most tests in the beginning the MCU was powered from USB.
When supplied from a battery there is a diode, followed by a 47uF capacitor to ground, in series with the buck converter so the voltage drop on the input is dampened by it allowing the MCU to keep on going. Once the motor is running the current drops down to ~700mA, which is fine for the MX1508, but when the speed is reduced it does starts to heat up more.
By using the FET Q1 as a diode due to disconnecting R1 and placing it in the place of ZD1 and adding a bigger capacitor across C2 the MCU stays up and running, and the motor can run. The power supply needs to be capable of delivering the start up current though.
All in all a redesign is needed. As is, it can work with small motors but not with the one it basically is all about. Not a big deal. The MX1508's are always handy for other projects with smaller motors and the PCB's costed less then 4 euros, not a big loss.
I should have looked more into the original receiver. It uses a H bridge made up of BD433's and BD434's. These can do 4A continuous and 7A peak.
A good learning experience.
Attached are a couple of screen capture of the measurements on motor startup. The first one has the input voltage on the yellow trace, the MCU voltage on the cyan trace, the MCU analog supply on the green trace and the reset line on the magenta trace. The second one has the input voltage on the green trace, the buck converter input on the cyan trace and the MCU voltage on the yellow trace. The magenta trace is not used in this case.