So turns out the issue is the Arduino Mega that was used.
It is an official Arduino Mega ( I have about 20, all work the same) but when I tested this setup with an Arduino UNO (official also) It works perfect.
I have seen other issues with the Arduino Mega ( Clone but it would lockup when enabling the watchdog timer).
Or just RTFM, Arduino mega has specific pins for SoftwareSerial library.