It turns out that the GPIO is indeed enabled when the MCU is running in FEL mode. I hooked up my MSO5074 to the touch panel connection (Port A) and used sunxi-fel to set the port configuration to output and was able to toggle the pins and see it on the scope. Wrote a program to do the same, uploaded it to 0x80000000, called exe on the same address and it did not work.
So now I'm wondering if it is the internal DRAM that needs to be enabled first. I read the three BUS_CLK_GATING_REG registers and the only part turned on is the USB_OTG_GATING.
There is a bit SDRAM_GATING, but most likely other registers need to be configured to make it work. It is back to the Ghidra output and check the SPL part to see what is done in there.
Edit: Had a brain wave
There is SRAM at address 0x00000000, that can be used. Tested the back-light code there and it just works. This means that for testing bigger stuff FEL is not the way to go, unless a script is made to use the sunxi-fel writel option to configure all the needed registers for enabling the DRAM.
The default PLL_CPU setting is 0x80011000, which translates into 204MHz instead of the 408MHz mentioned in the manual.
Edit2: Used sunxi-fel to set the clock to 600MHz and tested the back-light code again. It still works. Tested it with my port A toggle program and was able to verify that the MCU is indeed running on the higher speed.