You absolutely don't need the Linux kernel if you want to do some bare metal programming, actually that will be both useless and dangerous for your goal because you'll get quickly lost in the complexity of an multi-platform, multi-tasking and multi-user kernel.
The best way to do this is to get the uBoot source code and and work from there, the uBoot actually does all the complicated low level stuff, like setting up the RAM and MMU, basic interrupts, serial ports, GPIO and other stuff necessary for Linux to boot later. It also has nice primitives to access all the flash memories, interfaces, set-up USB controller and even mount USB sticks and not to forget, a full network stack. Of course this is modular and you can use only what you need.
The uBoot has a nice console, you can develop and run your own programs under it (in the source code there are some examples on how to do it) and the source code, if you prune all other useless files, is not that horribly daunting as the Linux kernel.
And, as I build, compile and load uBoot with Eclipse CDT, you should be also able to do it as well, if you want a nice friendly IDE, if you can live with compiling the code yourself, I strongly recommend Visual Studio Code for Linux, it's absolute user friendly.
Later, when you've had enough fun with uBoot and got to know better the platform, you can use a light RTOS of the many existing ones (FreeRTOS, Azure, etc).
Good luck,
DC1MC