It's possible (very likely!) the battery is weak and can hardly hold configuration data. Restoring defaults saved "ok" data into NVRAM, but that's only a temporary fix, another failure will happen soon.
I'm not sure if the battery is soldered, but I guess it is.
If you chose not to replace it, remove it - the Laptop should work fine, but will always using defaults and of course date/time will not be not set. Leaving a half dead battery in there might corrupt configuration data, I would not recommend that. It might start to leak, too.
The DVD-drive should not matter, if you don't need it, leave as is.
I would not recommend USB as a boot medium, a HDD is faster. Or if you want to fiddle around a bit, there are ATAPI to SATA converters, which sometimes work. That might enable you install a SATA SSD. However, that might not work, it's experimental at best.