Well, I am using a meanwell isolated DC-DC converter brick that has good line regulation, from LA battery to 5V arduino pin.
I had not noticed it right away, but it sunk in later, that after connecting power to Arduino all 3 LEDs were on and one was blinking (probably signaling data write to an SD card).
Now, why would all the lights be on? I recall when powering through USB that there was just a red LED, and another green/red (or yellow) LED would flash when programming. 5V should be the right pin since the voltage is regulated.
Oh, it is working as there is an OLED display.