What about using the LCD (maybe even dot matrix), but to hide for example red sticker behind it. When you turn off the power supply, than it will not be hiding the thing behind it. Also you can use 7seg LCD with reversed polarizer (numbers will be transparent for the active segments with black surrounding) and turn on the 1 and have green background. Such a approach will work even without battery, but will have higher power consumption with good battery.
Other thing what I have met, is used in smoke detector (Hütermann F9), which beeps once every 43s, when the battery is low.