Can't you just use an insulator instead of the battery terminal, maybe something like a piece of plastic or wood?
Think about this as a circuit. Ground is just a concept. What is ground? Is it the same in the space shuttle while orbiting the earth, as it is in your computer? Probably not, there is likely a voltage potential between those two grounds, so they are not the same.
Sometimes it is easier to think of abstract concepts when we choose something more visible. I am sitting in a chair at my desk. How fast am I moving? My chair is moving at 0mph with respect to the earth's surface, but the earth is moving around the sun at about 66,600mph! So, how fast am I moving? Until we set a referance, you really can't answer.
So, the circuit is much the same. For any power source to work, a circuit must be formed. You can draw the circuit showing all the nodes normally connected to ground as a wire, and the circuit will work just fine. The ground symbol is simply a convenience for us to label those nodes, in a way that shows what our reference is.
As far as connecting the bottom of the battery terminal to plastic or wood, those are both good insulators. What happens if on the positive side of the battery you wind a wire around one end of a wood stick, and the other end of the wood stick attached to a wire that goes to a light bulb, and the other end of the bulb to the negative terminal of the battery? The bulb doesn't light, as the wood doesn't conduct electricity, so no electrons flow. The wood acts like an open switch. If you put the open switch on the bottom side of the battery, do you expect a different result? Think, how will the electrons flow to complete the circuit?