No one is talking about giving the Government unlimited access to everything, "any time they want".
True no one is. This is about them having unrestricted access to all encrypted data - that is not "everything".
Just last week, two potentially major events were prevented in Sydney that were widely publicised.
Was that due to having unrestricted access to personal data? I don't think so.
Sure, terrorist attacks are prevented- good. But currently this is accomplished through traditional means and not through mass surveillance or accessing encrypted data.
History is clear - governments, police and intelligence agencies cannot be trusted with unrestricted access to personal data. Absolute safety from terrorist attacks can never be achieved and efforts to pursue it with expansion of the police state will breed more terrorists. The state would love to keep you in irrational fear to justify loss of civil liberties.
Even if mass surveillance and unrestricted access to encrypted data did reduce you risk of being killed by a terrorist (there's no evidence for this) then at what cost? The chance of being killed by a terrorist is currently vanishingly small (more likely to be killed by an asteroid, a shark, a dog, etc) with police agencies using traditional means.
As Benjamin Franklin said: "Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."