The battery CAPACITY is 4.2 Amphere-hour (Ah). It more or less means that if you draw 4.2Amperes it will last 1hour, and if you draw 2.1A it will last 2h However, on the battery you should see what the maximum current draw is (e.g. "max load 2A").
Generally, the fuse has to be rated just above of your PROTECTED DEVICE's maximum current (sinked or sourced).
Meaning that if you want to protect a circuit that draws (sinks) more or less 1A, you use a fuse "slightly" bigger than 1A, for instance 1.6A.
If you wish to protect the battery, than you have to figure out what is the maximum current that the battery can safely deliver (source). It's either written on the battery, or you can guess what it is. For instance let's take 2A as your battery's maximum (instantaneous) current rating. You'll need a "slightly-more-than" 2A fuse, for instance a 2.5A fuse. You will tend to choose a higher value here, because there might be high current surges on startup. However, a regular 2.5A fuse doesn't blow instantly at 2.6A. See .
All fuses have to be rated to the maximum working voltage, in your case 12V and that doesn't present a problem, as 5x20mm glass fuses are generally 250V compliant. Voltage rating is mostly about arcing and that tends to be a problem mostly at higher voltages, in the hundreds of volts area.