Together we stand, divided we fall... It's never a good thing to lose influence... Not for the EU and not for you Brits.
Trouble is, putting all the eggs in one basket means if the basket drops, all eggs are broken.
History shows there are bad people who manage to get into power. Had Europe been a single unified government (no Britain) when Hitler got into power, there would have been no recovery. America would not be able to do what it did without the "unsinkable aircraft carrier" that was the British Island.
The more nations, the more the competition, and the more the options. Competition helps every nation improve themselves. When a nation becomes a failed nation, that it is not a "one world government" means (a) the average folks has examples how other nations work better and (b) some place to run to for safety. With a "single world unified government", folks will have no where to go. They will be totally under the thumb of the bad lone government.
Nations are choices available for people. A smaller number of nations means a smaller number of available alternatives for people. Governments are never fond of giving there own people choices.
That said, it is important to add that that I am not in favor of unrestrained immigration. A nation must have control of its identity, otherwise, it is a a piece of land without distinquishing characteristics. Immigrants must understand that if they make their new home just like their old (non-assimulation), then the new home will have all the negatives of the home from which they escaped.