Residual magnetisation of the core is one issue. If the supply is switched on at the beginning of a half cycle that will give the same flux direction, unless the transformer has a significant excess core area, its likely to saturate.
Our workshop Variac often tripped the breaker on the isolating transformer even if it was switched on with no load. It certainly didn't have any capacitors. The only reason you don't see problems more often is that most fuses and circuit breakers have a surge capability significantly in excess of their continuous rating.
You are exactly correct, core remanence is one culprit, and the resultant inrush current can in some cases be very large indeed. Remanence is the magnetic property of steel, whereby it stays in a magnetized condition after removal of a magnetizing coil current.
How much of an inrush you get on any particular switch on is purely a matter of chance. In the worst case, the transformer is switched off at the peak of the magnetizing current, with the result that the core remains in a substantially magnetized condition. That is a matter of chance. Then, in the worst case, the supply is switched on at the beginning of a half cycle, of phase such that the magnetizing current magnetizes the core still further, in the direction in which it is already magnetized. If you are unlucky enough for both these events to be worst case, then there is a fair chance the core will saturate, after which the inrush current is limited only by the winding resistance. After a few cycles, the remanent magnetization disappears, and all is well.
Toroidally wound xformers are generally much worse that EI laminated types, because they use a strip-wound core that has zero air gap in the magnetic path, whereas conventional EI lamination types do have some magnetic air gap because the laminations are crudely butted together. The air gap helps to resist remanent magnetization in the first place, and also reduces saturation when inrush currents do occur, limiting the magnitude of the inrush.
I have a 2kVa Variac, toroidally wound as Variacs are, and every 4th or thereabouts time I turn it on, it trips it's breaker, as per the above, and as Ian has also experienced.
You may or may not have a problem with your 'huge' conventional lamination xformer. Sounds like it might be conservatively designed and thus less likely to saturate, so with luck you will be OK.