Exactly - which is why a manual for a heater-cooler appliance which only has a hot or cold switch would add the 5 minute warning.
Still, TECs are mechanically a little delicate - the Bismuth Telluride pellets are soldered to a ceramic substrate with some probably brittle, low melting point solder, and while they seem to have reasonably high compression strength, their shear strength is low. Shear stress would result from heat and expansion coefficient differences between the pellets and the substrate they're soldered to.
From what I can find, the thermal coefficient of expansion of the bismuth telluride pellets is directional, ranging from 13 to 22 ppm per degree C, whereas alumina, the typical substrate, is 7.2 ppm per degree. Not a huge difference, but with large temperature swings, large stresses could be created. Ferro Tec also said that modules with a large number of pellets will be less reliable, since these expansion coefficient problems are magnified with a large array of pellets. An aluminum nitride substrate has an even lower thermal coefficient of expansion, and would be worse.
So, a handful of TEC modules, each using a relatively small number of elements, soldered to small Alumina substrates, mounted in an arrangement that protects them from shear forces, and driven by a fairly slowly changing DC voltage, will probably be pretty reliable. This seems to accurately describe the small TECs that are used to drive laser diodes for optical communications, and those seem to be pretty reliable.