Always, always, always read the data sheet for the specific device you're using. Never just assume.
It's very common for a microcontroller which has high impedance, (usually) analogue input capability on a given pin, to have it default to that mode at power-up. This is completely correct and necessary if the pin is connected to circuitry that would react badly to being driven with a digital output, and safe otherwise. You can always add a pull-up or down resistor to your design if need be.
Unused pins should be set as outputs, or as inputs with a pulling resistor. The latter may be a better option if you anticipate adding additional inputs to a design in future.
A floating input may be harmless, but it will often cause excess current to be drawn by the CPU (think how a CMOS gate works, and what happens if the input is part way between fully high and fully low). Also bear in mind that the moment you probe a floating input with a scope probe, it'll immediately be pulled low, so you'll never see the voltage level it was really at (or whether it was, in fact, even oscillating).