Mass comes into it as well. Mass, surface area, airflow. The actual math itself can get quite complicated. Generally though, mass adds thermal inertia, which can be as important as heat dissipation, especially when dealing with a pulsating load, like a power transistor in an SMPS. As for surface area, we only use about 5 stock heatsink designs with increasing surface area and cost, so it's just a matter of balancing mass (that is, heatsink length), surface area, cost, and the needs of the layout.
The exception being in some of our really, really high-end PSUs, in which case yes the engineers use extensive math to optimize to get the most surface area in the smallest space with the lowest necessary airflow at an acceptable cost. But that's done as much or more by simulation tools than by hand-calculus.