I'm no SMPS expert but I think the reason bigger inductors are needed for lower currents has to do with the fact that most switch mode controllers have a minimum pulse width (5% to 10%, for example.) That is, the switch is never allowed to have a 0% pulse width (completely off.)
Because of this, there is a minimum on time for the switch. During this time a bigger inductor will have a lower di/dt as compared to a smaller inductor. This allows for a lower current at the load. This is also why most controllers have a minimum load requirement. (Some will provide this minimum load internally.)
Having said that, there are controllers that will turn off when certain conditions exist. Linear Technology call this Burst Modetm. This might be worth looking into in your case.
There might also be a "happy medium" value for the inductor, in your case. I'd have to look but the inductor value would likely have to satisfy a couple of functions that depend on things such as minimum on time and minimum and maximum current. It might be a trick though, since a bigger inductor may not be able to keep up with a larger demand in current.