Vacuum drying is actually pretty efficient, since heat isn't wasted for heating up the clothes material or the dryer significantly above ambient. It's just that you still need to supply heat or pull a progressively stronger vacuum. Microwave-vacuum drying is getting popular, where microwaves are used to heat up the water (and avoiding the vacuum heat transfer issue).
If one kg of water evaporates, it removes 2.2MJ from the system. You still need to add those 2.2MJ to the system, but since it's running close to ambient, heat losses are going to be much smaller. Don't forget that the heat transfer rate is Q' is proportional to DeltaT. This eliminates the major contributor to energy losses --- heat leaving the system to ambient.