Well, because you don't have any problems outputing higher power with different input suggests the problem is not in power supply nor is it in the main amplifier. Couple of things could potentially cause this. I will assume that the amplifier uses some kind of MOSFETs but the story is similiar for BJTs(because you didn't mention anything about distortion if you have BJTs, I will assume that you are getting propper biasing in case it's A class). So there are couple of things that the signal passes through, when it is on USB it has to pass through DAC and some filtering so you might consider checking it. After it goes through DAC and filters it goes either to some communication chip that connects it with main preamplifier IC(sometimes integrated in main amplifier IC) or it goes directly to it. All in all those systems aren't easy to deal with, and they sometimes tend to be very complicated. But the fact that you are getting good output with one input and bad output at the other, again, suggests that the problem is somewhere before the amplifier circuit. Also make sure you check outputs of voltage regulators because they can make ICs malfunction if they are oscilating.