In the US:
Resistors, caps (except electrolytics): Ebay sellers, buy the collections that have say 25 of each value
Connectors: ebay - straight from china (that's by the way where all the supplier get connectors these days)
Wire, cables: ebay - US suppliers, there are only a couple of good ones
PCB: Ebay has a few US suppliers that are well known and cheap, you can't miss them
Electrolytic Caps: don't use old ones, and don't get nobrand ones from ebay. Electronic Goldmine or Mouser are your best bet. Sometimes digi-key has them cheaper than mouser, but that's extremely rare.
IC's: Mouser - end of story. There is such huge variety in IC's and desoldering them is such a hassle, that you almost never find what you want. Also the tech moves so fast, that anything you scrounge up, is really not on par with any new product. On the other hand, Mouser rarely carries e.g. LT parts, so you will end up paying the premium. A last resort is Element 14. They charge crazy shipping and tax to the US, but they have some really exotic parts that you just cannot find here.
Transistors, diodes, etc. and other discrete parts: Mouser in small numbers. Digi-key in large numbers (though you probably never need more than 100 of one type just for experimenting)
LCD's: ebay US suppliers. The stuff coming in from china breaks all too often on the way, and you really won't be able to return faulty ones.
LED's Ebay - buy the value packs with 100 different color LED's for 3-4 bucks!
Chassis and boxes: let me know if you find a good supplier. The most common ones (Hammond, etc.) are very expensive, and with the eBay stuff you pay 20 bucks for the box and 60 for shipping!
Transformers (Toroidal) Antek Inc
Transformer EI and smaller VAs: look on eBay, but don't waste too much time - if it's not there for cheap, just get it from Mouser. They have a lot of Variety, though not as much as Digi-key, but they still are cheaper, if they have it. If not - Digi-key will have anything you need.
IMPORTANT: if you ever take something apart for parts, keep the transformer. They are almost always good (and easy to test) and you can never have enough variety. They don't go bad for hundreds of years, and by that time, we probably won't be using electric power anymore
So why don't I buy from All Electronics, or radio shack, or other websites? I actually do, rarely and only for oddball components - but the packaging is always suspect, and the parts are rarely good brands. Hope this helps. If someone disagrees, Let me know, and you might save me a buck or two.