Start with the basics like ohms law, basic transistor circuits there are a lot of websites and youtube videos for learning electronics which are a valuable resource. If you are into books the Art of Electronics is always praised around these forums but I cannot vouch for it as I have not read it myself.
As for as components go pick up some nice kits of transistors, resistors, LEDs, capacitors, diodes, breadboards and jumpers from ebay as they are quite cheap. Pick up some 555 timers and maybe a couple of op-amps and some 4xxx series logic as you need it. Micros can be a nice start coming from a PC background. Arduino is a nice micro environment to begin on for its simplicity and the clones are so cheap ( If your not one for buying genuine ones ). With all of these you should be able to make any simple circuits you find on the internet, beyond that what ever else you will need you will know you need it for a specific purpose so you can pick it up.
Kits like this are nice and cheap and come with the majority of things you may need.
http://www.ebay.co.uk/itm/ELECTRONIC-COMPONENTS-2000-PCS-KIT-FOR-BEGINNERS-STUDENTS-GCSE-ESENTIALS-/151417641781?hash=item234131db35:g:EQkAAOSw5VFWMqNXYou might not need mainy tools to begin with with basic breadboard circuits you can use usb for a powersupply. You may need some wire strippers, wire cutters and some nice pliers.I do recommend you get a multi meter and a soldering iron. A cheap one of each will do for the beginning just dont go near the mains with a cheap multimeter and actually dont go near the mains at all when you are still beginning stick to things like phone chargers as powersupplies. A bench powersupply is a valuable resource, the cheapest way to get one is to mod a shit PC power supply and there are thousands of videos out there on how to do that which would be a decent project for you to try.
Thats all I can think of off the top of my head, I hope you find this useful.