YoonLee :
http://support.hp.com/us-en/product/HP-Pavilion-Slimline-s5600-Desktop-PC-series/4269980/model/4300362/document/c02479775/
Just games in general. (New games, old games, etc.)
TLDR: First buy a new psu. Second buy a new gpu. Third buy a new cpu and thermal paste.
All right, with that slimline computer, your options are a bit limited. First, you'll need to buy a new power supply. My family has an hp slimline computer and I was only able to do minimal upgrades within a reasonable budget. You will need to buy a TFX form factor psu. Measure where your current psu is to make sure it fits a TFX form factor (or see if your current one is labeled as such). My family's slimline is from the same generation of HP slimlines (it looks exactly the same) so I'm pretty sure that's the right psu form factor.
Here is a 300W one made by Seasonic from newegg:
http://www.newegg.com/Product/Product.aspx?Item=N82E16817151090
There is also a 350W Seasonic version:
http://www.newegg.com/Product/Product.aspx?Item=N82E16817151114
Here's an HP replacement version that's 400W (but its not that great - I would go with the seasonic):
http://www.newegg.com/Product/Product.aspx?Item=9SIA6J12K10869
Then you should get a gtx 750ti for your gpu. I wouldn't recommend anything more powerful because you won't have a psu that's powerful enough. Make sure you take measurements of the inside of your case where your gpu is going to go. You need to buy one that's small enough.
For your cpu, you might need to buy one off of eBay. It's hard to find AM3 socket cpu's nowadays for sale as new. You're probably looking at buying an AMD Phenom II X4 900 series.
Here's an AMD Phenom II X4 955 on eBay:
http://www.ebay.com/itm/AMD-Phenom-II-X4-955-BE-3-2GHz-AM3-6MB-Quad-Core-HDZ955FBK4DGM-CPU-Processor-US-/271773258624?pt=LH_DefaultDomain_0&hash=item3f46f2d380
If that's too much, then you can try going for an AMD Athlon II X4 600 series, perferably the AMD Athlon II X4 640.
Your current fan/heatsink should probably work for now, but you'll still need to buy some thermal paste. You will need to make sure to dust/clean your heatsink and fan often enough to make sure dust doesn't build up. Those slimline computers don't have good ventilation so dust can be a big problem down the road.
Cost of new psu: $34-$68
Cost of new gpu: $130
Cost of new cpu: $40-$80
Cost of thermal paste: $5-10