From a gaming perspective, that is a waste of time, money and effort. You really need to upgrade the CPU as well even if it is only a modern low performance CPU, you will get a pretty large performance increase. Many popular games use 2 cores so having a single core CPU is a bottleneck. If you intend on playing the Sims 3 or Cities XL 2011/2012/2013 then what you propose is doable since those games only use one core.
In order to keep costs low, I will limit the cost of the CPU to $50 before shipping and taxes. Based on what's available at Newegg the best CPUs are the:
1. AMD A4-4000 Richland 3.2GHz Socket FM2 65W Dual-Core Desktop Processor = $46 + Free shipping
http://www.newegg.com/Product/Product.aspx?Item=N82E16819113343
2. Intel Celeron G1610 Ivy Bridge 2.6GHz LGA 1155 55W Dual-Core = $50 + Free shipping
http://www.newegg.com/Product/Product.aspx?Item=N82E16819116889
Even though the AMD A4 APU is clocked higher than the Intel Celeron, both should more or less be of equal performance. The Celeron may be slightly faster though. Since Intel released their Haswell CPU, socket 1155 (Sandy Bridge / Ivy Bridge) is basically dead. No new CPUs will be released for that socket. However, the same can also be said of socket FM2 used by AMD's Trinity and Richland APUs. Their next generation APU (Kaveri) will require a socket FM2+ motherboard. While it can physically fit in socket FM2, it lacks some circuitry required by Kaveri; possible a voltage regulator. For all intents and purposes socket 1155 and FM2 are dead.
Both A4 and Celeron cannot be overclocked. Both have integrated graphic cores. While the A4 has better integrated graphics compared to the Celeron, you are installing your own card. Therefore this point is moot. The only other thing is power consumption, Intel CPUs generally are more power efficient than AMD's APUs and CPUs.
The future upgrade path for the Celeron is an i5-4670 which basically Intel's most powerful quad core i5 CPU that cannot be overclocked. The best socket FM2 APU is the A10-6800k. While it can be overclocked it would still be slower than the i5-4670 and will also consume more power as well. While the i5-4670 is more powerful than the A10-6880 it is also more expensive. The good thing is that if you eventually want more power than an A10-6800k can provide, then going down the Intel route is the better option.