Looking to improve current build to increase recording quality

[Enaio]

Motherboard: ASRock 970 Extreme3
Processor: AMD FX-8320E Eight-Core
Hard Drive: WD Blue WD5000LPVX 500GB 5400 RPM 8MB Cache SATA 6.0Gb/s
Video Card: EVGA GeForce GTX 650 01G-P4-2650-KR 1GB 128-Bit GDDR5 PCI Express 3.0 x16 HDCP Ready Video Card
RAM: G.SKILL Ripjaws Series 8GB (4 x 4GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800)
Power Supply: CORSAIR CX series CX750 750W ATX12V / EPS12V
OS: Windows 7 Enterprise 64-bit

I am currently trying to record video of games (such as Rocket League, League of Legends, etc.) and have only recorded really choppy video. I've been looking around at other builds on here and comparing the specs to what I have (listed above). Trying my best to find the weak links of my build so I can improve my recording quality. The programs that I have used to attempt this are Camtasia and OBS. Camtasia worked well but the video was choppy. OBS tends to crash on me, which if needed, I can post the crash logs. Really appreciate any help possible. Thank you all!

Edit: I am currently attempting to record in 1280x720. Not sure if that might be an issue of any kind but want to put as much info as I can here just in case.

 

[Woody1999]

I would say it is your graphics card holding you back, along with the poor single threaded performance of your CPU. How much would you want to spend on upgrades?

Woody

 

[Enaio]

I would say it is your graphics card holding you back, along with the poor single threaded performance of your CPU. How much would you want to spend on upgrades?

Woody

I would say somewhere around $400, with a max of about $600. I wouldn't want to go higher than that. I know I could easily upgrade everything since this computer is about a year old and was built from parts that were on sale during Black Friday. If possible, give me a few ranges, such as a route for like $200, $400, and $600?

Edit: Did some more research. From what I can tell, for my current set up, if I were to upgrade both of those, I would want to go with the AMD FX-8350 and GTX 960. That would also be within the price range that I was looking for. Not sure if that would be the best solution, but should fit the cause. The only other thing that I would probably get, again from research, would be another HD, specifically a SSD, for rendering/storage of videos. If I am wrong, let me know. I am not rushing anything. Just don't want to lose interest as much as possible