It looks like you'll want a half-height or low-profile GFX card, like one of these: http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&IsNodeId=1&N=100007709%20600038363
you may need to swap out the mounting bracket, but all of those should have the proper size one included, if not...
it will automatically run at 1600 unless you tinker in the bios. should be able to remove the old ram, put in the new stuff, and boot right back up without issue.
download prime95 and gputest, they will stress test your cpu and gpu respectively. dont run them simultaneously, but see which one causes your issue to occur, and thats the part youll have to replace (or upgrade!)