You probably want to upgrade the graphics card. 1050GTX would be idea, or a 1060GTX model with a very lower power consumption. Some things to consider are:
- Your CPU is just fine for gaming. Still very good.
- Your case won't fit a large card like a monster 1080 GTX.
- Your power supply means you effectively only have around 300 watts to work with. The system wasn't really designed to a gaming beast if it has a 745. Adding a higher power card means you need more electricity which, and running without enough will cause big issues.
Also installing the newer card will be different than on a normal PC. This is a good video showing a card upgrade to a 1060.
https://www.youtube.com/watch?v=2dpa4vb0PW8