Installing your OS on an SSD will give you the most perceived performance across the board (starting your computer, response of apps, loading times in games). I prefer an SSD large enough to hold my games as well, but you will gain plenty of performance from using a smaller SSD for your OS and putting your games on a hard drive if your budget doesn't allow for a larger SSD.
If you want higher frame rates or to be able to turn up the eye candy on games you can't run at max, then upgrade the graphics card. Using Tomshardware's hierarchy chart will be a good and easy way to help you decide what to get.
Everything else in your system is more than adequate - but I would upgrade the quantity of RAM to 8 GB. This may help with loading times in some games, or help you run more apps at the same time without slowing down your game's performance. 8GB is more or less the sweet spot, more is always better - but with diminishing returns.
(A note on upgrading memory - I prefer to replace the memory, not add to it, as problems can arise from mismatched memory, and it has been shown that less sticks is better than more, So, buy a 2x4 8GB kit rather than add a 2x2 4GB kit.)
Your decisions should be heavily influenced by your monitor. If you are running at 1080p 60 Hz, your demands aren't terribly taxing for modern hardware. If you plan to upgrade to a higher refresh rate or pixel density (say 120Hz, 1440p, or 4K) then a top end card will be much better. In the case of 4K I'd suggest waiting until Nvidia's Pascal comes out (should be this summer), or AMD's next gen (Polaris, should also be out in the summer) if you prefer AMD, as no current single-card solution will get you a steady 60 fps today @ 4K resolutions.
Speaking of monitors, the single best thing you can do to improve your experience is get one with a variable refresh rate technology, Nvidia's G-Sync or AMD's FreeSync. However, I personally won't be buying one until the exclusivity issues get ironed out; I'm strongly against buying any kind of monitor that locks me into using one brand of video card.
And, the more you know, the better choices you can make. Read reviews on sites like Toms, The Tech Report, and PCPerspective (there are of course other good sites as well). Depending on advice from forum posts isn't the best solution because we are pretty much all self-taught and have our prejudices.