Computer Using All Of It's RAM Just From Idling?

[SuperShifty]

First time posting here so hello!
Here's my problem -
I have 8 Gigs of RAM installed (8.1gb according to Windows 8.1 ironically) and my computer normally only uses 17%-21% but recently it's been going up to 60%-98% just from idling, I did the math and everything listed on the task manager only adds up to 281 Megabytes...

Sometime's, for instance, I'll be playing a game and notice a performance hit (constant stuttering) after playing for a long period of time so I'll Alt+Tab out and see that I'm using almost all of my Memory or WAY more than said game calls for, I quit the game to see if my memory usage goes down and it doesn't at all. Other times I'll just have my computer on the desktop after booting up and then decide to run out and grab a bite to eat only to come back to all my RAM being used.
Anyone know what's going on?

 

[Tradesman1]

I'd check for both virus's and run a Malware scan to start

 

[SuperShifty]

I'd check for both virus's and run a Malware scan to start

Booted into Safe Mode and ran a full scan for both Malwarebytes and avast! and nothing came up.
I tried using HijackThis and TFC (Temp Files Cleaner) which definitely brought down my RAM usage at start but the problem still persists with it increasing and not going back down.

 

[Tradesman1]

How was DRAM use in SafeMode

 

[Silver Wolf]

Have you checked in Task Manager what is using all the memory? I found I had an issue with my 6 gb laptop with a fresh install of Windows 8/8.1 and it turns out it was the pre-installed Norton Anti-virus causing most of the problem (it is a Gateway, so the "fresh" install is a re-image with the factory image).