As you can see, I keep my comp running for days at a time, so I decided to flush my RAM by just doing a restarted and of course that seemed to help drop it to the average usage of around 2.5GB usage.
I'm guessing the fact that I kept my computer running for days is what is going to contribute...