There are two possibilities: a background program with a memory leak or malware
It could be one of the other programs installed with your OS when it was first built. Monitoring programs can cause trouble like this, or some other bloatware that manufacturers install. It could also be a program from your ISP. Comcast forces an install of their horrible "remote assistance" software when their cable internet is first installed. That particular program causes major memory leaks like you describe and constant crashes with certain games. I fool Comcast with it by creating a base VM that will do the install, and then delete the VM. (I've moved while being a Comcast customer 4 times now, and each time it won't let me access the internet until I run that stupid installer. The first time I moved, the tech they sent over to do the install did the install on my main PC, and I had to totally reformat and reinstall the OS to get it stable again.)
To check this out, run msconfig and check out the startup tab. There is likely to be a long list. Use this to then reference task manager or Sysinternal's Process Explorer to find programs that take up a lot of memory. In task manager, it won't list the program's full memory foot print. Just look for one that looks bigger than it should be. Try to end the program and see how your memory recovers.
If it is malware, you'll need a removal program. The malware that would do this sort of thing can take the shape of a spam generator program, DDOS bot program, adware, or a fake anti-virus. Usually, the fake anti-virus programs are a lot more obtrusive, and will push alerts in your face constantly until you pay them. spam generators and DDOS bots can sit in the background and take up memory without you even seeing it. Try running MalwareBytes and see if it picks anything up. That should be able to clean most anything up. If your system won't let you install it, that is a prime indicator of malware. you may have to take out your hard drive and scan it from a different PC.