Runaway RAM usage (memory leak?)

[guilty_ligand]

When running certain applications, usually games, my memory usage slowly increases until there is none left to use. When it caps out, I receive this error message:

When I try to peek at memory usage however, usage instantly drops to normal levels.
http://prntscr.com/bh9bhh
This even happens even when I use task view or hover over task manager on the taskbar.

Does anyone know what might be causing this problem and a possible solution?
Thanks.

 

[kanewolf]

Do you have the swapfile disabled?

 

[guilty_ligand]

Do you have the swapfile disabled?

Virtual memory is set to 1920 MB by default. I have 8 GB of physical memory.