I think I narrowed down the problem. I tried playing a low-demand game called Gemcraft - Chasing Shadows. It looks like my CPU usage on the Windows Task Manager is jumping up to ~90%. I knew my i5 3470 would be my bottleneck, but why would I have such a severe drop in performance for everything...