sounds like paging to me, not enough vram, so its put elsewhere. then it requires loading and thats gonna be slow.
increase shared video memory to 2 GB (if your system has 8GB), try readyboosting if you dont have ssd (probably want a large readyboost 16-32GB), defrag hdd and set pagefile size manually (because if windows operates it and its split on a hdd, it can be slower than if it was one whole solid sector).