I have this exact same problem with my gtx 680. the game is running fine, then suddenly dropping down to 10-20 fps. Afterburner shows that the gpu-usage spikes to 100, Fb usage goes up to around 90% from 30% while the gpu core clock dips down to 1/3.
It's really frustrating
Specs:
Windows 10...