Was the lower CPU usage with the GTX 1080 as well?
Check that you don't have some malware running in the background stealing clock cycles;
Malwarebytes.
Check Task Manager to see what other programs are running, or...
Check your TRAY to be sure there's no background program running that you weren't aware of.