Did you remove the graphics driver from the system before you inserted the card that was returned to you first of all.
If not use the program bellow, install it and run it.
It will remove any of the video drivers you had installed on the system, and any reference to them from the windows registry.
For a full clean uninstall.
http://www.guru3d.com/files-details/display-driver-uninstaller-download.html
Restart the PC. let windows detect the GTX 770 card again.
Then go to the Nvidia website, and driver support section.
Install the latest version of video driver for the model range of the card.
Once done restart the system again.
And test to see if you get the same result when launching a game or tabbing in and out of it with the windows key.
If it still produces the same problem then Rma the card as the other poster Advised.
I suspect if you did not uninstall the driver. it may fix the problem following the steps above and using the DDU program.
Try it and post back on the results, it would be nice to hear if it fixed the problem for you Techtony.