System just reboots in Windows, is ok in BIOS, possibly motherboard?

[KCGuy2016]

First, my machine:
Gigabyte GA-H77M-D3H revision 1.0
24 gigs of assorted DDR3 (all worked before shenanigans)
I5-3570 processor
Nvidia GTX 1080 Ti
850 EVGA PSU
assorted solid state drives

Two weeks ago I decided to buy that GTX 1080 ti, and when I first put it and my new power supply in, nothing worked. The system just got to the POST screen, then stayed there. After verifying the card wasn't bad by placing it in another system, and after placing my old graphics cards in the motherboard and running it off that power supply, I think my new components are not broken.

So I updated the BIOS on my motherboard. I put in my old graphics card, went to Windows, then downloaded the Gigabyte Bios utility. When I used the Gigabyte Utility, it loaded the F14 bios.

It worked then, and for a whole weekend I was able to use my machine. I played DOOM on it, it ran like a champ. Then, Monday this week, it starts rebooting randomly in Windows.

Since then, whenever I try to run my PC, it reboots randomly when in windows. I can stick around in BIOS just fine, but right now I can't even re-install Windows.

So, is anyone else having trouble with their old Gigabyte Motherboards? And about the F14 bios, is it stable? The website lists it as Beta, but its over three years old now.

 

[Calvin7]

You should clear CMOS when changing memory.

Just installing memory that fits the slot is not recommended. Supported memory for that I5-3570 CPU is DDR3-1333, DDR3-1600. Using matched pairs that come in a single kit is recommended.

Memory manufacturers don't guarantee that mixing different memory modules not sold in a single kit will work.

Memory Clock, DRAM Timings, and DRAM voltage should be set according to the memory manufacturers specification and not left on [Auto] in BIOS.

 

[Faike]

You mentioned you put in a new graphics card. Can you get it to boot with the old card right now? If you can you probably just need to run DDU and uninstall all previous drivers in safemode then install your new card.
Also clear your CMOS like recommended above.
It seems like a driver thing to me though as not properly uninstalling previous drivers can cause this issue. And with windows updates rolling around all the time it wouldn't be weird for it to just randomly show up.