So, a socket 771 cpu in a 775 mobo with mod for it. Sure, it's been one of the best chipsets of its time, capable of many tweaks.
Well, there has never been a real
official support by intel for 771-to775 and as long as I know, people use bios microcodes along with the mechanical 4-cuts-thing mod to customize the overall system in the proper way. And a part of the people going this way still 've been complaining sometimes about unusual issues while using the 771+775-combos.
Maybe you need to see about if there is a more proper bios microcode, I don't know what exactly you used to mod the system.
Maybe you need the latest possible bios version, and after it to inject the proper bios microcode by cpu-z/manually, or whatever it is. At least this is what comes to my mind.
In fact, I had something similar, though easier previos month and what I did was to set in bios latencies to higher values, with things working on 6-6-6-18 normally, I also did put the sticks one-by one with some restarts with changing the ram orders - one stick just wanted to be closer to the cpu somehow. But I am just with a 775-cpu & ram & cpu are not overclocked, just stock.
Some links:
View: https://www.youtube.com/watch?v=2bxcSJjrzsA
https://www.delidded.com/lga-771-xeon-microcode/