First, I want to say I'm disheartened to learn of these problems. I love my GTX 980 Ti FTW, and was contemplating getting a GTX 1070 FTW for a second computer. I hope EVGA improves their VRM cooling in ACX 4.0.
LoreeSelmer :
I'm guessing that experienced builders and overclockers rely on their feel for acceptable room temperature and case air flow, and for the performance of known cooling solutions. Less experienced folks have a trial and error process on their hands, which as noted above can get pretty exciting if it gets away from you!
Is that pretty much how this exercise goes, or is there a first-order engineering solution that, for a given cooler and specs from manufacturer X, allows you to ballpark a maximum IC temperature from enclosure air temperature?
Well, people look at testbench measurements of different heatsinks w/ various CPUs (like what you'll find in the reviews section of this site). Most of the better measurements will indicate temperatures relative to ambient. Some will post absolute temps and state ambient. I've almost never seen anyone try to control for humidity.
So, you can either use that experimental data to guide your selection, or you can read reviews on amazon/newegg and read people's experiences in forums on this site & others. Either way, it's more an exercise of replicating what someone else has claimed to work. That, or just trying something that seems adequate and being willing to upgrade if it's not.
It really comes down to your goals. A small overclock is easily achievable on a big air cooler. For larger overclocks, you're looking at water cooling, delid, lapping, and even mounting it directly atop the die.
Good luck. You'll find much better info in here:
http://www.tomshardware.com/forum/forum-29.html