~Oh no... now it all makes sense how you came to that conclusion; you left in too many variables and made some major miscalculations...
Lets start with the old chassis setup:
1)A single front intake and 1 rear exhaust? That's it? Was that actually the best fan setup for your hardware - did you test to be sure?
2)I'd bet the NH-D14 wasn't getting enough front intake - if the chassis does not flow enough cool air to the cooler, then the cooler reuses it's own heated exhaust air and cannot cool properly and resulting temps will be subpar.
If the rear exhaust fan isn't as strong as, or stronger than the
D14's NF-P14 fan, then it was just getting in the way. There are multiple Redux 120mm models; below the 1700rpm model would've been worse and would not have helped.
3)This is related to the 'overkill cooling' remark in my last post: An i7-860 isn't going to stress a high TDP cooler like the NH-D14.
Conclusion: A beefy cpu cooler that wasn't getting enough air, but it wasn't dealing with a hot chip either + whatever the gpu was at the time = thermal results were whatever they were at the time.
The new setup:
1)Is that the best fan setup and radiator location for your hardware - did you test?
2)8700K? Now an overclock on that would've been a more worthy opponent for the old NH-D14 than the i7-860 was.
3)Corsair H150i. This one's exactly what I was referring to with the noise optimized 360mm: they are worse than big air and 280mm performance.
The weaknesses of AIOs/CLCs is the cold plate and the pump speed, but the silence-focused H150i adds another one: weak fans, which is necessary to keep the silent profile.
You should've kept the D14...
4)Bigger chassis doesn't always translate to better, but hey, I shouldn't have to tell you that...
5)"i dont use any self adjusted fan curves or similar, i have all the fans and pump powered through the motherboard on default."

That's just adding another variable of error to testing...
6)"my temps on the i7 8700k got 2 degrees higher with the h150i and bigger case"
With all the mistakes you made, I can see why that happened...