These questions always tend to be "loaded"... If you are just talking about CPU cooling then heck, throw an AIO on it and be done with it. AIO's have come down in price recently and will produce the maximum possible overclock. I've seen R7 1700X hit 4Ghz and better on AIO cooling, of course that largely depends on the silicon lottery.
However you can't just throw on a custom air or water cooler and overclock your rig to the wall without knowing the rest of your hardware. VRMs are very important and if they are not adequately cooled it really doesn't matter how good your CPU is cooled, you will still be thermal limited and may even cause damage to your system by overclocking. As silly as it may sound sometimes the stock cooler is better for your system depending on the motherboard and VRM cooling as with the stock "down blow" cooler air is moving across your VRMs, RAM, ect... The custom tower air coolers and AIO coolers mover air differently and the VRMs and RAM get hardly no air flow at all.
You have to take into consideration what kind of motherboard do you have, what kind of heatsinks do you have on the VRMs, how many case fans do you have moving cool air through your system and how are those case fans configured (ie positive or negative pressure). If you just throw a custom cooler on your processor with a motherboard that has crappy VRM heatsinks and you are running two or three case fans that aren't producing adequate positive pressure and air flow then you are dooming your VRMs to thermal throttle or worst case burn up. You could damage your motherboard or RAM beyond repair by overclocking your processor without giving a thought to the VRM cooling.
My recommendation to anyone looking to overclock is first have adequate case cooling. I recommend at least 5 case fans (with case support of course) to anyone serious about overclocking. You also want at least 3 of those fans to be intake so you have positive pressure (ie more cool air coming in than hot air going out). Personally for air cooling I like two intake fans on top blowing cold air onto the VRMs and RAM, two front intake fans bringing in air directly in front of the GPU, and an exhaust fan in the rear blowing hot air out of the case. For liquid AIO systems I like a front mounted rad with 4 fans in a push pull configuration expelling hot air out of the system, and then the top fans and rear fan all set to intake cold air. On an AIO system if you have the ability for bottom mounted fans have them set as intakes as well. Doing testing on many systems I've found this configuration to be best with liquid cooled systems as all the important parts have cold air blowing across them.
Regardless of what you decide to go for with aftermarket cooling make sure you have adequate case cooling and air flow, especially over the VRMs, don't overlook that or you will be buying a new motherboard and possibly RAM.