Depends.
Blower style cooler: Doesn't matter. It was designed with tight spaces in mind.
Axial fan cooler: Give at least an inch or 2 or more for space. Psu shrouds impact gpu thermals because it can no longer pull air from below - just front and back.
Liquid cooler: doesn't matter.