I had this happen, and the system actually fired up and ran fine, although the next day when the capacitor finally dried out it could no longer start. I replaced just that capacitor (it was on the 5v line) + the PSU has worked fine ever since and it's been 10 years!
It's nearly always a secondary capacitor that blows in a PSU and those are used to control ripple on the outputs. Without a working cap on one of the rails, the voltage on that one output would be extremely dirty and possibly even noisy enough to damage something, although usually if it's bad enough the PSU will shut itself down after a second as it will never get a power-good signal.
I actually prefer the bulging/exploding capacitors as then there is no question which...