I'm not fully sure, but, I don't think the power supply should make any sound while starting. What kind of sound are you talking about, the fans spinning or some coil whine, or some other interference ? Which PSU model is it, by the way ? And, how old is the power supply unit ?
Some PSU models sometimes can make a mechanical click a few seconds after they start (eg in a switched-mode power supply), IMO. This is because of the mechanical relay (a switch), which is turned on in order to disconnect a component from the circuit, improving the PSU's efficiency and reducing amount of heat produced. That component taken out is only needed when the power supply starts, for the first few seconds.
Sometimes you may also hear a high pitch whine, which is generated inside transformer and inductors, inside the PSU. The noise is simply changes in currents in inductors causing magnetic fluctuations which results in a slight mechanical movement in and around them.
In SMPS, some changes in current or voltage will change the switching frequency or waveform, which can lead to a high frequency electrical noise/EMI