Update your post to include full system hardware specs and OS.
Include the PSU: make, model, wattage, age, condition?
If possible try an external, independently powered USB hub for your USB devices.
For the most part a failing PSU may not be able to provide the power being demanding by your system.
Especially at boot up when all devices are asking for power. The peak is too much for the PSU. Could be any or all of the provided voltages.