Electronic Contact cleaner like CRC brand is pure isopropyl alcohol in a spray can, and can be sprayed directly into a cpu socket if so desired (actually recommended for cleaning up paste stupidity), so is absolutely safe to clean Anything inside a pc.
Just make sure to not only unplug the psu, but push the power button after to help drain any residual power. Most times you'll hear an audible 'click' from the psu and all the various lights/leds will shut down.
The reason for the spray cans is simple. They don't contain enough kinetic energy to manipulate sensitive solder joints. Rags and fingers do, and the chances of bending a transistor or cap and breaking its seal is multiplied exponentially on a motherboard.
If the dirt is that bad, in your opinion or not, breakdown the entire pc, clean as required by rag or spray, let it dry, then rebuild.
Maintaining a decent cleaning schedule means a deep clean by full breakdown may or may not be necessary for a year or more. Much depending on the schedule, thoroughness of the scheduled cleanings, attention to details, environment.