You'll need to be a detective, but your initial sense of what may be wrong is very likely on target. First, before you turn on your computer, remove the side of your computer case. Then, when you turn on your computer put your head next to the open side and see if you can identify from whence the hum is coming.
If you can't identify the source that way, a little trial and error may work. With your computer off and unplugged (make sure that the little light on the motherboard is off -- as long as your computer is plugged in, it will be receiving some electricity -- so unplug it and the light should go off after a few seconds). Then unplug one of the case fans. Plug in the computer and reboot. IF the hum is gone you know that the fan you unplugged is the culprit. Replacing case fans is inexpensive and easy -- although it is possible that the fan is simply really dirty. Try each case fan this way, one by one.
Another possible culprit is the fan atop your heat sink on your CPU. Chances are pretty good it's clogged with dust if you've never cleaned it in the four years you've had the computer. First, though, test the fan by unplugging it's power cord from the motherboard while the computer is unplugged. Then turn on the computer and reboot. If no noise, you've found the culprit. Quickly shut down the computer (within a minute or two at the longest) and replace the fan or clean it as explained in the next paragraph. Do not run the computer longer than a minute or two without the fan on the heat sink -- you don't want to let the CPU overheat (which could destroy it).
If it looks dirty, you might to try spraying it (when the computer is off and unplugged) with compressed air like Perfect Duster or Dustoff -- cans cost $3 to $5. Just be sure to follow directions and do not tilt the can while spraying (if you do liquid could come out in addition to the air). If the CPU heat sink's fan is clearly dirty and you're comfortable working inside the computer, you might be better off removing the fan and then cleaning it or replacing it. Some fans are easily removable from heat sinks; some are not. If this is stock heatsink/fan combo, you can probably remove the fan via four screws -- just very careful and don't let any screws or washers fall into the computer. Take the fan to your local computer store and get the same size to replace it -- making sure you have the same sort of connector to the motherboard as your original fan.
If the hummer isn't one of these fans, it could be the power supply or a hard drive. Frankly, if it's the power supply, I would think you could determine that via the hearing test described above. But if you can't tell, the only way to test the power supply is to try another one in its place. Since places like Best Buy and Fry's Electronics allow returns without penalty (as long as the item is resellable with full packaging etc.), buy a new power supply there that has at least the same wattage as your current one (it should say what it is on the side or in your computer's documentation). One possible hitch -- if your computer is a Dell, it might use only Dell's proprietary power supplies. Replacing a power supply is a pain, but not really difficult, just time consuming and exacting. You'll need to unscrew the 4 screws that hold it in (from the back of the computer case) and make sure that it doesn't fall into the case. First, you may want to unplug the power supply's various cables from the motherboard (you'll need to squeeze a small "handle" on the one or two cables that go to the motherboard) and from the components like the floppy drive, hard drive, CD/DVD drive, possibly some fans, maybe even the video card. Since your machine is four years old, it probably has an IDE hard drive, so you won't have to worry about Serial ATA cables. To play it safe, you might want to make drawing of where all the power cables plug in.
When replacing the power supply, reverse the process. You must first mount the new power supply first by screwing it in place, and then connect all the power cables. Be gentle, but firm, especially with the one or two cables for the motherboard since you do not want to bend the motherboard (which could break it). A lot of power supplies come with printed instructions. And don't buy some cheapo-cheapo power supply -- you get what you pay for. Stick to a name brand like CoolerMaster, OCZ, Thermaltake, X-Infinity. I'm cautious about Antec since I've had nothing but hassles with their overpriced power supplies -- so I haven't used them for at least 4 years. Maybe they've improved their quality control by now. You might want to go to newegg.com and read the user comments about a board before you buy it.
If it's not a fan or power supply, the hum could be coming from a CD or DVD drive or your hard drive. With the computer off and unplugged, unplug the power supply cable from your CD or DVD Drive. Then restart the computer. If the hum doesn't come back you've got your culprit (actually if you get the hum only when there's a CD or DVD in the drive, that's a sure sign that's the culprit). New recordable DVD/CD drives are available on sale for as little as $25. You should be able to score a good one for $25 to $45 online like at egghead.com or mwave.com. Replacing it isn't hard depending on your computer case. Follow the printed instructions that come with the drive. Be sure to unplug the computer, unplug your power cable and data cable. Removing the drive depends on your case -- there are too many variations. Most likely, though, it's screwed in with two screws on one side (and maybe two on the other side as well - in which case you'll have to remove the other side of the case too, pun intended). Slide the drive out through the front of the case. (Of course, replacing it can get funky if it's one of those propriety cool looking drives matched to the case in which case you'll need to get the replacement from your computer manufacturer.) Be sure to set the jumper on the back of the new drive to the same setting as the old drive (those wonderfully offensive terms "master" or "slave" -- why can't the IT industry use "primary" or "secondary" instead?).
If none of these is the culprit, it's likely to be your hard drive -- and it's likely that your hard drive could be on its last legs. Hard drives tend to last three to six years depending on use. Buy a new one (if the data cable is a wide cable, it's an IDE drive; if it's a narrow, say 1/2 inch wide cable, it's a serial ATA drive) that is the same type. If your current hard drive's capacity is less than 130 gigabytes, check with your computer's manufacturer to see if the computer's BIOS can support a larger hard drive. You may need to update the bios (the manufacturer will give you instructions online). If it can support a larger drive, go wild and get one. Hard drives of 160 GB size can generally be had for as little as $60 these days. Stick with name brands like Hitachi, Samsung, Western Digital, and Seagate.
You will want to clone your current hard drive to the new one. You'll need to install the blank new hard drive and format it. This can get complicated. It's very likely that the printed instructions that come with your new hard drive will walk you through the process. (Just make sure you buy a boxed new hard drive.) The drive will likely include a CD with software to install the new drive and clone the old one onto it. If not, software like Acronis True Image Home works great for cloning. There are also freeware programs available as well as other purchasable ones.
That pretty much covers all the possibilities. Put on your Sherlock Holmes cap and start detecting.
Hope I helped!
Luke