Okay, so in the bios settings, you can go to frequency settings, usually it's space bar to change it, sometimes enter. It will probably be set at 20 as the ratio, you can just slowly push it up until you get to whatever you want.
I'm running 4.5 Ghz with a hyper212 evo, and slightly higher than stock voltage.
So you can probably get to 4.2ish on stock, 4.4-4.5 aftermarket non-water cooled. Then possibly 4.8 with water if you're lucky.
Remember that you need to run tests after overclocking, if you get really high temps, force the computer off, reboot, and clock down.