If you have an overclocked system that is unstable, reset the CPU (and GPU) to stock volts, multipliers, and clocks and see if the problem goes away. If it does, then your CPU or GPU was unstable and you need to either leave it at stock or re-find your overclock properly. If the problems do not go away, then it's something else such as Windows getting crudded up enough, a dying HDD corrupting data, a corrupted filesystem, the PSU not being up to task, or an overheating IC somewhere causing system instability and not your overclocked CPU or GPU giving you trouble.
Good overclocking is a real pain in the arse and takes a long time to do. You need to slowly find the limits of your memory and CPU, isolating each one and continually finding the stability from the bottom (stock) up, not the other way around. This will require numerous reboots and runs of Prime95 or Folding@Home and Memtest86+. (A GPU could use something like ATi Tray Tools to do a similar method.) Here's the rundown of how to do it:
Numero uno: find how much your RAM can be overclocked:
1. Start with stock clocks, SPD timings, and voltages and the memory strap that your RAM is rated at. If you are running RAM that has a higher frequency than your chipset will officially support, such as DDR2 >800 in a 965 board, then use the highest strap that your chipset offers.
2. Drop the CPU multiplier down a notch so that the CPU is not overclocked when you raise the FSB or LDT clock.
3. Raise the FSB/LDT clock in increments of a few MHz, such as 200 to 205 MHz. Make it pass a full test of Memtest86+. Make sure that the FSB/LDT is not raised too much as to overclock the CPU yet. If it does, then drop the CPU multiplier more so it isn't. Intel users might have trouble here, especially on the E6300 that can only drop 1 multiplier. We're trying to isolate the RAM's overclocking here and don't want to touch the CPU yet.
4. When the RAM fails Memtest86+, raise the RAM's core voltage one increment- 0.05V or whatever your board supports.
5. Repeat steps 3-4 until your RAM's voltage is at whatever the maximum rated voltage is. The clock rate just before the RAM fails Memtest86+ is your maximum at peak volts and stock timings. If you want to tighten up timings, repeat the process from Step 1 with the new timings. If you want to loosen them, loosen them and then repeat steps 3-4 until you find the maximum overclock at peak volts and your looser timings.
Nummer zwei: finding the CPU's maximum overclock
1. Start with the CPU at stock Vcore, multiplier, and FSB/LDT. Set your RAM at stock speed, max rated volts and whatever timings you wish to run at and found the max stable speed at with up above. If you have an Athlon 64, drop the HT multiplier such that you never exceed 1000 MHz on the HT bus. The HT bus speed is LDT bus clock * HT multi.
2. Increase the FSB/LDT by increments of X MHz at the stock multiplier and Vcore such that your CPU goes up by about 20 MHz. After each tweak, test with Prime95 or Folding@Home for an hour or so and see if it's stable. Also do not let full-load temps get within 5 C of the max temps of your CPU, preferably 10 C below. That works out to mean keep it under 60 C. If you get too hot, get a bigger heatsink or accept the highest overclock you can get without it getting too hot.
3. If it's stable, repeat step 2 until it is not or you reach the maximum speed that your RAM can handle. If you reach the max RAM speed, drop the RAM strap one notch. If the CPU is not stable, raise the Vcore by one increment without exceeding the highest Vcore you're willing to take. Ask around about how much Vcore your particular chip can handle for as long as you wish to run it.
4. Once your CPU isn't stable with peak Vcore and the clock speed you pick, back it off about 50 MHz core speed and that's your peak overclock.
This will take a long damn time but will yield a stable overclock if done as shown above. That's why I don't overclock. I undervolt my CPUs to reduce heat production and leave the value RAM I buy at stock speeds. Undervolting requires stability testing too- start at stock Vcore and work our way down one Vcore notch at a time, stability testing with Prime and FAH as you go until you fail the tests. Then raise the Vcore up one notch and it's stable. Finding the minimum undervolt is quicker than an overclock as the CPU and RAM speeds don't change and the CPU gets cooler and not hotter, but it's the same methodology. I did this with my 1.350 V X2 4200+ Manchester and it runs happily, cool, quiet, and rock-solid stable at 1.225 V.