You need to OC in small increments first, and check for the system/GPU stability..
Try to not increase the frequency higher than 5% on both the core and memory clocks. Example: suppose if your card runs at 1500 MHz, then you shouldn't increase the frequency any higher than 25-50 MHz. If the system is showing artifacts, polygons, and/or it freezes, then you need to dial down the speed in small increments, like e.g. 25 MHz.
Start with 25MHz. You need to observe at which frequency the GPU gives you a stable result. OC can be risky, if done incorrectly, so be very careful when overclocking.
For the system stability, you need to look for artifacts, polygons, or a system FREEZE/hang after applying the overclocking settings. If you don't observe any of these, then you are safe.
Voltage and heat are the things you want to watch out for. Only increase voltage if your temperature values are safe at FULL load. All components have different voltage tolerances.Voltage increases always should be tiny.
The basics of overclocking:
Test (Prime95 or Heaven, or any other benchmark for ~5-10 mins)
Repeat until it fails.
Lower Mhz OR increase voltage
Once you have your maximum stable core overclock figured out, you can do the same for the Memory. Now, memory OC's generally don't provide that much FPS gains, so it's up to you if you even want to tweak this setting as well. You use the exact same process as your CORE overclock, and take it as high as you can.
Move the Core slider up 25-50 Mhz. Run Heaven for 5-10 mins, watch your temps, etc. It's the same as CPU overclocking. Once you get a driver crash, system freeze, or see visual anomalies, then dial it back to 10-20 Mhz, and then try again.