For me, I generally start increasing the memory slowly and leave the core where it is. When I start getting visual errors then I take note of the last memory speed that didn't produce errors. I use that number as my rough max memory setting.
Next, I reset my memory to default and work with the core. This will be different because when you get near the limit your system is likely to crash. It's best not to make the overclock permanent until you find your limits. If you do and the overclock is not stable your system might crash at or before reaching the desktop.
Now I have rough max overclock of GPU and memory. I then play with both values and keep benchmarking and stay below the max.
I have a Tbird at 1500mhz (150 * 10) but it's an SDRAM system so I don't get much benefit beyond 275/300 even though I've gone as high as 310/340 (with fans blowing directly on the top and bottom of my video card). 275/300 didn't need any extra cooling and in this heat wave my room temperature (basement) is about 26 C and my case temp is a loft 33 C. My card is also a Radeon 8500 64MB (250/275 default).
One more thing, it helps to know the speed of your memory before you start. My card has 3.3 ns memory which means it's specified to be 300 Mhz capable (at least the memory is). The quick way to determine the Mhz speed from number on the DRAMs is to divide the number 1 by the cycle rate (in nanoseconds) and multiply by 1000. For example, 1 / 3.3 * 1000 = 303.03 Mhz.
Knowing my memory was 300 mhz capable I jumped right to this level when overclocking. Normally I increase in small steps, like 2 mhz at a time.
When I settle in at max GPU and max memory speeds. I then open my case and prop up some case fans to see if more cooling helps and how much. If I get enough encouragement from this I will consider replacing the GPU cooler and add Ramsinks.
Right now my system seems to be the limiting factor so I won't bother modding my video card for additional cooling.
When I've settled in at the max overclock I generally just use regular game play to determine if I have stability. If I run into problems I reduce the overclock and loop some 3D game or benchmark for about an hour or so. An hour is about all I can stand because I don't like to walk away from my system while doing this kind of testing.
I've never done any voltage modding except that my motherboard allows AGP and I/O voltage adjustments. These haven't helped my level of overclock with the Radeon 8500. With my old Geforce2 GTS-V increasing the AGP voltage made it a lot more stable at max overclock.
<b>I have so many cookies I now have a FAT problem!</b>