Sorry I am late to the party, but I have some insights into overclocking the 1080ti
Zebarjadi's method is almost correct. The thing he (and most people) are missing is
GDDR5 errors and error correction. GDDR5 will correct many errors
before you see artifacting and still remain stable, but will result in a loss of performance.
Also GDDR5 has a cyclical nature to overclocking shown in the graph here:
View: http://i.imgur.com/mteJi6p.png
(People speculate that it is due to the memory timings being loosened)
Don't use Unigine. Use MSI Afterburner's
Kombustor which needs to be downloaded separately. (Use Unigine and others for final stability tests).
In Kombustor you can see how many artifacts are being generated via a counter on the screen (I assume these are errors being corrected). These are not normally visible on screen, until just before the point where memory overclocks will crash. You can also see the drop in frame rate in real time when artifacts manifest (even if they aren't visible as dots, lines, or other visible anomalies). Kombustor is also a static benchmark so the frame rate is constant due to a constant scene being rendered.
You can also use Kombuster to dial in memory overclocks 1mhz at a time in real time to find the top of the peak to get the most out of your OC. You can do this by activating the on screen display and watch what the frame rate counter is doing with each setting. Just make sure to run Kombuster for long enough until your card hits 80 degrees C before doing any fine tuning on your OC. When you think you hit a sweet spot, save it as a profile, so you can switch between potential candidates swiftly to see which one is best.
I found that for 4k, you will get more FPS from memory rather than clock frequency. I could reset the clock frequency to +0 on the fly and only drop 5 or so frames, where going back to stock settings of +0 and +0 would loose me 20 frames. Therefore I would dial in your memory OC first, then up the core clock. **caveat that water cooled cards can get a lot more core clock out of their OC.
**note that let Kombuster run for 10 seconds after each change so that the FPS stabilizes out.
My story:
So my 1080ti OCed to +50mhz clock, and +500mhz memory. I used Heavily modded Withcer 3 in 4k to find the best stability. However once I used Kombustor, I saw that thousands of artifacts were being generated every minute which were not visible, despite the OC being stable. Turns out, that artifacts started at about +350mhz, would be visible past +500mhz, and outright crash at about +550. Despite loosing +150mhz in my memory OC, I could dial in the peak memory frequency a lot easier and ended up with a
better overclock at +60mhz core and +343mhz for the memory. Also the lower memory frequency will use less power.