There are plenty of GPU stress tests, and I personally use more than one. There's Unigine Valley, Unigine Heaven, MSI Kombustor, and 3DMark11 (the free version makes you do 3 tests (one for low-end systems, mid, and high) but you want to use the high-end test though you can only do that separately if you buy the retail version).
Some things that indicate instability:
-graphics card drivers crash (restart the system if they do)
-monitor goes blank (restart, obviously)
-there are glitches/artifacts on screen (run the stress test before you start and make sure you watch to see what's normal; always watch the stress test to check for these errors)
Here are some examples of artifacts:
http://i.imgur.com/Gu1x8.jpg
http://i740.photobucket.com/albums/xx48/darkhour_photos/Elemental_1276819590.jpg?t=1276825829
http://www.playtool.com/pages/artifacts/badgpu.jpg
Also, make sure you monitor your temps.