@hang-the-9 You have the patience of a saint.
@anishkalsi, Your power supply is absolutely horrible quality, whether it's new or not. We routinely use a power supply tier list, and yours is listed at the worse tier level, 5. You need to replace it with a quality unit, if you want your system to be stable. No way around that.
http://www.tomshardware.com/forum/id-2547993/psu-tier-list.html
Tier Five
Avoid IMMEDIATELY. These units are highly unsafe to use. No such protections added, very thin gauge wiring used, false advertising and too much to list. Reference to a higher tiered unit for a better, money saving and a much safer unit. For your safety's sake, please don't order or pick one up for use in your system. These units are a potential fire hazard and could even kill you, let alone your system.
Your CPU will remain a performance bottleneck, even if you were to one day purchase a faster graphics card. You can not fix the issue with overclocking because it'll only yield you only so much of a performance improvement. But more importantly overclocking will pull more power, from a power supply that's clearly already unstable in your system.
Sorry for repeating what's already been stated, but you've already received first class advice from people who are very knowledgeable about the subject. In addition to the Ultra's bad rating on the PSU tier list, I've personally come across these units. They are absolutely horrible. I have a sneaking suspicion that their wattage rating is actually peak and not continuous.