The time to replace it is when you are no longer happy with its performance in the games that you play. Until then, you are happy. Why change anything?
Sometimes, even once you reach the point where you start wondering if it isn't time to upgrade, there is nothing at a reasonable price that would be worth the money to do the upgrade. So then it is a waiting game to see what comes out in the next wave of video cards.
Right now, both AMD and Nvidia have been stuck at the 28nm node for nearly 4 years. Both had planned on moving to 20nm several years ago, but TMSC, the fab both companies rely on to make their GPU's, never released the technology on the 20nm node to allow for GPU's to be very high performance.
Now TMSC has created the high performance technology that both companies need on the 14nm/16nm node to create high performance GPU's. Rumors say both companies have already taped out their next GPU. Now the process of making first run silicon starts, and then months of testing, evaluating the silicon, and the beginning of new driver code begins. Sometimes they find issues with the silicon, and have to make modifications, and start all over with a new tape out. Eventually, they have silicon they can sell, and they go into production.
At this point, a good guess on when we might see some of these new smaller but faster chips is second quarter next year.
So when to upgrade is when you want something faster, and there is a faster GPU available at a price you are willing to pay.