There is nothing to be hesitant if the card fits in the slot. The only issue is that you will lose graphical power, to put it simply and not go into detail.
For example using a 2.0 instead of a 3.0 might cost you up to 20% performance overall.
So if you don't want to upgrade your motherboard and RAM (probably need new RAM too) expect to lose a little performance. But adding a new GPU will still, generally, provide you the best boost in performance, unless you upgrade very little like, one generation to the next, then don't expect a huge boost unless the cards are also different tiers. The higher the last 2 digits are, the higher the tier usually. Like *60 *70 **80 at the end of the card. First 2 digits indicate the generation.
(unless it's AMD 300 400 500 then they are basically the same GPU with slightly better berformance)