Yes you would notice a difference and I think it would be a worthwhile upgrade, however:
1) Wait for more availability of the RX 480 4gb, it will retail around $200 and outperform the 970.
2) A 500w power supply concerns me as a low quality one could not handle either setup properly. Could you tell us the specs of your PSU?