For only 1 GTX980? yes it is enough but only if the PSU is from high quality. If you have some cheap 650W PSU, I would have doubts.
BTW, do you have already the E3-1241V3 and GTX980? If not yet, I would recommend to change to E3-1231V3 and GTX980Ti or GTX970.
GTX980 is only 10% faster than GTX970 but cost almost 2x GTX970.
E3-1241V3 is more expensive than E3-1231V3 but the price difference does not bring much.
If you have them already, you are good to go.