The matter is kind of a toss up, actually, if you don't install latest drivers, the newer games will be reduced. If you install said drivers that cripple performance, you're getting reduced performance in games.
It's been documented where drivers will cause people to migrate for newer hardware. What games are you playing? You should be fine as is.