Generally they say that if it's not broken, don't try to fix it.
If the new driver(s) have optimizations for the game YOU play or general optimizations for all games (direct x12 comes in mind), or specific fixes for security, stability, etc then yes.
If they have optimizations for game(s) that you don't play or add things that you don't need (streaming, etc) then certainly not.
I update 2-3 times per year for the reasons I mentioned above.