I have tried reinstalling drivers, installing beta drivers, reverting to some really old 2018 drivers. It happens on multiple games: RUST, Squad, Counter-strike. GPU temps under load are always under 75 degrees celsius my fan speed is set to 90%. It also happens when I don't have my GPU on overlocked settings.
EDIT: When the driver crashes the screen also blinks for a couple of times and goes full black for around 5 seconds and after it has recovered the PC is very laggy until I close out of the game that crashed.