Alright, so the issue is finally solved. I had the Epic Store version installed which seemed to have problems with detecting controllers (e.g. PS4 controllers on PC), so it wasn't just me with this weird issue. I switched to the Steam version now and the issue has disappeared, no GPU load drops...