Question Encoding video with NVENC while gaming crashes my computer

[physicsme]

I use my gaming PC to encode videos to H265 with FFMPEG and NVENC very often. Sometimes I just forget it's encoding and start a video game. When the game is not very graphically demanding it's fine, but more demanding games cause a black screen. I think it's just the graphics card that's knocked out not the whole system because there's still audio. When I reboot, sometimes there are artifacts on the screen and it won't get into windows. Rebooting the system several times usually brings it back. There's this one time, after a couple of reboots I could get into windows but the display resolution got locked to something like 480P and when I checked Device manager, it says the device had a problem and windows stopped it. I sent the card out for RMA but they found nothing wrong with it. I put it back into my system, and it can boot up, resolution normal, games, furmark, no problem. But when I game while encoding the system would crash again. I know I can easily resolve this by simply not doing the two things at once, but sometimes I just don't remember.
Is there a way out of this without crashing the system and potentially damaging the graphics card?

The games that cause crashes:
Starcraft II (usually plays fine in CO-OP mode, but crashes when playing cutscenes in single player)
Doom Eternal (crashes immediately at startup, right about when the logos are showing up)
Eve Online (crashes a few minutes into gameplay)


The ffmpeg command I used for encoding videos:

ffmpeg  -hwaccel_output_format cuda -i INPUT -pix_fmt p010le -map 0:V -map 0:a -c:v hevc_nvenc -preset slow -rc constqp -qp 23  -c:a copy OUTPUT

My system spec:
Ryzen 7 3700x
HyperX DDR4 3000 8GBx2
Gigabyte B450 I aorus pro wifi
EVGA 2080super SC
HP SSD EX950 1TB
Silverstone SX650-G SFX PSU
1440P 155hz monitor

 

[ötzi]

Don't do any encoding while gaming? I think most cards, even faster ones will run into troubles when doing encoding and gaming simultaneously without setting boundaries for the tasks at hand. Which software do you use for the encoding?

 

[physicsme]

Don't do any encoding while gaming? I think most cards, even faster ones will run into troubles when doing encoding and gaming simultaneously without setting boundaries for the tasks at hand. Which software do you use for the encoding?

I was using ffmpeg.

 

[TerryLaze]

Yeah artifacts and rebooting into safe mode sounds like the card is overheating or is otherwise going above specs.
You could put a frame limiter on your PC, using rtss you can lock FPS to 60 for anything that runs on windows. That way there might be enough headroom for the encoding without crashing the system.

 

[ötzi]

I was using ffmpeg.

Could be a conflict in resources since you're using the CUDA Toolkit.
Did you try to profile ffmpeg to see what's up? You can use Nvidia Visual Profiler for this, it's part of the CUDA SDK.