[SOLVED] What encoder is best to use for an RX 5600?

[JoshTheDogBuildingNewPC]

Hello! I am using a Ryzen 5 3600 and an RX 5600 in my PC setup, with 16 GB of RAM, and I am trying to record some gameplay. I am getting choppy recordings using the x264 encoder, AMF H.264 encoder, and HEVC H.265 encoder. What encoder should I use to record high motion games in 1080p60, and what rate control? I already know about setting my monitor resolution to 60 Hz or 120 Hz so it scales correctly, and about process priority. I am still getting frame drops in recording (not during gameplay). My question is whether it is worth using the AMF or HEVC encoders or if I should just use the x264 encoder, and if what rate control I should use.

 

[Frag Maniac]

With my Nvidia setup I always use ShadowPlay, now just incorporated into GeForce Experience. Reason being it uses the H.264 architecture in the GPU to record, or cache then record. Both are far more performance efficient than recording via other methods, but the cached is pretty much zero lag, since you don't record while playing. The 20 min of cache limitation is rarely a problem for me, but I've even done videos much longer than that just by hitting Esc to pause the game, hitting the key to save the cached file, then waiting for it to write. If you don't let it write first, the next cached segment will look choppy after saved to file. If you minimize the game and go to the folder where the file is writing, you can actually see when it's done writing, as the video thumbnail will change appearance to verify that. All that's left other than that is some clever editing to make it look seamless. The only caveat is this type of capture doesn't not work for MP.

AMD for some time has offered a capture tool very similar to ShadowPlay.

 

[JoshTheDogBuildingNewPC]

With my Nvidia setup I always use ShadowPlay, now just incorporated into GeForce Experience. Reason being it uses the H.264 architecture in the GPU to record, or cache then record. Both are far more performance efficient than recording via other methods, but the cached is pretty much zero lag, since you don't record while playing. The 20 min of cache limitation is rarely a problem for me, but I've even done videos much longer than that just by hitting Esc to pause the game, hitting the key to save the cached file, then waiting for it to write. If you don't let it write first, the next cached segment will look choppy after saved to file. If you minimize the game and go to the folder where the file is writing, you can actually see when it's done writing, as the video thumbnail will change appearance to verify that. All that's left other than that is some clever editing to make it look seamless. The only caveat is this type of capture doesn't not work for MP.

AMD for some time has offered a capture tool very similar to ShadowPlay.

I do not want to use ShadowPlay, as I am using OBS for recordings longer than 1hr and sometimes more.

 

[Frag Maniac]

I do not want to use ShadowPlay, as I am using OBS for recordings longer than 1hr and sometimes more.

I wasn't suggesting you use ShadowPlay, I was saying AMD also has an encoder that uses the hardware in the GPU, which is accessible solely in the AMD software without a 3rd party tool, like ShadowPlay is. OBS supports that method if you choose the encoder that makes it work that way (AMF/VCE), however I've read some recent reports of it not being as well supported by AMD as Nvidia does NVENC. You should still try it if you haven't though.

I used AMD's first version of their built-in capture tool Re-Live, but it was pretty buggy back then. I've read that it's vastly improved since then though, and some even prefer it to ShadowPlay. It will depend whether you stream your content though. Most say streaming works best with OBS, but you have to be careful of stream settings, mainly output resolution, and bitrate, the latter of which requires good upload speed to use the recommended 5000 or 6000 for good quality.

The takeaway for me is, if you want best capture performance using an AMD GPU, the latest version of ReLive, using the latest beta GPU driver, is worth a shot. You may or may not experience the glitches he showed in the video, especially since your GPU is a much newer model, but as he demonstrated, they are easily bypassed anyway. I recall even way back when ReLive first launched, their sidebar was always buggy and best avoided.

The horrible software support was mainly what drove me away from AMD and toward Nvidia a few years ago. Sure, there are now and then problems with Nvidia drivers and ShadowPlay as well, but they don't happen nearly as often, are usually much smaller, and fixed much quicker. At worst you can roll back a driver or GeForce Experince version or two, as a temp workaround. With Adrenaline and ReLive, you're often stuck waiting for them to fix the problem, which can take a while. It literally took 7 MONTHS for them to finally fix the 5000 series GPU driver problems. Sure, AMD always offer great bang for buck on their hardware, but you definitely end up getting what you pay for.