Im just looking for a better alternative for format factory.
maybe there's something better that has better hardware acceleration support.
That support comes from AMD in the form of the codec they supply, no software uses anything else. What other apps could do would be to use different settings for default which might produce better looking results. (or not)
Format factory is just a frontend for the completely free
ffmpeg so I would suggest you learn how to use that, it's a command line only tool so it's tough to get into but once you get the basics down it gets manageable, also support is great.
You can use this webpage to make the command line for whatever you want to do.
alfg.dev
Select the 264 or hevc options for nvidia and change the nvenc part with amd
(hevc_amf h264_amf)
You could also try handbrake, but for any software you should always play with bitrate and other settings to find the best for your use.
https://handbrake.fr/docs/en/1.5.0/technical/video-vce.html