So... I would first start by just using the NVIDIA encoder and copy directly to your C-drive (I believe it defaults to Documents->Videos).
It's slightly confusing at first how to use the settings but open Geforce Experience, In-Game overlay ON, click settings in In-Game overlay then: Video Capture.
Experiment with the BITRATE, 30FPS vs 60FPS encoding etc, and try a few games to capture a few minutes to see how it works (even the DESKTOP).
I just test with ALT + F9 to start then stop recording manually (desktop or game).
Once you confirm that's working fine you can move on to either figure out how to upload/broadcast in the same program or switch to another program but again CPU encoding is different. It's very demanding on the CPU whereas GPU encoding like NVidia does (or OBS can do) is just using the GPU encoder and is not very demanding it all. In fact, I started recording Skyrim SE with VSYNC OFF (so no FPS cap) and it didn't even drop the FPS at all. It stayed right at 78FPS when I left my character looking in one position.
*I've seen very little reason to use the CPU to encode. I've encoded at 2560x1440 with small text and it looks nearly perfect to me. In fact usually YOUTUBE ends up applying some compression so even if you can do better with a CPU encode usually nobody can tell the difference once they watch the compressed version.
Anyway, hope this helps.