First, make sure you're running the latest GPU driver, which will also have the latest version of GeForce Experience and ShadowPlay. If you're playing via a game browser service (Steam, Uplay, Origins) try disabling it's overlay.
Try turning off some of the telemetry settings in W10. I don't even have Game Mode enabled in it. MS claims it gives better performance in games, but I've seen lots of forum chat indicating the reverse is true. I have pretty much ALL telemetry settings in W10 disabled, and even use SpyBot Antibeacon to Immunize W10 from such telemetry services. It actually frees up about 1GB of RAM.
I also recommend disabling any services in ShadowPlay you don't use. For instance I disable broadcasting, highlights, and the notifications for them, as well as other things I don't use. The less processes ShadowPlay has running, the more RAM is freed, and the less chance for latency problems.
Make sure the drive you select to write the capture file is not heavily fragmented, and try to pick a fairly fast drive if you can. I have my WD Black 6TB HDD write the capture files via Instant Replay, and it is currently benching at 208 Mb/s. Also make sure your page file size in W10 is adequate and preferably on a fast drive.
You didn't list your spec, so I have no idea what games you're referring to, or whether the settings you're using are pushing your hardware too much. Since you are on an Nvidia GPU though, I highly recommend using Nvidia Inspector. It has profiles for most games and can help make them run smoother.
If a game's Vsync is horribly optimized and you get bad screen tear with it off, disable it and force Vsync in Nvidia Inspector. You can also select any FPS frame rate limit with it if that's better performance wise, and even use it's Power management mode to select Prefer maximum performance. This I prefer to setting the W10 power plan to High Performance, because it only enables High Performance power while the game is running. It's great for summer time when heat is high, or especially laptops.