Hey Ken,
Don't know if you are still interested in an answer since you made your post back in May, but for the sake of anyone who Googles the same question and finds this later...
It isn't surprising at all that your rig has trouble streaming. I have one with similar specs. I have a GTX 1060 and an i-6 6700k clocked at 4.0Ghz. The fact is, you are not going to be able to get much out of that hardware for encoding a stream if you are running demanding games, especially at 1080p60.
So, for the most optimal setup and best quality, you want to build a second PC dedicated for streaming.
First off, if you want to stream at 1080p, please use YouTube and not Twitch. I know Twitch is the go-to streaming site, but streaming at 1080p there usually returns unsatisfactory results. This is because they limit your bitrate to 6000 kbps, which is about half of what you need at minimum for 1080p.
And despite popular belief, you will probably do much better on YouTube growing a channel from scratch. I've done both, and YouTube is setup far better. They have an algorithm to promote you, they allow custom thumbnails, they transcode their videos, I could go on and on, but the point is, if you are going for best quality, go with YouTube. You will not be disappointed.
Second, you are going to need a capture card, a high end CPU, a lower end motherboard, and 8GB of RAM. A video card is not necessary, as you will not be using that PC for gaming.
You could use a video card to encode with NVENC, and that would be much cheaper than gettign a high end CPU for encoding with x264. However, it will show. On lower bitrates, NVENC will be too pixelated and blocky looking, compared to a much clearer and cleaner video encoded with x264. If your upload speed is 40000 KBPS+ (lucky you!) than NVENC will deliver about the same qualtiy as x264. But at lower upload speeds, you definitely want to use x264.
For capture card, I recommend the Elgato HD60 Pro. It is a capture card you put in a PCIe slot, an internal card. I have it and it is amazing! Absolutely zero latency, and doesn't have any audio desync issues or anything like that.
For CPU... well, it is pricey, but for the best quality, go with an AMD 1950X Threadripper. It can handle your 1080p on a slow preset, which will help produce the best quality possible.Without having to encode quickly, it can do a more thorough job, getting you the best quality looking picture. And it can do it quickly enough to keep your stream from lagging.
For a motherboard, you should grab the ASRock X399 Phantom Gaming 6. It is an expensive motherboard, coming in at $350, but it was the cheapest I could find that supported this chipset. It works perfectly fine, can do what is needed, etc. You'll need a cooler for the CPU. Not sure what to recommend here, since you could use anything from heatsink/fan combo like the 212 Evo, or a custom watercooling loop. Probably shoudl try and keep it on the cheap side, so a good heatsink/fan cooler should do. Probably $50 to $70.
For RAM, just get 8GB of DDR4 for RAM, whatever brand you like. Corsair, Kingston, whatever, should run you maybe $70 bucks.
Add a 250GB WD Blue 7200 RPM Hard Drive for $20, and a mid tower case for $50, and you got yourself one of the best streaming PC out there!
Hope that helps!
Facebook
Twitter
Instagram