I agree with nborcalc as the drop seems to be a bit too much. I would double check by running some bench tests against other systems with the same setup to make sure this is not some other issue causing the low performance as the 1080 should be taking the load. Overclocking will help and should be done on the 1600.
As to the high refresh rate gaming at 144Hz, the 8700 will provide better performance matched to the GTX 1080 and 1080p...no getting around that and it will help on the streaming and rendering with the 6 cores and 12 threads as opposed to the 8400 which is just 6 cores and 6 threads.
Do some testing first...then look at the options if not satisfied.