UPDATE 3.0
I have been fooling around with my system for months and still have not been totally thrilled by the performance but I have zeroed in on the problem. It was never about the GPU, yes the 1080 should be able to blow away this game at any settings, but the problem still lies in the game's optimization not the card, furthermore, it relies more heavily on the CPU than anything and most of the time you can attribute low fps to the CPU itself.
I believe I mentioned it in the original post but I upgraded from a 3770K slightly overclocked, to a slightly overclocked 6700K and got about 15-20 more fps out of it. I have since overclocked my CPU from 4.2GHz to 4.6GHz and got another 15-20fps. I also upgraded my monitor to an Acer Predator X34, overclocked it to 100Hz and used G-sync, which came with a whole other set of issues which I will detail shortly.
So here are my system specs and the method for getting a stable fps, some may not find it ideal but the game runs great, save for the most CPU heavy zones, such as the Nighthold raid, Broken Isles Dalaran and Stormwind.
For the most part the game cannot be run at max settings, no one has a CPU capable of running it at max and keep a stable FPS, it is impossible, but there are a few tricks that can be done to keep the game smooth 95% of the time.
Firstly my system specs:
https://pcpartpicker.com/list/hYP4Gf
I am able to run the game at 3440x1440, the game runs pretty poorly in Fullscreen mode, so I run it in Windowed (Borderless) mode, this also allows me to flawlessly use both of my monitors at the same time without windowing out. It doesn't work for all games but it is extremely convenient with the games it does work on.
Firstly, I am on Stormrage, an incredibly highly populated realm. I have had to turn off all AA not everyone may need to do this but it definitely helps, especially in highly populated servers.
Firstly scale back Render quality (under Advanced) down from 200% to 100%. There is absolutely no visible to the eye difference in quality but should drastically increase fps.
Second it is highly recommend to use G-sync, Freesync, or V-Sync. These all will limit the game to the monitor's refresh rate when used properly (I know nothing about Freesync) but this will keep the game extremely smooth and keep the GPU from working too hard and getting hot and loud.
G-sync stuttering and crashing fix: The day I got my new monitor and changed it from 60Hz to 100Hz I had horrible stuttering and the game would lock up my entire system. I went so far as to change out my PSU and completely reinstall the game as well as deleting all the temporary files to no avail. For a month I suffered with random lockups requiring a hard reset of my computer sometimes 5-6 times a day. Finally I narrowed down the problem to the new monitor, which really should have been my first guess. Looking into the game it has a problem with G-sync (not in itself but more on that soon). So I fiddled with the Nvidia Control Panel, and set the program refresh rate to use the programs settings, and turned G-sync off. The problem vanished. Since, then I had other issues with vertical sync not working in any game, turns out I was using G-sync incorrectly the whole time. The proper way is to turn G-sync on in Nvidia control panel, allow it to work with Fullscreen and Windowed Borderless mode, turn on Vertical sync through the NCP, and turn OFF V-sync in the game itself, viola, G-sync working flawlessly, every game working flawlessly and buttery smooth. So I realized the problem was the refresh rate in the game was not working properly and was somehow in conflict with NCP I am still unsure if this is a V-sync issue or a forced refresh rate issue, but either way the stuttering is gone and no more lock ups.
But back to the matter at hand.
Thirdly I had to scale down the Environment sliders, except the last one to 7 (down from 10). View distance at 7 is perfectly fine and does not overwork the GPU, The last slider being at 10 is also fine as long as you have the GPU for it. This might have been an actual GPU issue fixed by Blizzard.
I also scaled down Particle density to Good instead of High, which helps in fights with a lot of effects, which are CPU-bound, if you have a slower CPU you might consider lowering this further, especially in raids.
Since I am running in 3440x1440 at 100Hz I have had to scale back a little further than most, my graphics slider is at 7, but with a 60Hz 16:9 monitor you could easily run the game at 10 except the few listed above, though it results in almost no visible difference save the environmental sliders, which is why I run at 7. With a 144Hz monitor I expect you could easily run at 120-130 fps 95% of the time with the same settings I use.
WARNING: DO NOT mess with the graphics API, and whatever you do DON'T RESET AFTER YOU DO IT. I messed up my game by switching it and I had to delete my Interface/WTF folders, redownload ALL my add ons and redo all my interface settings.
With these settings my game stays fairly stable at 100fps save for the heaviest of fighting, where the CPU gets worked hard.
GPU utilization hovers between 35% and 75% depending on how much activity is going on.
GPU temp is almost never over 50C with my card and custom fan curve
GPU clock speed varies heavily from 1300MHz all the way to well over 2000MHz
CPU is the culprit with a single core utilization of up to 90% at times and almost never lower than 30%, 3 other cores cap out at roughly 50% and bottom out at around 20%, and the 4 threads are not used at all, only background usage around 10-15%. And this is on the 64-bit client, the 32-bit client is MUCH worse, and the game will ONLY use one core.
Although this is the case, single core temp on the heavily used core never exceeds 60C with my cooler and heavy overclock.
So in reality the problem with the game has never been the cards themselves, rather it is crappy optimization and the fact that the CPU is bottlenecking the GPU because the game is simply CPU-heavy rather than GPU-heavy in most cases, and the game is just not optimized well for multi-core usage.
Hopefully this helps someone relieve their frustration with this game and allow them to actually sit down and enjoy it. I like the way it looks and with the G-sync my frames can drop all the way to 60 before I notice any difference in fps, where before I would know if it dropped so much to 90. So this gives really smooth gameplay, save for heavy raids, and keeps me gaming smooth.
I have been fooling around with my system for months and still have not been totally thrilled by the performance but I have zeroed in on the problem. It was never about the GPU, yes the 1080 should be able to blow away this game at any settings, but the problem still lies in the game's optimization not the card, furthermore, it relies more heavily on the CPU than anything and most of the time you can attribute low fps to the CPU itself.
I believe I mentioned it in the original post but I upgraded from a 3770K slightly overclocked, to a slightly overclocked 6700K and got about 15-20 more fps out of it. I have since overclocked my CPU from 4.2GHz to 4.6GHz and got another 15-20fps. I also upgraded my monitor to an Acer Predator X34, overclocked it to 100Hz and used G-sync, which came with a whole other set of issues which I will detail shortly.
So here are my system specs and the method for getting a stable fps, some may not find it ideal but the game runs great, save for the most CPU heavy zones, such as the Nighthold raid, Broken Isles Dalaran and Stormwind.
For the most part the game cannot be run at max settings, no one has a CPU capable of running it at max and keep a stable FPS, it is impossible, but there are a few tricks that can be done to keep the game smooth 95% of the time.
Firstly my system specs:
https://pcpartpicker.com/list/hYP4Gf
I am able to run the game at 3440x1440, the game runs pretty poorly in Fullscreen mode, so I run it in Windowed (Borderless) mode, this also allows me to flawlessly use both of my monitors at the same time without windowing out. It doesn't work for all games but it is extremely convenient with the games it does work on.
Firstly, I am on Stormrage, an incredibly highly populated realm. I have had to turn off all AA not everyone may need to do this but it definitely helps, especially in highly populated servers.
Firstly scale back Render quality (under Advanced) down from 200% to 100%. There is absolutely no visible to the eye difference in quality but should drastically increase fps.
Second it is highly recommend to use G-sync, Freesync, or V-Sync. These all will limit the game to the monitor's refresh rate when used properly (I know nothing about Freesync) but this will keep the game extremely smooth and keep the GPU from working too hard and getting hot and loud.
G-sync stuttering and crashing fix: The day I got my new monitor and changed it from 60Hz to 100Hz I had horrible stuttering and the game would lock up my entire system. I went so far as to change out my PSU and completely reinstall the game as well as deleting all the temporary files to no avail. For a month I suffered with random lockups requiring a hard reset of my computer sometimes 5-6 times a day. Finally I narrowed down the problem to the new monitor, which really should have been my first guess. Looking into the game it has a problem with G-sync (not in itself but more on that soon). So I fiddled with the Nvidia Control Panel, and set the program refresh rate to use the programs settings, and turned G-sync off. The problem vanished. Since, then I had other issues with vertical sync not working in any game, turns out I was using G-sync incorrectly the whole time. The proper way is to turn G-sync on in Nvidia control panel, allow it to work with Fullscreen and Windowed Borderless mode, turn on Vertical sync through the NCP, and turn OFF V-sync in the game itself, viola, G-sync working flawlessly, every game working flawlessly and buttery smooth. So I realized the problem was the refresh rate in the game was not working properly and was somehow in conflict with NCP I am still unsure if this is a V-sync issue or a forced refresh rate issue, but either way the stuttering is gone and no more lock ups.
But back to the matter at hand.
Thirdly I had to scale down the Environment sliders, except the last one to 7 (down from 10). View distance at 7 is perfectly fine and does not overwork the GPU, The last slider being at 10 is also fine as long as you have the GPU for it. This might have been an actual GPU issue fixed by Blizzard.
I also scaled down Particle density to Good instead of High, which helps in fights with a lot of effects, which are CPU-bound, if you have a slower CPU you might consider lowering this further, especially in raids.
Since I am running in 3440x1440 at 100Hz I have had to scale back a little further than most, my graphics slider is at 7, but with a 60Hz 16:9 monitor you could easily run the game at 10 except the few listed above, though it results in almost no visible difference save the environmental sliders, which is why I run at 7. With a 144Hz monitor I expect you could easily run at 120-130 fps 95% of the time with the same settings I use.
WARNING: DO NOT mess with the graphics API, and whatever you do DON'T RESET AFTER YOU DO IT. I messed up my game by switching it and I had to delete my Interface/WTF folders, redownload ALL my add ons and redo all my interface settings.
With these settings my game stays fairly stable at 100fps save for the heaviest of fighting, where the CPU gets worked hard.
GPU utilization hovers between 35% and 75% depending on how much activity is going on.
GPU temp is almost never over 50C with my card and custom fan curve
GPU clock speed varies heavily from 1300MHz all the way to well over 2000MHz
CPU is the culprit with a single core utilization of up to 90% at times and almost never lower than 30%, 3 other cores cap out at roughly 50% and bottom out at around 20%, and the 4 threads are not used at all, only background usage around 10-15%. And this is on the 64-bit client, the 32-bit client is MUCH worse, and the game will ONLY use one core.
Although this is the case, single core temp on the heavily used core never exceeds 60C with my cooler and heavy overclock.
So in reality the problem with the game has never been the cards themselves, rather it is crappy optimization and the fact that the CPU is bottlenecking the GPU because the game is simply CPU-heavy rather than GPU-heavy in most cases, and the game is just not optimized well for multi-core usage.
Hopefully this helps someone relieve their frustration with this game and allow them to actually sit down and enjoy it. I like the way it looks and with the G-sync my frames can drop all the way to 60 before I notice any difference in fps, where before I would know if it dropped so much to 90. So this gives really smooth gameplay, save for heavy raids, and keeps me gaming smooth.