http://www.techspot.com/review/1006-the-witcher-3-benchmarks/page5.html
http://www.techspot.com/review/1174-nvidia-geforce-gtx-1080/page3.html
http://www.techspot.com/review/1006-the-witcher-3-benchmarks/page4.html
May get a little confusing, but here's how it stacks up:
-- at 1440p on Ultra, an i7-5960X gets 40FPS, while an i7-6700K gets 47 FPS
-- with the same i7-6700K, a 1080 gets 74 FPS, or about 57.44% more performance
-- based on the compared performance of your i5-2500K (93%) to the i7-5960X (98%), that translates to an expected performance of ~37-38FPS @ 1440p on Ultra with a 980, which would translate to ~59-60 FPS with the 1080.
-- For the 980, dropping the resolution from 1440p to 1080p increases performance by +40%, so
theoretically that should put you at ~83-84FPS @ 1080p on Ultra.
However, the key in your post was
"crowded areas". More items on the screen = more items to draw = more data for the CPU to process before the GPU can even render it. Make sure you check your FPS when you're not in crowded areas: if it's significantly higher, then that's the usual "crowded areas/multiplayer" issue that affects all games.
Also, have you tried adjusting the quality settings, & especially taking HairWorks off? Even the 980 lost 20FPS or so with HairWorks turned on (http://www.techspot.com/review/1006-the-witcher-3-benchmarks/page6.html), so try turning that off to start. If that doesn't help (or it's already turned off), try turning down the detail settings. If your FPS is unaffected by changing the quality settings, then your CPU might be having trouble keeping up with your GPU.