I would definitely go with the 5700 XT for the best bang for buck overall performance. However though, if you are dead set on Nvidia, then of those two options I would opt for the 1080.
However, if you can increase your budget to about $400 USD, you can probably find a used 1080 Ti