As I understand your post the GTX1080 does not work in your H610 but does work in another older computer - correct?
More needs to be known about the problem.
Are there any error messages or codes, any beeps, any lit (or unlit LED's).
Did you use the same monitor and video cable?
= = = =
Update your post to include full system hardware specs and OS information.
Include PSU: make, model, wattage, age, condition (original to build, new, refurbished, used)?
Disk drive(s): make, model, capacity, how full?
List all attached peripherals.
Wired or wireless network?