1080 is probably better. Titan X Maxwell has more VRAM but the architecture is older. Due to the newer architecture the 1080 will run games in general a lot more smoothly than a Titan X Maxwell.
To get into it, Titan X Maxwell has more CUDA cores by about 500 IIRC. However the 1080 has a much, much higher base clock which means the extra CUDA cores means pretty much nothing, since in games clock speed is everything. Titan X has 12GB of VRAM, but its memory speed is lower than the 1080 (7 Gbs vs 10 Gbs). Really that doesn't make too much of a difference in gaming, but as I said, faster clock speeds in games in better. The memory bandwidth between the two is negligible. The newer Pascal architecture means the 1080 will run more...