Actually, that form factor is not a problem. There are some great video cards that can fit in that space. The price and the fact that you will be using an i3 CPU are the limiting factors.
From my perspective, I would recommend the ASUS GTX960-MOC-2GD5 card for $204 (after $10 MIR):
http://pcpartpicker.com/part/asus-video-card-gtx960moc2gd5
It is an excellent Geforce GTX 960 card, and is just over 170mm long. The NVidia cards will use less power and normally make less noise than the AMD ones, so I tend to favor them. If you plan on upgrading CPUs in the future, an i5 would be much better for gaming than an i3 (4 real CPU cores in the i5). With a 960 GPU, you will usually not be CPU limited for current games.
The best ITX value system...