It is going to be long post, so please do not quote, as it will take a lot of space, lol.
What I typically see that Optiplex owners do not realize that Optiplex was designed ans business model - it is so crowded inside and it is so difficult to fit components inside, let me explain.
First of all, I don't see anywhere to be mentioned the form factor.
In order to better provide you with possible choices please identify your case form factor. This is Optiplex 755 family line picture, please name your form factor:
So, judging by the comments it is seems like you have MT, Mini Tower, if I am not correct, then the rest of information below is not applicable.
Mini Tower is not that "huge" as you think. You can not fit just any video card. Watch this video first, all Optiplexes share about the same hardware architecture
http://www.youtube.com/watch?v=tbVoe-4_oZ0
Unfortunately no card that is more than 7.5 inches long or 2 slots wide will PHYSICALLY FIT inside the case - it was found that only single bracket video card can fit due to BTX MOBO, where card internals facing upward towards PC internals, not ATX, where dual card design is blocking another PCI slot by facing downward.
If you prefer the card with beefier fan, then the length of the card must not exceed 7.5 inches, otherwise only "skinny" single bracket, single slot cards can physically fit.
Pictures below will prove my point.
View shows no room for dual bracket
Single bracket only, or you have to cut dual bracket in half
Here you can clearly see only single place for the bracket
Visual explanation
7.5 inches explained, also shown "obstacles"
This is the picture of actual "skinny" single slot video card (not sure which model)- perfect, "glove" fit.
So, I found 2 GDDR5 7750 cards (GDDR5 is better than GDDR3, even if GDD3 offers 2 GB RAM), which confirmed in reviews to fit Optiplex gx620, 745 and 755 cases (the difference between those models are different processors).
Skinny, flimsier in terms of construction, completely single slot XFX 7750
http://www.amazon.com/XFX-DisplayPort-PCI-Express-FX775AZNP4-FX-775A-ZNP4/dp/B007Z3T5JC
A bit beefier cooler, not as skinny, more durable in construction PowerColor AX7750
http://www.amazon.com/PowerColor-AX7750-1GBD5-DH-Video-Graphics/dp/B00775OC28/ref=sr_1_1?ie=UTF8&qid=1362783129&sr=8-1&keywords=AX7750+1GBD5-DH
I cannot say which one of those 2 is better, both are good and got less bad reviews related to structural build.
I saw recent recommendation regarding drivers "Changing to the just-released AMD catalyst 13.2 BETA driver improved performance noticeably in just about every game" posted 03/05/2013.
Since I heard that you want stronger video card, here are a few more options, which will physically fit inside.
So far I found only one 7770 video card which will fit inside - PowerColor AX7770
http://www.newegg.com/Product/Product.aspx?Item=N82E16814131477
From NVIDIA I see SPARKLE 700025 (SX650TI1024KD) GeForce GTX 650 Ti
http://www.newegg.com/Product/Product.aspx?Item=N82E16814187200
HIS H777FN1G Radeon HD 7770 very tight fit, almost touching heatsink
http://www.newegg.com/Product/Product.aspx?Item=N82E16814161433
I just check all newegg for single bracket again and noting more powerful!
So this is it.
7750 cards do not require power supply upgrade, 650 TI and 7770 are.
If you decide to go with power supply upgrade cards let me know, I need to check fit limitations, it is not that bad, but there are some problems as well.