From the hardware side, you should not have an issue slotting one of those cards into that system.
However, if you are running the OS that you got it with, then you will be out of luck since no drivers will be available.
It's an older system and it may not work with newer video cards. Those cards would be a huge overkill in such an old system even if they would work. You are better off with something like a GTX 750 Ti, nothing faster is needed for your CPU.
It's a dual CPU system and it supports LGA771 Xeons so 2x 3.33GHz Quad cores is possible. The FBDIMMs are a mixed blessing though. But with modern GPUs shouldn't be too much of an issue. The PSU can't be swapped out due to the extra power cables required for the dual CPUs and FBDIMMs. So be sure you have enough GPU cables to run the card you want. Those came with different PSUs depending on what type and how many GPUs they came with. 2x6 pin = 1-8pin cable. Don't use a 6 to 8 pin adapter on the dell multirail PSU.
Here are some T5400 builds. CPU scores of almost 50% are common and GPU scores much higher. http://www.userbenchmark.com/System/Dell-Precision-WorkStation-T5400/3763
Here's a dual Xeon build scoring 74% multicore rating. http://www.userbenchmark.com/UserRun/4157687
You will notice no newer Radeons. they don't work with the Dell BIOS.
Since the new GPU can be moved forward to another computer later i'd say go for it. But the 1070 is what I would suggest. It's basically a budget Intel Skulltrail clone.
The T7400 guys (400FSB) like the 1080 http://www.userbenchmark.com/System/Dell-Precision-WorkStation-T7400/3398
I'm starting a T7400 project and was looking at the PSU issue. It looks like the T7400 1000W psu will work in the T5400. It has the strange 20 pin MB cable.
The 20 pin cable is 9-12V pins, and 10-GRD pins. So it might be possible to splice it onto some GPU cables on a modular aftermarket PSU.
On page 160 of the T7400 manual they give PSU pinouts But connector 24, and 19 seem to be reversed in the drawing. One is on the GPU cable, and the other is close to the PSU like a MB connector. It provides 2x6pin cables and 1x8 pin cable with the correct GPU pinout. One cable has a 6pin, and the other has a 6 and an 8 pin. I've heard rumors that you can't use both of those(on the same cable) at once. A study of the layout of the separate 12V. rails might reveal something there.It looks like the T7400 1000W is the way to go. It should support a single 300W GPU at the least,
On page 167 of the T5400 manual they have the pinouts there.
They each have a 3rd power connector that is different in each one, and has other than 12V. power to it. I think it's for optional storage solutions. if it's not used then it isn't a problem. I don't have my T7400 yet so I can't be more specific.
This expains the T5400s using GTX1070, and the T7400 using GTX1080. The PSU supports it. If you can resolve the 3rd power cable discrepancy there might be a solution.
I'm wanting to run an HD6970/HD6990 Trifire CFX with 2x CPUs so I'm definitely looking of some GPU cables. 3X8 pin and 1-6 pin. I may have to attempt the aftermarket PSU mod myself.
I think I've solved the mystery of the P3 connector on the T7400 PSU. They offered an Nvidia SLI dual GPU riser board that needed an 8 pin power cable with various voltages.
That would explain P3. I suppose the P3 in the T5400 serves some similar purpose.
Just to let you know I have installed the 1070 GPU to the T5400 Dell. I have used and 2x6 pin to 8 pin adaptor. It works fine. Currently I am using it to run some models and works perfect. The consumption is around 120 average when running the model with peaks more tha 151W according to the nvidia-smi appliation in Ubuntu.
So I hope that helps with you project on the T7400