It can't talk with the Mac's EFI. Meaning that MacOS cannot use the card even in generic mode until the drivers for it load. You need to install nVidia Web Drivers for Mac first. Using your old card. Then you can install the 980 Ti and boot.
You will get a black screen then until the drivers load during the boot process. Keep your old card around. You'll need it if you ever have to do any diagnostics or other boot options which require the screen to work pre-boot. You'll also need the old GPU after an OS update to install the appropriate nVidia Web Driver.
Note that nVidia Web Drivers only work up to macOS High Sierra. While nVidia has updated them for Mojave. Apple has decided not to sign them for whatever reason and nVidia won't release them without Apple's approval. So, Mac Pro users are either stuck with High Sierra using nVidia GPU or they have to switch to AMD GPU.
The drivers are available from nVidia. Download for your version of Mac OS X. You won't find anything for the GeForce 900 series. You look for
- Quadro
- Quadro Series
- Quadro K5000 for Mac
- Show all operating system (then select your version of OS X 10.10 minimum for 900 series)
- Optimal Driver for Enterprise (ODE)
- Your language
https://www.nvidia.com/Download/index.aspx
If you also want CUDA drivers for Mac. Find the latest version for your version of Mac OS X.
https://www.nvidia.com/object/mac-driver-archive.html
More detailed info
Guide to using unflashed Nvidia GTX 9XX GPUs in a classic Mac Pro. In this guide we will show you how to save money, time, and keep your workflow rock
create.pro
Mods, as the op is asking for help in getting an unsupported GPU working in an official Mac product not a hackintosh. I assume the above falls within acceptable forum posting rules.