It can cause problems. It's best to uninstall the old driver before removing the old GPU. Use Display Driver Uninstaller. You can run it even if you have already removed the old GPU from the system.
I know personally, oculus software crashed when I had both amd and nvidia drivers installed. Probably harmless, still a good idea to remove old driver.