Update your post to include full system hardware specs and OS.
Clarify: are you saying that the GPU continues to run after the computer is physically turned off?
And there should not be any need to remove the motherboard's battery - that battery is what helps the computer "remember" its' configuration settings.
If the power button is not immediately turning on the computer then the power button may have broken.
More information needed.