I believe I have it fixed "permanently" now, or I hope. I did the following:
Unplugged psu, removed SSD and insert Ubuntu install USB
Turn PC on, allow Ubuntu USB to load and shutdown (do not install Ubuntu
Put SSD back in PC, turn PC on, BIOS/POST appears, enter BOOT menu
Installed Ubuntu on...