Is the new HDD detected?
You may need to enable "CSM" or "legacy" support to be able to boot from the USB.
There should be a prompt at post (F8, generally) to bring up a "one time" boot override menu, which is all you should need to boot from a USB/Disk to install your OS.
Which OS? If Windows 10, make sure you create a new bootable USB/Disk from here:
https://www.microsoft.com/en-ca/software-download/windows10
It'll be a 'newer' image, with updates etc. If you install from an older USB, you';ll then need to wait for Windows to install more updates.