There are no CPUs worth the upgrade, not really, on that machine. If you are running a 64 bit version of Windows then you CAN upgrade the memory to 16GB but it's probably not going to be particularly useful since the rest of the system doesn't, and won't, run fast enough or support the proper instruction sets on some newer software to make it worth it.
You MIGHT be able to upgrade to a recent Gen graphics card like the GTX 1060, but you also might not. As has become apparent from a number of recent threads, some of these new graphics cards have firmware that is not supported by older legacy bios motherboards. It is becoming more frequent that a motherboard with UEFI bios is required on many newer cards. You will need to check with both your motherboard (HP) system manufacturer and the GPU card manufacturer to verify these compatibility issues as nobody aside from them really has those details and you won't find them on any product pages.
Honestly, your money would be better spend upgrading to a newer system, but if you CAN find a newer GPU card that is supported on your motherboard then in that case it would make sense to get the GPU card first, since you could immediately gain a performance advantage until you are able to upgrade the rest of the system.
Finding confirmation of a specific card that has been proven to work on your exact motherboard would be the best way to avoid any complications.
Also, the power supply is likely a pile of OEM junk, as mentioned above, so you'd best plan on replacing that with a higher grade replacement before or at the same time as any GPU card upgrade.