OEM micro PCs are not designed for large upgrades but some are possible. You could replace the motherboard with a brand name one. That case should accept a standard microATX board. If you did this:
1. You would have to reinstall Windows.
2. Your OEM Windows license would no longer be usable as new motherboard = new computer in the eyes of Microsoft.
So the expenses would be: CPU, motherboard, new PSU as the HP one sucks, new Windows license, a new case if you want to ever have a big GPU, graphics card of course.. this is basically a new computer at that point. That is too much expense for an old system.
I would get a 1050 GTX first and see if it works. If it does, upgrade the CPU and go from there. If the machine won't boot with a 1000 series card, I'd consider looking at another PC or building one and then move the 1050 into the new machine.