There are drawbacks to what you are looking to do...
1) Replacing the motherboard will mean the Operating System will also need to be replaced since it is tied directly to the motherboard.
2) You will also need a new CPU since yours is soldered onto the motherboard (good idea anyway) and possibly ram which add some more expense
3) The HP case likely does not support a standard µ-ATX size motherboard (9.6" x 9.6") since your current board is a bit smaller at 9.6" x 6.8" so a little more expense for a case.
Assuming your new PSU is of decent quality and has enough power, here's what you're looking at to get a decent upgrade (not including whichever graphics card you decide on)
PCPartPicker part list /
Price breakdown by...