There are a few low profile GTX 1060, but they are generally pretty expensive. Try to track down a used card if you can.
Should also be a few GTX1650 low profile.
Keep in mind that you will be pulling up to 75W for a GPU like the 1060 or 1650, and the standard PSU may be as low as 240W. With a 65W CPU, some fans, drives, etc, it may drastically reduce the lifespan of the PSU and make the computer a bit loud.
There are replacement HP PSUs that you can get, but at some point you might consider transplanting that motherboard to a new chassis and get an ATX to HP power adapter (you can find these on ebay/amazon/etc) So you can use full height GPUs and standard PSUs. (Or take the plunge and go with all non-OEM parts)
3rd gen Intel can easily be replaced with something like a first or second gen Ryzen. Start with a B450 board, and a few years down the road you can pick up a used Ryzen 5000 CPU to keep it contemporary. This will mean new DDR4 though, but it is quite inexpensive right now.