Since you have HP prebuilt, it has proprietary MoBo and PSU. PC case too.
At this point, it would be wasteful to throw any money to it, since none of the standard hardware works with it.
Here's an image of your MoBo;
As you can probably make out, it isn't powered by 20/24-pin ATX socket, but instead it has 6-pin and 4-pin power sockets. <- These connectors require proprietary PSU, compatible with this specific MoBo. So, PSU upgrade would be out of the question, that is, unless you'll find bigger capacity PSU with same, proprietary power connectors.
Also, MoBo itself isn't standard from factor ATX either. It's too big to be mini-ITX and too wide to be micro-ATX. So, MoBo won't fit into normal PC cases either (well, it fits, but standoffs are in different spots and you can't fasten the MoBo to a PC case). MoBo layout is also reversed compared to ATX standard.
Since you have SFF PC case, only low-profile GPUs fit in there. Most GTX 1050 GPUs are full-height.
To get and idea of the low-profile i'm talking about, here's comparison between two cards:
So, don't waste your money on this proprietary prebuilt. Instead look for a new PC. And you can also go with small build, mini-ITX in a small PC case.
E.g this modern SFF build, in a similar HTPC PC case as your current build is;
PCPartPicker Part List
CPU: Intel Core i3-12100 3.3 GHz Quad-Core Processor
($142.78 @ B&H)
Motherboard: Asus PRIME H610I-PLUS D4-CSM Mini ITX LGA1700 Motherboard
($108.99 @ B&H)
Memory: Corsair Vengeance LPX 16 GB (2 x 8 GB) DDR4-3200 CL16 Memory
($44.99 @ Amazon)
Storage: Samsung 970 Evo Plus 500 GB M.2-2280 PCIe 3.0 X4 NVME Solid State Drive
($59.99 @ Amazon)
Video Card: Zotac AMP Core GeForce GTX 1650 G6 4 GB Video Card
($169.99 @ Newegg Sellers)
Case: Fractal Design Node 202 HTPC Case
($98.98 @ Newegg)
Power Supply: Corsair SF600 600 W 80+ Gold Certified Fully Modular SFX Power Supply
($124.99 @ Amazon)
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2023-02-04 11:50 EST-0500
All what is needed are included. But to keep the price low, i put in 500 GB OS drive, which is M.2 drive and lives on your MoBo. So, no additional space is taken up by 2.5"/3.5" drives. GPU is GTX 1650 and it's the best 75W GPU currently out there. PSU is SFX and it's the smallest capacity unit (600W) to choose from. CPU cooler is included with CPU and PC case is personal choice. If you don't like the Node 202, you can switch it out.
Note: if you have money money, you can easily replace the M.2 NVMe SSD with bigger one, go with more RAM (currently put in 16GB DDR4) and better GPU.
Just for comparison, your current build as Baseline, my suggestion as Alternative;
Userbenchmark PC Build Comparison
Baseline Bench: Game 15%, Desk 75%, Work 13%
CPU: Intel Core i7-2700K
GPU: Nvidia GT 1030
SSD: Samsung 860 Evo 2TB
RAM: G.SKILL RipjawsX DDR3 1600 C10 4x8GB
Alternative Bench: Game 48%, Desk 103%, Work 42%
CPU: Intel Core i3-12100
GPU: Nvidia GTX 1650
SSD: Samsung 970 Evo Plus NVMe PCIe M.2 500GB
RAM: Corsair Vengeance LPX DDR4 3200 C16 2x8GB
Note: Wasn't sure what SSD and RAM you have, so put in those that i think you might have (860 Evo and 1600 Mhz DDR3).