It is indeed the HDD LED
It is connected from your case to the motherboard via the front panel connector
Knowing when your computer's hard drive is being accessed is helpful so you can avoid pulling the battery or unplugging the computer while the operating system is still accessing files on the drive, a mistake that can cause corruption of important files.
That is normal behavior if you are idle and aren't doing any writing activity to your disk. If a disk were always in usage all the time, wouldn't that wear out the disk faster and not give you a chance to power off your computer?
But I wouldn't do it on a Dell Optiplex, as it is a pre built. Pre builts usually limit customization.
Secondly, GPU prices are high. Even in my place, low end GPUs have a high mark up due to the GPU shortage. If you can afford to bite this bullet, then that would be up to you.
You have greater flexibility if you build your pc from scratch (aka custom built PC).