On Windows, whenever you install a program, the installer not only places the files for the program in the location you specified, but may also make changes to the registry, and usually adds a desktop icon and a start menu entry, that point to the executable in the install location. The registry entries may also contain the install location, and that information may be used by the program to find its settings, data files or libraries.
Copying files doesn't do this - you're only making a backup of the files. While most times you can start the program from the copy without problem, there may be some oddities if it uses registry to locate its files (since those still point to the original install location).
So if you make a copy / backup of a program, you're free to delete the copy anytime.