I don't see why not!.. you may even want to shred them instead normal delete... and if they are many and take up gigabytes of disk space, you could save time creating a small disk partition, move the .exe files there and format the partition in quick mode. http://www.fileshredder.org/
Just wondering; what type of .exe are those files... downloaded program installers?.