When you plug a USB drive on your PC it reins to write and read data from it, of which some information are cached. Caching is a way of keeping information in your PC’s memory (RAM) instead of writing it immediately to the USB device, Thus if you were able to unplug the USB drive out of your PC before this information is written, or while its being written, you’ll end up with a corrupted file or even more to a corrupted media file system.
The "Safely Remove Hardware" is an extra level of security preventing you from destroying your drive and your own files. Using the proper procedure will assure that your data and your device are in good shape.