That is not a "USB Port" but a USB drive, or a Flash drive. USB Port is what you plug that thing into.
Right-click on the drive letter in My Computer, select Format. Will wipe everything off it.
Getting a new one would be like buying a new trashcan each time yours fills up.