you can't buy the blue thingy. is is simply a piece of plastic that helps provide support for whatever you plug into it. if it came out then the port is broken and you need another one to use.
replacing a port is not really doable for a novice even if you could get a new port to put in. you likely have to install an add-on card that gives you extra usb ports you can use instead of the broken one if you have no others you can use.
also note that continuing to try to use the port can damage more than just the port. every time it turns on and off it is because the power connections is being cut over and over. this can lead to sparks, power surges and possibly damage to the device plugged in. i HIGHLY suggest you do not use...