Start >> Programs >> Accessories > System Tools > Disk Defragmenter.
Or refresh it by re-formatting.
If neither method fixes it, toss it way and buy a new one (quality brand if possible).
No pen-drive should take that long to access so it's probably going to fail completely anytime soon - replacing it is the safest option.
My 16GB pen-drive (a SanDisk model) is detected the instant I plug it in on both Windows 7 & Vista.
That's irrespective of how much data is on it.