Well, I wouldn't say it slows it down "dramatically". That was true back in the USB 2.0 days (max about 25 MB/s). But with USB 3.0 drives, you're usually able to hit a HDD's max speed of 125-180 MB/s.
As for SSDs, the issue is that everyone obsesses over sequential speeds. Yes USB 3.0 will cap the SSD's sequential speeds to about 200-250 MB/s. But unless you're copying large files (e.g. movies), sequential speeds don't really matter. SSDs are slower at reading small files - about 25-50 MB/s, which is nowhere near USB 3.0's speed cap. So in most read/write operations with mixed file sizes, the vast majority of the time is spent reading/writing the small files, so the sequential speeds don't really make much difference.
HDDs are...