The answer is "it depends". HT abstracts your single physical core as two independent cores, this is to allow usage of unused resources inside a CPU. A CPU's resources isn't percentage based, there is no such thing as 20/50/100 "percent" usage, we only use that term due to how easy it is to understand. 1 SB core has three ALU's that do integer math which happens to be the vast majority of a CPU's workload. x86 ISA is a very serial ISA, mechanically it's hard to execute more then one instruction at a time inside code, hence a special vector co-processor exists within the CPU (SSE/AVX) to handle vector instructions. Knowing this you can see the problem, having 3 ALU's worth of processor resources to handle one x86 stream isn't very efficient, that stream is unlikely to ever utilize more then one of them, branch-prediction and super scalar preprocessing enables the CPU to attempt to utilize the other two ALU's to do work. HT is a more direct method allowing a second x86 instruction steam to execute on unused ALU's.
So could HT slow down single threaded processing? Theoretically possible yet highly unlikely. You would have to be running some incredibly efficient code to actually utilize all of a processor's resources.