"how video encoding/decoding works so i can optimize the cpu "
It works by splitting up the data,each picture or a segment of video i.e. a number of pictures, and sending each packet of data to a core.
The more data there is the more cores a codec can use,4k can use more cores than 1080.
But not all codecs are the same,some can only use a certain number of cores before the data that goes to each core becomes too small to make any (additional) difference.
The codecs that can use close to infinite threads have moved on to use the GPU since GPUs have many times the threads of CPUs.
"Can encoding use multiple cores and scale linearly? "
It almost always uses multiple cores and does scale very linearly up to the point where the data can't benefit anymore from being split up any further.
"Also, does hyperthreading the cores help with improving encoding performance? "
Yes it does,additional computing power helps.