Running Windows 8.1 and only recently has this started happening, really annoying and slowing everything down.
Ran a full virus scan and turned up nothing.
Task manager does incur a bit of an observer effect while it is running. It should not be excessive though. If it is taking up a large amount of CPU time it may be malware masquerading as the Task Manager.