Not sure about the circumstances that you describe.
Do you pause and leave a YouTube video for some reason? Are there any screen or power savers in use?
In any case, once you return to the video, Windows will need some time to "reload" or otherwise return to using whatever processes were running before you left.
As will everything else downstream to the source of the video.
The delay is not something that I would be concerned about.