But your frame rates don't drop? Sounds like a hard drive issue or other driver issue. I agree with Anotherdrew, check to see what if CPU or HD is being maxed out.
Also run Crystalinfo (or whatever SMART utility) to see.if there are any problems. If it's a mechanical hard drive, make sure you don't need a good old fashion defragging.
Fortnite ain't that VRAM intensive.