Yes exactly, let me explain why that happens -
First whenever you start a new game, all the textures have to load from Hard Disk which is much much slower than RAM's read - write speeds.
Secondly the graphics card has to render the whole scenario for the first time.
What VRAM (Video RAM or RAM of Video Card) does that, it stores the loaded textures by hard disk and rendered stuff too. So next time you view the same thing, it does not have to load it all over again which makes things go light on video card and hard disk and thus its faster and smoother.
Mainly the lag is caused due to slow hard disk and video card. You should get at least 1 TB 7200 RPM HDD. I believe your CPU and Video card are just ok. Not recommended for Crysis 3...