If you run into a situation where the NVidia drivers do not gracefully handle the 3.5 & 0.5 GB memory segmentation, your framerate will tank and you will notice the performance drop as clearly as the nose on your face. Anything else you are seeing is psychological or the result of some other system metric.
If you're that concerned, perhaps you should be doing some benchmarking so you remove yourself as the deciding factor as to whether or not it "seems" slower. The human brain and sensory system is hardly calibrated to measure frames per second accurately, anymore than your finger can accurately determine tire pressure by pressing on the sidewall of a tire. 🙂