The thing about VRAM is that it's not JUST used as a framebuffer, it also stores in use assets and assets not currently in use but stored for quick loading in the future. If more room for the framebuffer is required, currently unused assets are paged to RAM (if space is available) and then to the page file on the hard drive which increases loading time when those assets have to be used again.
Also as newer, more computationally efficient ray tracing algorithms are developed and more accessible to lower end hardware without DLSS, the extra VRAM will allow these cards to have the space available to utilize those effects, as those effects are quite the VRAM hog.