The GAME itself and your game settings are what mainly determines video memory usage.
However, very few games use more than 2GB let alone 4GB of video memory. So I wouldn't worry about it.
*Your only concern should be tweaking game settings so your frame rate is acceptable. Such as:
1) 40FPS average or higher with VSYNC OFF (use FRAPS to view frame rate), or
2) 60FPS with VSYNC ON, or
3) 60FPS with Adaptive VSYNC ON (recommended if you drop below 60FPS at times but want VSYNC)