Ultra is indeed quite punishing, and isn't something you're going to pull off with a midrange graphics card. The advanced hair physics option alone is tanking performance on every GPU out there and it isn't advisable to turn it on without a GTX 980 at the very least. Depending on what sort of framerate is acceptable to you can do high settings with hairworks turned off, that should get you into the 40s most of the time with the framerate, though in some towns you will likely dip into the 30s. If you want 60 FPS solid everywhere, you're probably going to have to play on low to medium settings to get that.
You actually should be okay on the VRAM front, on high settings at least, Witcher 3 isn't a huge VRAM hog unlike say GTA V, so you can do high or ultra textures on a 2GB card no problem. Running on ultra for all settings might just push it over the 2GB mark, but that is pretty much irrelevant as no graphics card with only 2GB of memory is even fast enough to do ultra settings and maintain a playable framerate.