A good PC VR gaming experience, including the HMD (head mounted display) and the full PC build is going to be in the $1500 range to start.
Games, like on Steam for instance, will indicate whether they have VR support or not. The Oculus store, of course, is for the Rift HMD and are all VR experiences. While there are ways to take games that don't have VR support and run them (VorpX is one), these are generally not ideal solutions.....or take a bit of tinkering to get running smoothly (but again, they will never be as good as native VR support).
Is it worth the cost? Depends on you....PC's are the very best platform to experience VR on (leaps and bounds better than any mobile you've happened to try yet, and certainly will blow PSVR out...