On what basis have you concluded that you're underperforming where you should be and that your VRAM is your bottleneck here? You've told us nothing about your FPS or the game in question. Or, really, anything about your PC whatsoever aside from the GPU.
In most cases, an RX 550 will have trouble using 4 GB of VRAM in gaming since games that require that much are usually way too much for an entry-level GPU. The most likely thing is that the RX 550 has low FPS because that's the limit of its processing power. But since you haven't given us much in the way of information, it's hard to guess further.