Undervolting only improves FPS if your card isn't maintaining it's max clockspeed due to power or temp limits. It does lower power/temps/noise though, regardless of FPS effect, which is always welcome in my eyes.
Have you ran GPUz to determine if either of those are causing performance limitations?
I helped someone undervolt their RX580 in
this thread. You'll find what you need there to get started.
To paraphrase from the linked thread: Undervolting is to set the voltage for a given frequency(ies) lower than what the Auto/default values are (this applies to both CPUs and GPUs). AMD/Nvidia/Intel can't spend the time required to test and custom set a perfect freq/voltage curve for each chip they manufacture. Instead they test a bunch of chips, get a band of variance, and set a freq/voltage curve that encompasses all chips. Depending on how lucky/unlucky you got in the chip lottery, you may be able to undervolt your chip a lot or a little respectively from the stock values without changing the operating frequency.