Will overclocking CPU really affect FPS too much? Because, my CPU has 4 cores anyway.
As others have said, if you're trying to play new AAA games at max settings your GPU will likely be the limiting factor, such that OCing your CPU may not get you much.
If your GPU is preventing you from getting the FPS you want, you can turn down graphics settings. If you do this, you may end up with your CPU becoming the limiting factor, in which case overclocking might help. If the CPU is still the limiting factor and you can't get satisfactory FPS, that more or less means your only option is to upgrade. For the most part it can't be solved by changing settings, as most settings don't really impact CPU usage.
In general, you want your GPU to be the limiting factor in gaming, not your CPU, as CPU bottlenecks are more likely to result in stuttering/uneven framerate.