It really depends.
On a motherboard that supports quad-channel ram, you will get more memory bandwidth with 4x4, which benefits performance in some instances.
The problem is most consumer motherboards and processors are dual-channel only, meaning your ram cannot run in quad-channel. Running 4 sticks limited to dual-channel will provide no bandwidth advantage over 2 sticks and does not allow your ram to run with a command rate of 1, which hurts performance.
The difference is marginal for most workloads either way, so it's not worth swapping ram kits.
What are your full system specs?