First, before I look up your board, a general answer. An experienced overclocker (not me!) will generally tell you that system memory runs faster with 2 modules than with 4, and with 4 modules that came in a matched kit than with four random modules or two matched pairs.
From what I've seen posted, though, the effect ranges from significant to "only noticeable in benchmarks," with significant being rare. What I mean by that is that dropping in mismatched extra memory is much more likely to help than to hurt if your system needs memory. You would get the best possible results from a matched pair of 8 GB modules, followed by a matched quartet of 4 GB modules, followed by what you are suggesting doing. The cost of mismatched memory varies from not noticeable to "oh, crap" and you have to try it and see.
EDIT: I looked up the board. It can take up to 4 modules and up to 32 GB, so it will handle 2 GB, 4 GB, or 8 GB modules just fine. It's dual-channel, so be sure to match the modules: Put the two 4 GB modules in the first slot for each channel (which is almost certainly where they are now), and the two 2 GB modules in the second slot for each channel.