Both, CPU and MB have to support quad channel memory and for that combo it's not possible. Best possible is to have pairs of RAM in dual channel.
Processors like that one have 2 IMC (internal Memory Controller)s each supporting a pair of RAM in dual channel. Quad channel memory is supported only on high end server CPUs and MBs. In such servers special (server) RAM is also required. Your system is far from having options like that.
Performance therefore can't "suffer" because it's not available at all in first place. You might also have problem to run memory at full rated speed because of IMC overload and/or have higher temperatures in 2 x dual channel memory.