When I open Core Temp, I notice that the frequency of my CPU will randomly jump between 4400hz, 800hz, 2200hz and etc. The same happens with the power, it'll jump from like 12W to 62W. Is this something to worry about? My CPU is an Intel i7-4790k and has not been overclocked.
Your CPU will jump around in power draw & clock speed as necessary. Basically, it'll clock lower/draw less power when there's very little for the CPU to do.
To verify this, you could enable the "high performance" power plan in Windows - that'll peg your CPU to ~4.4GHz on all cores 100% of the time.
Increase in power draw, but to most people it wouldn't make a huge difference in the grand scheme of their electricity bill.
Absolutely no harm in leaving it as is though, that's how it's designed to run.