what the software did was change the RAM speed settings in the BIOS, and you did it without also adjusting the timings and voltage.
You really shouldn't have gone that high trying to overclock RAM.
The most you should've tried going was 3600MHz speed, and that's even if the RAM you have can do that.
Some RAM is budget for a reason, and just can't go any higher than the rated overclock, which yours is 3200MHz.
Just do a CMOS reset to reset the BIOS and see if you can boot.
Also, don't use software to do overclocking.
It's more reliable and safer to just do it the normal way through the BIOS itself.