First of all, I think your Vcore is a little high for just a 5.0 Ghz overclock (but again, each chip's overclocking ability is different).
When overclocking cpu, there are a lot of things to look at. Depending on your motherboard, each of these things might be named slightly different and appear under different sections of the BIOS.
Therefore, I recommend that you watch some youtube videos to know what parameter should be left on Auto and what parameter should be manually input with a number. Try "How to overclock i7 8700k on MSI motherboard"or things of that nature.
Although this is not all of them, I can tell you several things which I remember about this motherboard (in no particular order):
1. Set to Fixed mode to ensure clock speed is applied to all core
2. Ring ratio could be set at lower (doesn't have to be equal to your clock speed)
3. Disable EIST, turboboost (This should automatically be greyed out once you apply Fixed Mode)
4. Disable C-state and make sure you select High Performance plan in Power Options in window
5. Try Override mode and this is where you manually enter your Vcore (Always stay under 1.4V, but preferably under 1.36)
6. Select LLC mode 4 to ensure constant voltage is supplied under 100% load (If you're at 1.36 Vcore and your LLC is at mode 3 or less, it could really spike your voltage up to over 1.4V, which is not recommended)
Other than that, I would say the most important thing right now is upgrading your cooling system. Since you say you only have 1 fan on your AIO, I assume this is a 120mm AIO? If this is the case, you will need to back down to 4.0 Ghz or just leave it at stock (3.7 Ghz). It will NOT be enough for any substantial overclocking. 5.0 Ghz is definitely not recommended. Try upgrading to at least a 240mm or 280mm if you can in the future.
Every time you try a new clock speed, temp test your cpu using Prime 95.5 version 26.6 and select Small fft test. This will quickly tell you if you are in the safe range (under 80C preferably) or you are hitting thermal limit. For stability testing, use asus realbench.