Please list the brand and exact Model number of your Power supply unit/PSU, which you have been using to power up the RTX 3080. Have you also tried a CLEAN re-installation of the GPU drivers, and even any OC software you have been using, like MSI Afterburner, EVGA PrecisionX etc. ?
Remove all previous saved OC profiles, and any fan curve settings which you might have enabled before to overclock the graphics card. Since you are getting a BSOD, then something might be wrong with your hardware, and/or even some software settings.
Which Power Plan have you enabled in Windows OS ? Windows 10 or 11 ? Always OC the GPU in small increments.
For the system stability, you also need to look for artifacts, polygons, or a system FREEZE/hang after applying the overclocking settings. If you don't observe any of these, then you are safe. You are getting a BSOD though.
Voltage and heat are the things you want to watch out for. Only increase voltage if your temperature values are safe at FULL load. All components have different voltage tolerances.Voltage increases always should be tiny. Some basics of overclocking, which you might be already aware of:
Increase Mhz, and Test Prime95 or Heaven, or any other benchmark for ~5-10 mins. Repeat until it fails. Lower Mhz OR increase voltage. Repeat.
Once you have your maximum stable core overclock figured out, you can do the same for the Memory. But, memory OC's generally don't provide that much FPS gains, so it's up to you if you even want to tweak this setting as well.
You use the exact same process as your CORE overclock, and take it as high as you can with the memory. Move the Core slider up 25-50 Mhz. Run Heaven for 5-10 mins, watch your temps, etc. It's the same as CPU overclocking. Once you get a driver crash, system freeze, or see visual anomalies, then dial it back to 10-20 Mhz, and then try again.