According to Asus, lowest what your MoBo supports is 4800 Mhz. And if you do have one of the earlier DDR5 RAM sticks, that were made to run 4000 Mhz, then there are compatibility issues.
Though, according to wiki, lowest JEDEC standard for DDR5 sits at 4400 Mhz. So, i have no idea how your RAM can even sit at 4000 Mhz. Yours looks like engineering sample or similar in this case.
DDR5 wiki: https://en.wikipedia.org/wiki/DDR5_SDRAM
You might get lucky with manual adjustment of memory timings, to make your RAM stable enough. But going with new RAM could be the only fix, in the end.
As far as memory timings go, and what to do with those; you have two options:
- Tighten the timings.
- Loosen the timings.
Tightening the timings are best done when you have stable RAM and you want to squeeze every bit of performance out of the RAM. Loosening timings is best done when default XMP doesn't hold and you need to get to the frequency target. (E.g when XMP timings are 40-40-40-80, then loosening would mean 40-42-42-86. Tightening would be 38-38-38-78.)
Then there are frequency increase and voltage adjustments as well, that go along with the manual RAM OC,
further reading: https://www.tomshardware.com/how-to/overclock-ddr5-ram
So, if you have willpower and time, to put into manual RAM OC (RAM tweaking), you can put in the effort for it. Though, keep in mind that it all could be in vain and if you'd rather not waste your effort in doing all this, then going with new RAM set would be better.
Your GPU seems running fine, since it was able to complete Superposition. And it wasn't bottom of the barrel either (compared to the submitted results),
Just put 3090 into search bar and select Nvidia RTX 3090 to view results. Out of all submitted 3090 results in 1080p Extreme, yours is around page 8. (Then again, your GPU isn't OCd either.) So, i think that we can assume your GPU not to be culprit in all of this.
As of why 3D Mark doesn't want to work, well, perhaps your RAM is interfering with it.