Of course they dont want sell you their best of the best its a pure money machine ..
No, it's delicate balance of how much IPC they can deliver at a certain clock speed, on a certain process node, at a certain area. Increasing IPC comes at the expense of die area and clock speed. It's not just a dial you can turn up or down, independent of anything else.
It also takes time to increase the sophistication of their designs, which mostly build on what they did in the previous generation. Realistically, they can only change a certain amount between one generation and the next, which goes towards explaining some of the improvements in Zen 3, which was made on mostly the same node as Zen 2 and has only like 10% more die area.
Engineering is a very incremental exercise, which a lot of people might not fully appreciate. You can do all the modeling and estimation you want of a chip, but at some point you just have to build the thing. Then, take detailed measurements, do thorough analysis, and figure out what worked and what didn't, so you can decide what to build on or scrap, in future generations.
Time is also a factor. Not only do they need to do all of the modeling and design, but still leave time for testing, debugging, and a couple respins. And they do
a lot of testing, since chip bugs can be so costly. At one point in time, I think the industry average was 2 test engineers for every 1 design engineer.
Nvidia dont sell you their best cards first ( disregard the 5090 ) 100% there will be a 5080ti and a better Vram something to keep buyers buying .
The reason they release the "Super" and other up-spec'd editions is because manufacturing yields improve over time, giving them margins to unlock more of the chip and push clock speeds a bit further.
Ask yourself why do AMD bother with the non x3d CPU when basically now the 9800x3d is the best gaming CPU and the 9950x3d is basically the best gaming /production CPU ??
Because the X3D die adds cost and there are some use cases where it adds basically no performance.
Another reason is that it takes more time to do the X3D CPUs. If they ship the non-X3D version first, they have time to perfect the base CPU. Then, they have a stable platform to use for optimizing the X3D version.
Also, their chiplet desktop & laptop CPUs are sort of a byproduct of their server CPUs, which is a much bigger and more important market for them. The desktop CPUs again function as sort of a development vehicle and secondary market for those chiplets. And plenty of server workloads do not significantly benefit from the extra L3 cache (which is why they can get away with their C-cores having half the per-core L3 as their normal CCDs).
For example with the current state of Intels desktop CPU's if AMD gave us a 50% uplift with 10000series then what are they going to do give us another 50% with 11000 at some point they will only be competing with them selves and thats bad for business !!
The problem with this logic is you're assuming it's even possible to deliver that kind of improvement. Look how much trouble they had in delivering an improvement in either Meteor Lake (so much so, that the desktop version even got cancelled) or Arrow Lake!
Most of the time, the obvious answer is the right one. These companies are trying to give us as much as they realistically think they can deliver and sell. Maybe, around the time of Haswell or Skylake, Intel was holding back on IPC just a little bit, due to little real competition. That's obviously no longer a luxury they can afford. I think Golden Cove was the point where they basically gave their best effort to offer as much IPC as they could, establishing a sort of new baseline.