In my opinion this is very much a Windows problem. Many Linux distros run well on both x86_x64 and ARM 32/64 (also seen in the Switch "hack"). macOS actually still runs on both x86_x64 without issue. I think the WinTel runs deep within Microsoft and for some reason they either can't or are not actually interested in seeing it succeed. Back in the RT days these things were difficult, but today supporting ARM and x86_x64 in a single product is fairly strait forward.
Over the last months I've taken the plunge and got myself both an RP5 8GB and an Orange Pi+ 32GB to play with ARM vs. x86. They are pretty similar in cost and performance to Atom based NUCalikes I've also used for low-cost HCI clusters on oVirt/RHV and Proxmox.
And ThomasKinsley is quite correct in his estimation, that hardware diversification is causing issues and I'd say that they are getting far bigger with the way the industry is moving.
We are used to think of ARM vs x86 vs RISC-V mostly as an ISA or instruction set architecture issue. But that has almost become the
least of our worries these days, after people have stopped coding in assembler and even avoid issues like byte sex. All that's needed with open source is a re-compile... or let's just say most of that work has been done already.
But nobody sells CPUs any more. Everybody sells SoCs, where the parts that run the ISA may actually be minority surface area share holders. And the relative importance of those non-CPU IP blocks is only getting bigger, GPUs, DPUs, VPUs, NPUs are pushed to overcome the scale limits of general purpose single threaded code and then there are ISA extensions, which push those xPU functionalities back into processor cores. RISC-V was started years ago precisely to make that easier, but both x86 and ARM have been doing ISA extensions for decades now, to the point where the previous paragraph seems almost false.
Hardware dependencies require software to take care of them and software gets somewhat easier to make when it gets well architected abstractions to build on, also in hardware. And of course what is considered well-architected changes over time and with the workloads that are run: nothing designed in 1990, not even really forward thinking super computers using Inmos Transputers was designed with LLMs running on wafer-scale silicon in mind.
Coming back to the PIs, I can't take a single OS image and run that on that on both PIs, neither Windows nor Linux.
Well again, actually, yes, I can, to a degree. Using a per-device EFI bootloader to deliver some hardware abstraction I can run the same image on both. But the best I can hope for in terms of functionality is CPU, RAM, storage and basic networking, typically via USB interfaces. None of the GPU, VPU, NPU or DPU parts of the SoCs are working, unless that OS does include modules or drivers for each variant.
And when it comes to the various ISA extensions, of which there are really, really interesting ones in terms of hardware pointer validation, return-oriented programming protection etc. like CHERI present or coming to ARM, your OS is rather unlikely to use them until they are really, really common in existent hardware. On the application side, it may be much easier to profit from ISA extensions as long as they won't affect the size of the register file or other structures that need saving/restoring on process switches. Just managing that on x86 is filling whole books.
In practical terms, I've been trying to run Proxmox on the OP5. Promoxmx requires a Debian base that isn't Ubuntu, but full Mali GPU support only seems to be available on Ubuntu (and their proprietary Linux variants).
That might never change, because not all parts of the full Mali GPU software stack seem to be open source and Rockchip itself doesn't think it worth spending time on a full debian port.
I don't know about the NPU, even less if it's something these licensed from elsewhere, but by the time it's fully supported the caravan may have long moved on to some new variable precision IP blocks that deliver 3x the inference at the same Wattage or all that AI has gone bust.
Writing these device drivers is hard work. It's just as expensive to do that for Nvidia's newest and biggest GPU as it is for a Raspberry PIs VPU, which is, if I understand it correctly, a DSP design mophed into a GPU.
In other words the sales price doesn't dictate the effort, which makes it very hard to recover on a chip that sells for a couple of bucks and not in billions of units.
Having to do that with Windows and Linux, won't do well for the economy. Some OS have gone and started to work with Linux compatible drivers, like VMware. Not sure that's an approach Microsoft has chosen for Windows on ARM and actually Dave Cutler might have put better abstractions into his VMS clone than Linus T. put into his clone of Unix: if I remember correctly it's something Andrew Tannenbaum used to chide him about...