Still, there is also a possibility some of these PI4:s will end up in custom IOT-devices (maybe even sold as part of another solution) and left unpatched in later exploits.
The patches to the original spectre/meltdown vulnerabilities that affected ARM should already be in the Raspbian kernel used by the Pi v4. Perhaps the Pi 4 even used updates of the A72 that has them fixed in hardware.
As for other attacks that are discovered in future, the issue of vendors & users failing to patch their devices is currently true of all IoT platforms. As for the A72 being OoO, we can hope that, by now, any side-channel vulneabilities in it have been found - it is basically a 4-year-old core that's probably already in hundreds of millions of devices. But there are no guarantees.
In any case, IoT has less exposure to side-channel attacks, since those require executing malicious code. The main way that most people execute malicious code is through their web browser, which most IoT devices don't have. The other way is for cloud-based servers to have a malicious VM running alongside its target, on the same physical machine - also not an issue for IoT. So, I think your concern is slightly misplaced.
there is already a problem with many of them not being updated (either due to the person owning them no knowing/caring about it or that there are no updates made possible by manufacturers.
They will often just "sit" there. It is often best to have these type of devices running on its own little subnet, if that is possible to set up on the router.
This is an ongoing issue, and it's not helped by the fact that many IoT devices need to access the cloud as part of their function. I agree with segregating them on their own subnet, but we both know that most people won't do this.
What's needed is for some kind of certification program where manufacturers not only undergo 3rd party testing, but also set aside funds to provide fixes and firmware updates for some pre-determined amount of time. There could also be a standardized way of updating the devices, so that either they could self-update or you could automatically scan your network and update all the devices on it.