Unless your server's hardware is unique in the world, it likely contains multiple chips that share common secrets with every other similar chip in the world. Hackers don't need to get your specific server, they only need to get any of those other chips to expose those common secrets and build from there.
In the case of breaking through Apple's Secure Enclave, you can buy a bunch of iPhones and use them to train your bench to figure out where the bits you need access to for extracting password hints are on the SoC before making any attempts on your victim device(s).
The point is that designs that were formerly thought to be pretty secure may not be considered secure for much longer.