It's slightly misleading to call them passive, since server heatsinks depend on high airflow from a bank of fans that feed directly into them - they just don't happen to be directly attached. The servers I've seen even have airflow guides to help force the air from those fans through the heatsinks.
Because passive cooler doesn't work itself. At certain point, heat doesn't spread further.
If you have air around the passive block, it remains hot. It won't spread in whole room ;-)
You have to always force air; otherwise efficiency would be crap.
I used to have large Alu plate with 5x GPUs on it...plate was warm about 5-10cm from heat source...that's it. It won't spread to the whole plate ;-) I am guessing it something like resistance or so...no idea...I just did CAD simulation.
It is simple nature of seeking opposites...in this case warm-cold
You see it from this example...CPU idles...no issues...capacity of near copper material sinks heat at same rate...you add 100W...you are done. Whole block won't suddenly get hot, sinking all heat from CPU ;-)
Efficiency is linear with temperature delta/difference. It may work if ambient temp is 0°C or below 0.