Between the physical problems or the pretty bare/weak PCB, :coolers/sockets are designed for the IHS to be the contact point.... the die would be lower in the socket without the IHS wouldn't it? So physically, a cooler, mounted correctly, would probably bearlu make contact.
And, it's kind of in the name.... Heatspreader. It will (somewhat) "spread" the heat out from the die, across the much larger heatspreader & allow your cooler to evenly(ish) dissipate heat.