Thermal conductivity determines how fast the heat transfers. It really is that simple. Thermal diffusivity determines how quickly something will cool itself, but not how well it works as a heatsink. That is almost exclusively determined by the thermal conductivity, and gold simply loses to silver in that respect.