After reading up a little more I found out that the issue, most likely, is that my card has a very hot hot spot.
As you can see from the max value, the hot spot exceeds 100C while the overall GPU temperature is still around 75C. It seems reasonable to assume that's what's triggering the fans...