What's not really clear from that is the increase number of devices and architectures that were added to make it one of (if not the) most flexible OS around. And, sure that's a lot of lines of code, but how much is actually parsed and built into a normally-configured kernel?
This issue was recently brought up with the brouhaha and sharp words from Linus himself about the sad state of ARM additions (the typical "I can do better, let me reinvent the wheel for my version of the ARM" software engineering mindset of devs from TI, Marvell, etc. caused huge code bloat due to hacking together "board support" files instead of proper rearchitecting the ARM-specific bits of the kernel to share same/similar code) and has gotten better. It's nice to know that there are still people at the helm that care about that sort of thing.