Actually, Bufferbloat does matter, in most situations, and almost all the time. Unless you're the only person using the internet connection (no family, no roommates, no phones uploading pictures to the cloud) any other traffic can seriously increase your latency/lag.
Even "normal" web browsing...