Yeah I don't think it's that big of a deal unless you've got an aftermarket cooler on an overclockable chip and you're wanting the best possible cooling.
*Also, (if it's smudged enough and you are worried about it) instead of a credit card, you could take a plastic bag or cellophane and put it over your finger and spread it out with your finger in a circular motion.