What is the make/model of your cpu cooler and case?
What is your fan arrangement?
Have you overclocked your cpu?
If your idle temperature is much more than 15-25c. over ambient, your cpu cooler is not functioning properly, or it is not mounted well.
If your case does not have sufficient intake airflow, that can also be a problem.