As you may know, a peliter 'throws' heat towards one of its sides, and cools the other one, so you need to actively cool the hot side before the heat is conducted to the cold side by thermal conduction.
Maybe the power consumption of that peltier module is too high for the fan you are using...