The case is designed with the expectation that the front panel will be in place.
I am not sure about the difference that 2 - 5 degrees more will make especially if you are monitoring temperatures and keeping an eye on things.
It would be worth the effort to double check your fan installations/placement etc. to ensure that they are within the suggested/required locations.
Overall, the system may run cooler with the front case off but the resulting change in airflows could be facilitating some unmonitored/unmeasured hot spot being created.
Front panel on.
Note: also a deterrent to dirt, debris, bugs, and little hands......