If you've only got the one fan as exhuast removing the side panel is probably helping the air flow to the CPU and GPU coolers and reducing the fan speed because it's got more cool air to cycle through. Since you don't have any other fans you're not really disrupting the case' natural flow by taking off the panel and it is indeed helping.
You might want to think about adding some intake fans though which should remove the need to remove the panel and expose your system to dust etc.