Rather than removing the fans entirely, invest in some quieter fans. You can get two ML120s for about £20 on Amazon and run it in push/pull. They're quiet and good performance, especially for the cost. Other good options would be Silent Wings fans or Noctua, both high quality and low noise fans.
Replacing case fans is also a good option for reducing noise and possibly reducing temps depending on the quality of the fans you already have installed.