Open Activity Monitor (Applications > Utilities >) and look in the System Memory tab when you notice a slow down. See how much FREE memory you have (the higher the better), and how big the PAGE INS/PAGE OUTS is (the lower the better).
4GB of RAM is pretty standard nowadays. It is too bad Apple sells machine with only 2GB. At least, memory is cheap now (if you don't buy it from Apple).
Here's a good place to buy memory for your mac:
http://www.macsales.com/
At 29$ for a 2X2GB kit (mac mini 2011 model), it is a no brainer for me. I would even consider the 2X4GB kit for 50$ so I get an eventual good resale value on my machine.
edit: RAM won't affect your connection speed, at least not directly, but web pages might load slower if your system is starving for RAM.