For memory, usual rule of thumb says the more the better. I used to have 512MB in my laptop and upgrade 2 months ago to 2GB (max supported). My memory usage usually stays at about 30-35% average. So I guess I'm in the same boat as you.
The point of adding memory is to make sure you won't have to work from the HDD, unless loading something. Wait untill you start importing pictures and working with them and you'll be very happy you didn't get back to 1GB, believe me. You might loose 0.5% performances points in some benchmark, but it'll feel MUCH faster on everyday use. Plus, benchmark are for bragging rights unless it's your encoding time that's at stake. And that'll be faster with more RAM.
Also, do a google search about putting WXP kernel totally in RAM. That'll put rarely used XP components in RAM and make your system even more responsive. I did it and, else then making my system more responsive, all it did was adding about 100MB more in memrory, so about 5% of my total memory. Big deal with 512-1024MB (20 and 10% of total) memory, but nothing with 2GB.
Keep on using your "extra" RAM. Nothing bad comes from it!!!!!!!!!