As pointed out, due to addressing limitations inherent in 32 bit OS's installing memory modules over and above (roughly 3.2GB for a single card system) will result in the RAM being unused.
Gordo's post is correct as well - latency will go up slightly as you use more RAM. Not to mention your motherboard has the added load of driving modules that are merely taking up space.
In plain english - You gain nothing because the RAM can not be used, and it is more stressful on your hardware.