You will not see better performance unless you run programs that use(combined) more then 8 gigabytes.
If you did run such programs with less memory, your computer would have to swap data from the hard drive to memory(back and forth at times).
Honestly, I do not think most users would see the difference(And I say this as a user with 16 gigabytes on one system 12 on the other and 8 on another 2 systems.).
4 was kind of cutting it close at times, but 8 seems to be as you say the "sweet" spot.
If you DO use lots of programs or multi game + host lots of game servers(and VMs/massive Photoshop use/ect), the extra ram will help keep things smoother(less need to go to the hard drive to swap data).
Loading will be the same as long as you do...