When you do install more One of two things generally happens. The memory works, but is limited to the maximum amount supported by the motherboard, or the memory doesn't work at all.
On every motherboard, there is a controller for accessing the RAM. The limiting factor is how much memory can be accessed (or addressed) by that memory controller. Theoretically, a 64-bit CPU can access 2^64 bytes of RAM. For practical reasons, however, the number of addresses lines actually etched into a motherboard is much smaller, and the controller is created to be able to access up to a specific number of addresses. It can address fewer memory locations just fine as well. That determines the range and maximum amount of memory.
So when memory is installed with more addressable bytes than the controller understands, the best outcome is that only the lower portion of the RAM is used. However, because of the way memory is constructed, it's also possible that the larger memory won't work at all.
Edit: If you g above the motherboards supported RAM.