Short answer, yes.
The memory on the GPU has no relation to the system's memory.
PCI Express is more or less fully backwards compatible as well. As long as you have an x16 (or 8x wired) slot available and enough power, you can plug in any GPU. Now if you go too new, they might not work on boot, but should in Windows. But that comes down to BIOS vs UEFI BIOS types.
As for the fastest card you can put in there without wasting it, well, I always say, doesn't matter. If you plan to ride this thing until it dies, you are still getting the maximum possible out of it. Or if you plan to upgrade the CPU in the near future, then overbuying on GPU isn't terrible, just means you won't have to factor a GPU into your next immediate upgrade.