Since google sent me here, thought I'd update it.
According to this site, it can take DDR3 1066 and 1333 MHz
http://www.dellmemoryram.com/dell_optiplex_memory.html
The specs:
http://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/optiplex_780_tech_spec_sheet.pdf
also mention 1066 and 1333.
However, in the footnote it says "4 This system supports both 1066 MHz and 1333MHz DDR3 memory, however, the system will operate at 1066MHz speed due to chip set specifications."
I also know from experimenting that it supports DDR3 1600 MHz RAM (but runs it at 1066). I have had success with DDR3-1333 as well. There have been some problems reported with getting 1600/1333 MHz to "take" as it were (it's "finicky" as it were--possibly might help to update BIOS/drain the power:
http://en.community.dell.com/support-forums/desktop/f/3514/t/19579385?pi21953=1 though some report even with that they can't get 1600 to run [?]). On many websites the "recommended" is 1600 MHz RAM, though I've definitely gotten 1333 MHz to work OK as well).
It seems that it supports "4GB max/slot" with 2 slots max in the small form factor, 4 max in the "normal desktop size" factor. As is normal with DDR you will want to install 2 at a time for best performance.
Buffered, Registered, or ECC ram is also not possible apparently, so don't get that kind of RAM for it. Only unbuffered will do. Also you'll want to mount them in the "same colored" RAM slots so they can work together in parallel for best speed.