1. Max ram you can install is 4gb comprised of 2gb x2 ddr2 modules. Larger ddr2 modules weren't generally made or supported.
2. The highest you can go that I've found is a q8400 or e8600 processor. It doesn't seem to support higher quads. You cannot install any other processors outside of an lga775 variant.
3. This is an area that is a lot more open. As long as the card fits and works within the power limitations of your system and well as power connectors, I think it would be quite easy to find a card that will support 3 monitors. The gtx 750 ti low profile is such a card that would work well. If you're not worried so much about graphics even cheaper cards like a gt 710 would work great. The only caveat you have to look out is that the card supports bios vs uefi as your system is not uefi.
Now, if you a biggest boost, you can install an optiplex 380 motherboard as it is a bolt-in swap with no modifications. The 380 supports up to the q9650 as well as even the lga775 xeons like the x3280:
https://www.cpubenchmark.net/compare/Intel-Xeon-X3380-vs-Intel-Core2-Quad-Q9650/1286vs1050
But you need to see if the extra cores will help you more than just the raw single thread speed you may already have installed:
https://www.cpubenchmark.net/compar...6700-vs-Intel-Core2-Duo-E8600/1102vs1106vs957
I have a 330 with a 380 motherboard in it. I upgraded to a x3360 (q9500 equal) and 4gb of ram with an older 9500gt and it works great.