Looking at this manual
https://www.dell.com/downloads/global/products/optix/en/opti_755_techspecs.pdf you have a 275W PSU.
In which case, you need to look at cards that are powered by the PCIe slot only. I would think about the GTX 750(Ti), 950 or at the most a 1050.
You have 4 DIMM slots, which can take a maximum of 8GB (4 x 2).
This will be either 677MHz or 800MHz.
Use CPU-Z to determine what type of memory you already have installed and buy more of the same.
You must have a 64-bit OS to use more than 4GB RAM.
You can also install a Core2 Quad CPU to improve performance, depending on what is already installed ofc.