The i5-750 is also not compatible.
The way you can check is to look for numbers that are the same on your motherboard and processor as well as between your motherboard and RAM.
In your case, if you look at your GX 520, you will see that the motherboard has LGA 775 written there. That means whatever processor is in it is LGA 775. There are a couple different possibilities depending on when you bought the GX 520, but all of them have a 775 processor.
The motherboard you selected has 1155 written on it.
The i5-750 you looked at has 1156 written on it. That extra 1 makes a huge difference. The 1156s won't work in the 1155s and vice versa.
The only kind of processor that will work in the 1155 boards are ones that have 1155 on it. Two low priced options are the i3-2120 (about $120) and the i5-2400 (About $160).
Those are the two best bets if you want a 1155 motherboard.
As for RAM, the 1155 boards and pretty much all boards these days require speed 1333 RAM or 1600 RAM. They both perform pretty much the same and the 1333 is cheaper, so I would suggest you start there.
If you look at the specs of both your computers, you will see RAM numbers like 533 and 800. This pretty much means the old RAM won't work in the new motherboard.
A really good RAM chip is CT51264BA1339. That will give you 4 GBs of RAM for about $20. If you are trying to minimize up front cost, that is the best way to go. It really begs to be paired with another of the exact same RAM chips, but you can wait on the second one for a while until money is less tight.
A decent H61 1155 motherboard will be about $50 or $60, the processor maybe $120 or $160, and the RAM another $20. That puts you at about $190 to $240ish.
The 3rd piece would be the video card, which as you mentioned would need to be PCIE. Every new card is PCIE and every new board supports that, so compatibility isn't a problem as much as price is.
A really good budget card is the HD 6770. That will run you maybe $90 or $100 and it is so far better than most things cheaper as not to be comparable. I really would not go below this level. If anything will give you a playable game of Diablo 3 on a $400 budget, it is this one.
That puts you at maybe $290 - $340.
You would ideally want a new high quality PSU powering all that and a cheap low end option that can easily handle that much power is the Corsair CX 430 for $30 after rebates.
That would put you at $320 to $370ish.
That would be all the cheapest things you could get to bring yourself relatively up to date with.
You would have to keep the old CD drive and the old hard drive. You would also have to reinstall Windows after you changed all these parts out so make sure you have a windows CD and your CD key you are already using handy. If you right click on My Computer and hit properties it should have your CD key listed there.
Anyway, it isn't super cheap, but it should get you on most games in at least medium settings, especially if your monitor is only 1280 x 1024 or so in resolution.
There really isn't much of a way to step back from any of those parts. They are all about the cheapest things you are going to get that are worth the money and they won't be obsolete in another year or 2.
Any processors and video cards you could get for $50 each are pretty much obsolete already. I wouldn't bother with those.
The lowest I would go on video cards would be the 6770, but the 6850 is a big step up for about $50 more. The lowest I would go on processors is the i3-2120, but the i5-2400 is a big step up again for maybe $40 or $50 more.
If you have a Micro Center nearby, a lot of these parts may come in well under the prices I quoted here for online shopping.