Yes, you want straight connections from all PCs to the router. You would only use a cross over from one PC to another PC. However, most modern equipment can correct for the wrong cable being used.
Plug your PC into the wall and then find the right cable by the router and plug it into the router, you should have service then if everything was terminated correctly.