This sounds like the port is not negotiating to 1g. Most times this is the cable. Some device, maybe your pc, can tolerate cables that are just slightly defective.
When you using the router you have 2 cables that can be the problem. Your pc should tell you if the port negotiated to 1g or 100m. Maybe plug both cables into lan ports on the router just for testing. Your pc will logically be directly connected to the wall jack but will physically pass though the lan ports on your router.
There is not a lot you can try other than buy some new cat5e cables and hope to get lucky. The only other things are defective ports in equipment which would be really bad. There is of course a tiny chance there is a problem with the in wall...