What makes and models modems and router (or combination modem/router) do you have? Router's logs (if available and enabled) may provide some clue.
In addition, the modem and router information can help with troubleshooting and the overall physical connections within your network.
For example, if you have a modem/router and a router then the two routers may be both trying to assign IP addresses and the conflicts will slow performance. Common configuration error.
First, contact your ISP and have them check things on their side. Likely to come back with "not us" but you never know...
Then check all of your cables and connections. Ensure all cables are fully and firmly connected.
Ethernet cables - Cat5, Cat5e or better? Coax...