So what is this device you call a "modem". I will assume you somehow connect multiple device to the ISP "modem". This would mean the box is actually a modem and router. Most actual modems only have 1 port so you could only connect it to the garden router. It isn't even the port limitation it is that the ISP only gives you 1 IP address. That is the key function of a router to allow multiple devices to share the IP.
If the ISP device actually has router function then you can add a second router and plug your PC directly into the ISP device. I would recommend that the second router be run as AP rather than running it in router mode. When run 2 actual routers in your house you can have issue with devices not being able to talk to each other, but they can all use the internet fine.