What country are you based in? I'm Australian so I'll try my best..
Ring up your ISP (Internet Service Provider) and tell them your issue, see if there is something on their side that is affecting it.
Maybe also try checking for hardware issues. I see you've already swapped routers which is great, but have you checked connections? For example, in Australia, majority connect our routers to the Internet with phone line cables (DSL). I don't know what types of cables are used in fibre based connections. Maybe see if the cable could be an issue?
Then again, as you said, your download speed is fine but your upload is not. Best guess is there is a "bottleneck" the ISP can try remedy if you call them up and tell them your issue.