Your sure it's not simply because fivem uses independent servers?
Usually, that is the problem and you are clearly not the only person. This is a long running issue. Try playing without the mod on a standard server and see if there is a difference. I'm betting there will be a huge one.