[SOLVED] Spectrum routers drop our websocket connections (unless IPv6 firewall is disabled)?

[Erveon]

Hi!

Apologies if this isn't the correct forum, but I'm out of options. I'm developing an online game and we use websocket connections for server events like matchmaking, chat etc.
Since the very start, people with a Spectrum router have consistently reported connection drops to the websocket, anywhere from 20 seconds to 2 minutes after establishing it.

After an ungodly amount of debugging we figured out that disabling the IPv6 firewall on the Spectrum router fixes the problem.
Now I see myself helping people fix this issue daily and would love to be able to work around this on our end to stop wasting people's time and opening them up to potential security risks.

The socket connection is made from a native client in C# to a Node.js back-end server, switching out libraries for different ones on either has not resulted in any improvements.
Our Node.js server is behind Cloudflare and a reverse proxy using Nginx. The game is created using the Unity game engine if that matters.

Important to note is that ONLY Spectrum routers are experiencing this so it's not the good old Cloudflare socket timeout. The Node.js server uses an IPv4 address but being behind Cloudflare it resolves an IPv6 address. It is not possible to disable this functionality on Cloudflare and we do not have the option to remove Cloudflare from the equation at the moment.

Any ideas or even crumbs to lead us to a potential solution are greatly appreciated!

Thank you

 

[bill001g]

They have been claiming IPv6 is the future for 20yrs and we still see large number of strange problems reported.

I suspect your only real option is going to be to try to send email to spectrum customer service documenting exactly how you get the bug. The easier you can make it for them to see the bug the faster it will get fixed. Now this is likely a long term project, if you get it resolved in under 6 months I would be surprised.

What happens when a customer does not have IPv6 support. Does the cloudflare use IPv4 ip addresses or does the game not work. I tend to tell everyone to just disable IPv6.

It has been years since I actually wrote software. Can you force it to use IPv4 only. I know for example DNS will give you both IPv4 and IPv6 addresses. Can you choose to use only IPv4 at the programming level in the client or this one of those things that is hidden from you by the game engine code.

In some ways at least you know what the problem is. The messy virtual server concept of things like cloudflare mean that you can have all kinds of hard issues where you can't actually test things because different people have very different paths to the server.

 

[Erveon]

They have been claiming IPv6 is the future for 20yrs and we still see large number of strange problems reported.

Yes, it's honestly really sad. I don't think the protocol itself is to blame, but rather all the companies involved seeing it as a cost and not an investment.

I suspect your only real option is going to be to try to send email to spectrum customer service documenting exactly how you get the bug. The easier you can make it for them to see the bug the faster it will get fixed. Now this is likely a long term project, if you get it resolved in under 6 months I would be surprised.

Hearing from the people that experience issues, I'd be glad to just get ahold of someone for a simple customer question let alone for a problem like this. I'll give it a shot, though.

What happens when a customer does not have IPv6 support. Does the cloudflare use IPv4 ip addresses or does the game not work. I tend to tell everyone to just disable IPv6.

It falls back to an IPv4 address I'm fairly sure - haven't had any issues with the game just not working while I know a lot of people still don't have proper IPv6 support.

Can you force it to use IPv4 only. I know for example DNS will give you both IPv4 and IPv6 addresses. Can you choose to use only IPv4 at the programming level in the client or this one of those things that is hidden from you by the game engine code.

I really wish I could force it to only use IPv4 on Cloudflare's end but they don't let you do that, they want IPv6 to be more normalized. Pushing it through the hard way I suppose. I wasn't aware that DNS returned both though, from a DNS lookup I see the AAAA records returning IPv6 and A records returning IPv4. I wonder if there's a way I can hack at the client socket library to only use the IPv4 result somehow. It'll either have to be that or getting rid of Cloudflare altogether I believe.

Thank you for your response!

 

[Ralston18]

And I will add the suggestion that you simply add the applicable notes and instructions to your game installation documentation.

I.e., "If problems occur try disabling IPv6 ....."

The note could also be added to Website FAQ's, applicable Forums, game troubleshooting tips, etc..

And you do not want to be in a position where "hacking" client socket libraries is a solution. That could easily backfire.

 

[Erveon]

And I will add the suggestion that you simply add the applicable notes and instructions to your game installation documentation.

I.e., "If problems occur try disabling IPv6 ....."

The note could also be added to Website FAQ's, applicable Forums, game troubleshooting tips, etc..

And you do not want to be in a position where "hacking" client socket libraries is a solution. That could easily backfire.

I think you may be overestimating people's willingness to read error messages 😅
Our disconnect message says to check the FAQ page if it happens frequently, which gives them the reason and the fix, but most of the time they just contact us or give up without reading.

Definitely don't want to be in a position where I have to hack at the socket library, but if it's nothing too crazy to implement (it's open source) then it's probably worth trying I believe

 

[Ralston18]

"I think you may be overestimating people's willingness to read error messages " - no disagreement per se. Just tend to be hopeful... sometimes.

Those are often the same people who get very upset with the "Is it plugged in" type questions.

Just use (or at least try to use) very large colored font that presents the fix: first in the Disconnect message, then copy/paste the same into the FAQ page(s).

Will need to defer to others with regards to hacking socket library within open source. Implementation of such things is not within my comfort zone.

 

[bill001g]

hacking I think is more in the technical meaning rather than trying to bypass security. Games companies make changes to the engine all the time. This is not something on the end user machine most these things it is basically a bunch of files you use when you compile a program. Now how hard it is to change something like this I have no idea. I can barely make tiny changes to software example and they click build.

 

[Erveon]

hacking I think is more in the technical meaning rather than trying to bypass security. Games companies make changes to the engine all the time. This is not something on the end user machine most these things it is basically a bunch of files you use when you compile a program. Now how hard it is to change something like this I have no idea. I can barely make tiny changes to software example and they click build.

Yes sorry, did not mean to confuse! Hacking at it meaning debugging the library and changing its code to hopefully be able to force it to only look at the A records