Router packets loss under stress

[sam1275tom]

Hi everyone, I test some routers and found some of them will have packet loss under heavy load, here's details:
Test method: ping router continuously while run speedtest.net or torrents, all by wired connection.
NIC: gigabyte Z77X-ud3h on board

D-link di-604 HW.E1 FW 3.53: Reach max speed @ 30mbps, ping lag up to 50ms, NO PACKET LOSS.
Linksys WRT54GL v1.1 FW tomato shibby 1.24: max throughput observed@36mbps, minor ping lag, PACKET LOSS observed even without max the router, higher the speed, more packet loss, very high chance to lost a packet when establish the test connection.
Firebox II with monowall or smallwall: max speed reached@20mbps, very unstable, nearly no ping lag, MASSIVE PACKET LOSS.

I tested some Netgear and Mikrotik but I either forgot the result, or didn't have the high speed fiber at that time, so only these 3 results are valuable. I really confused why some routers dropping packets instead of queue them behind, and the best result I like is even saw on a old wired router...
Welcome to discuss this my friend.
PS Don't tell me my routers are too old, we all getting old as time passed, only technical discuss please.

 

[Kewlx25]

TCP uses packetloss to signal to backoff. Otherwise networks would be constantly flooded. Packetloss is a normal thing when you have congestion, it's latency that is not normal.

When people are getting 400ms pings, that is because of bufferbloat. Companies putting overly large buffers on networks in attempts to not drop packets, but dropping packet is normal. TCP does very strange things when packets are not dropped in a timely fashion. A lot of the performance issues people have during peak hours is because these buffers are so large, they don't drop packets quickly enough and TCP keeps flooding the networks until a bunch of packets are lost, then it backs off a whole bunch.

If you actually look at a network graph of a network with normal sized buffers, you get a small oscillation of bandwidth near the maximum bandwidth, and the average bandwidth is very high. If you look at a network with too large of buffers, you see a large swing of bandwidth and huge bursts of packetloss. The average bandwidth is quite low.

In trying to not drop packets, they made things worse.

 

[sam1275tom]

TCP uses packetloss to signal to backoff. Otherwise networks would be constantly flooded. Packetloss is a normal thing when you have congestion, it's latency that is not normal.

When people are getting 400ms pings, that is because of bufferbloat. Companies putting overly large buffers on networks in attempts to not drop packets, but dropping packet is normal. TCP does very strange things when packets are not dropped in a timely fashion. A lot of the performance issues people have during peak hours is because these buffers are so large, they don't drop packets quickly enough and TCP keeps flooding the networks until a bunch of packets are lost, then it backs off a whole bunch.

If you actually look at a network graph of a network with normal sized buffers, you get a small oscillation of bandwidth near the maximum bandwidth, and the average bandwidth is very high. If you look at a network with too large of buffers, you see a large swing of bandwidth and huge bursts of packetloss. The average bandwidth is quite low.

In trying to not drop packets, they made things worse.

Thank you VERY much for this reply.
I still have a few questions however...
1. Don't TCP use ACK to signal if it's too fast to respond? I think packet drop is only used in UDP...
2. Why the di604 have a good and smooth speed without packet drop? I see it have a big lag at the beginning of the test, but then it recover to normal respond interval, even the test continues.
Thank you.

 

[Kewlx25]

Indirectly, yes, TCP uses ACKs. When a a segment of data is sent, the sender expects an ACK from the receiver within a timely fashion. Something around 1-3 seconds. If an ACK is not received, the sender will resend the missing segment and backoff. This means the sender can't distinguish between a lost packet and a highly delayed packet.

If the receiver sees packet 1 then packet 3, it can send an SACK(selective ACK) to request the missed segment. If this happens, the sender backs off.

The difference between TCP and UDP is TCP does all of this stuff automatically for you. With UDP, you need to do it all your self. You could make UDP work just like TCP if you want, but why reinvent the wheel? If you need a TCP like protocol, then use TCP. If you need something entirely custom, use UDP.

di604 is actually quite slow. My guess is that the di604 is slow enough that it is effectively the bottleneck and not your Internet connection. Because of this, you get an artificial limit on queuing which reduces bufferbloat. If you had a faster firewall, your latency may actually become worse.

 

[sam1275tom]

Indirectly, yes, TCP uses ACKs. When a a segment of data is sent, the sender expects an ACK from the receiver within a timely fashion. Something around 1-3 seconds. If an ACK is not received, the sender will resend the missing segment and backoff. This means the sender can't distinguish between a lost packet and a highly delayed packet.

If the receiver sees packet 1 then packet 3, it can send an SACK(selective ACK) to request the missed segment. If this happens, the sender backs off.

The difference between TCP and UDP is TCP does all of this stuff automatically for you. With UDP, you need to do it all your self. You could make UDP work just like TCP if you want, but why reinvent the wheel? If you need a TCP like protocol, then use TCP. If you need something entirely custom, use UDP.

di604 is actually quite slow. My guess is that the di604 is slow enough that it is effectively the bottleneck and not your Internet connection. Because of this, you get an artificial limit on queuing which reduces bufferbloat. If you had a faster firewall, your latency may actually become worse.

Thank you very much.
You make me get more deep understand than before.
Yes the Di604 is slower than my WAN. I buy the internet speed @ 20mbps, but I get up to 40mbps for a few seconds if test a very near server, then drop to 27mbps before the speedtest end, I guess the constant speed is 25, this is very similar when I buy the 4mbps but get 5mbps without bufferbloat.
A interesting thing is I get near constant 30mbps on DI604 with very little speed drop during the test, I think it will drop more if the test is long enough.
A few little things to be clear, I ping the gateway, not any internet address, so if I have a faster gateway, it should be lower latency TO GATEWAY.
BTW, you still feel normal if I get packet loss when ping my GATEWAY?
Thanks again.

 

[Kewlx25]

Normally you shouldn't get loss to your gateway because that means your LAN is saturated or your gateway is overloaded.

 

[sam1275tom]

Normally you shouldn't get loss to your gateway because that means your LAN is saturated or your gateway is overloaded.

Thanks.
It begin to drop packets before reach the max speed, so I still feel it's do it on purpose, I want to change this behavior but donno how.
Another interesting thing is the max latency (in windows resource manager) is 30000ms on di604 when torrent, but only 5500ms on watchguard with smallwall, maybe because there are not enough connections to observe, the smallwall son;t have UPNP so I cannot receive incoming connections.