Router for gigabit internet using QoS

[juggalojcox]

Hello everyone,
I am having a bit of an issue. I have WOW for my internet services and i have the 500/50 speeds. this is great at all levels but one, my asus TM-ac1900 doesnt seem to be able to handle that kind of speed when i have QoS trund on. I have a obi200 for voip and i game online alot and QoS helps with ping and UDP trafic for my VOIP services. Now I have 2 options. 1) buy a router that can do what i need, I believe the unifi has their USG pro for $300 can do what i am looking for. 2) DIY it. i know it might be a bit more than the unifi USG pro but for something i can build my self and make sure its doing what i need ill go that route. if buying one is a good option please let me know some knowen good routers or if DIY is the way to go, give me some Hardware sugestions for it. just an FYI, i would like to make sure it can do gigabit internet speeds cause WOW will be bring the gigabit internet to may aera soon

 

[nigelivey]

For your needs a small pcengines pf box would do, im not sure why youre looking to build a super computer. The APU2C4 should be more than enough unless you are running a vpn server.

 

[Kewlx25]

Serious question: Why do you feel you need QOS enabled anywhere? The only reason to require QOS is if you're fully loading a link and even then it all depends on the direction of the traffic. To that point, if you're fully loading your outbound internet bandwidth (50Mbps from the looks of it) then QOS on your edge router is only going to be able to put certain packets on to that wire with priority, which is the only benefit it can give you.

QOS will do NOTHING for you in regards to inbound traffic coming in to your router from the internet.

I don't think you realize how often a link becomes "fully loaded". Even at 1Gb/s, a link can get fully loaded for 100ms-250ms watching an 8Mb/s video stream. Imagine playing a game and getting periodic 100ms-250ms ping spikes because someone is watching Youtube or Netflix., even though you have a 1Gb/s internet connection. remember, they're bursting 40Gb-80Gb/s at you.

I found this out while investigating why I was getting small bursts of packetloss when I spam-jumped around video timelines in either Youtube or Netflix. I have a 150Mb/s dedicated connection over a 1Gb/s rate-limited line.

 

[marko55]

Serious question: Why do you feel you need QOS enabled anywhere? The only reason to require QOS is if you're fully loading a link and even then it all depends on the direction of the traffic. To that point, if you're fully loading your outbound internet bandwidth (50Mbps from the looks of it) then QOS on your edge router is only going to be able to put certain packets on to that wire with priority, which is the only benefit it can give you.

QOS will do NOTHING for you in regards to inbound traffic coming in to your router from the internet.

I don't think you realize how often a link becomes "fully loaded". Even at 1Gb/s, a link can get fully loaded for 100ms-250ms watching an 8Mb/s video stream. Imagine playing a game and getting periodic 100ms-250ms ping spikes because someone is watching Youtube or Netflix., even though you have a 1Gb/s internet connection. remember, they're bursting 40Gb-80Gb/s at you.

I found this out while investigating why I was getting small bursts of packetloss when I spam-jumped around video timelines in either Youtube or Netflix. I have a 150Mb/s dedicated connection over a 1Gb/s rate-limited line.

You don't think I realize how often a link becomes fully loaded???

Video is bursty and greedy, yes, but even a 4K video stream eating 15Mbps isn't going to burst to saturate a 1Gbps link and impact other users. Come on man. On a 150Mbps internet link, yeah, I can definitely see a slew of traffic maxing out that link.

Where we see gig links get saturated typically is on LANs where large chunks of data are moving from server-to-server or server-to-client computer. At 1 gig you're only talking about 100MB/s which most computers these days (certainly with SSDs) can easily utilize. Multiply that times 50 users and the links to your servers can take a beating. Inside the DC forget about it; 10Gbps is a must when you start getting big.

I administer a WAN in the US with 170 branches and even the IPSec sites that are accessing data all day long in our data centers rarely max out a 50Mbps internet link. Some of these sites are on our VoIP system and use our SIP trunk back in our DC, over site-to-site IPSec, for PSTN calling for up to 20 users in some cases. No QOS at all outside of the Cisco LAN where auto-qos is configured, and we very very rarely get complains of voice loss or jitter. Occasionally we'll get a couple users pulling a huge file from the DC and we get a complaint. This is thwarted easily enough in some cases by having two internet links & pinning voice traffic to one & all other traffic to the other. Naturally our MPLS branches don't ever experience this pain because of QOS across the WAN where we DO have full control in both directions (to you point of egress tagging). Regardless, we can't tell an ISP to prioritize our inbound voice traffic, obviously. Especially when its all encrypted inside an IPSec tunnel.

The point about inbound QOS, as has been mentioned here, is you can't control traffic coming in to your network from the public internet. You can only attempt to restrict non-critical sessions from fully eating all your inbound bandwidth to leave room for your priority traffic, and doing so is tolling on your edge device as juggalojcox has found. Its also probably not going to work well for UDP traffic as the edge device can't toy with the Windowing to get the sender to slow down.

The thing about QOS is if its needed all the time you should really be upgrading bandwidth. Its more for protection of "just in case I occasionally come under load." It shouldn't be OK for your edge device to have to constantly be utilizing it, for just the reason this post started. It can be very tolling and in the end the real fix is to upgrade bandwidth (which is being done here of course by moving to 1Gbps at least on the download side). I understand though, its not that cut and dry in the residential space where you can only get your hands on so much bandwidth, so its tough.

 

[juggalojcox]

Ars Technica has several VERY interesting articles on home router performance at gigabit speeds.
https://arstechnica.com/gadgets/2016/01/numbers-dont-lie-its-time-to-build-your-own-router/
https://arstechnica.com/gadgets/2016/04/the-ars-guide-to-building-a-linux-router-from-scratch/
https://arstechnica.com/gadgets/2016/09/the-router-rumble-ars-diy-build-faces-better-tests-tougher-competition/

Very interesting. Smallnetbuilder.com is changing their testing to more closely match the testing shown above.

wow, i must have missed all this when i was looking for info about this online. so lookes like a mini-pc or built one is a good way to go, or that edge router pro8 looks really nice and its rack mountable. if i built one my self and got a rackmountable case it would cost 2X as much after everything. i think i am looking more at the edgerouter pro8. not for sure yet though

 

[juggalojcox]

Serious question: Why do you feel you need QOS enabled anywhere? The only reason to require QOS is if you're fully loading a link and even then it all depends on the direction of the traffic. To that point, if you're fully loading your outbound internet bandwidth (50Mbps from the looks of it) then QOS on your edge router is only going to be able to put certain packets on to that wire with priority, which is the only benefit it can give you.

QOS will do NOTHING for you in regards to inbound traffic coming in to your router from the internet.

I don't think you realize how often a link becomes "fully loaded". Even at 1Gb/s, a link can get fully loaded for 100ms-250ms watching an 8Mb/s video stream. Imagine playing a game and getting periodic 100ms-250ms ping spikes because someone is watching Youtube or Netflix., even though you have a 1Gb/s internet connection. remember, they're bursting 40Gb-80Gb/s at you.

I found this out while investigating why I was getting small bursts of packetloss when I spam-jumped around video timelines in either Youtube or Netflix. I have a 150Mb/s dedicated connection over a 1Gb/s rate-limited line.

You don't think I realize how often a link becomes fully loaded???

Video is bursty and greedy, yes, but even a 4K video stream eating 15Mbps isn't going to burst to saturate a 1Gbps link and impact other users. Come on man. On a 150Mbps internet link, yeah, I can definitely see a slew of traffic maxing out that link.

Where we see gig links get saturated typically is on LANs where large chunks of data are moving from server-to-server or server-to-client computer. At 1 gig you're only talking about 100MB/s which most computers these days (certainly with SSDs) can easily utilize. Multiply that times 50 users and the links to your servers can take a beating. Inside the DC forget about it; 10Gbps is a must when you start getting big.

I administer a WAN in the US with 170 branches and even the IPSec sites that are accessing data all day long in our data centers rarely max out a 50Mbps internet link. Some of these sites are on our VoIP system and use our SIP trunk back in our DC, over site-to-site IPSec, for PSTN calling for up to 20 users in some cases. No QOS at all outside of the Cisco LAN where auto-qos is configured, and we very very rarely get complains of voice loss or jitter. Occasionally we'll get a couple users pulling a huge file from the DC and we get a complaint. This is thwarted easily enough in some cases by having two internet links & pinning voice traffic to one & all other traffic to the other. Naturally our MPLS branches don't ever experience this pain because of QOS across the WAN where we DO have full control in both directions (to you point of egress tagging). Regardless, we can't tell an ISP to prioritize our inbound voice traffic, obviously. Especially when its all encrypted inside an IPSec tunnel.

The point about inbound QOS, as has been mentioned here, is you can't control traffic coming in to your network from the public internet. You can only attempt to restrict non-critical sessions from fully eating all your inbound bandwidth to leave room for your priority traffic, and doing so is tolling on your edge device as juggalojcox has found. Its also probably not going to work well for UDP traffic as the edge device can't toy with the Windowing to get the sender to slow down.

The thing about QOS is if its needed all the time you should really be upgrading bandwidth. Its more for protection of "just in case I occasionally come under load." It shouldn't be OK for your edge device to have to constantly be utilizing it, for just the reason this post started. It can be very tolling and in the end the real fix is to upgrade bandwidth (which is being done here of course by moving to 1Gbps at least on the download side). I understand though, its not that cut and dry in the residential space where you can only get your hands on so much bandwidth, so its tough.

well as of right now I only have 500 down, and when WOW brings the 1gig over to my aera i want to be ready for that when it happends. and QoS is really good for lowering ping for gaming ( went from 60 with it off and 15 with it on) and i know it will be better for VOIP even if i cant fully fix the download i can still regulate all of the download coming into the network and steer it for thoese 2 services and of course the video streaming going on from the plex server witch i belive is UDP too??? kinda of unsure about that one but i know that obi200 does use UDP traffic for the voice and i believe online gaming does too for latency issues for like overwatch and COD

 

[Kewlx25]

Serious question: Why do you feel you need QOS enabled anywhere? The only reason to require QOS is if you're fully loading a link and even then it all depends on the direction of the traffic. To that point, if you're fully loading your outbound internet bandwidth (50Mbps from the looks of it) then QOS on your edge router is only going to be able to put certain packets on to that wire with priority, which is the only benefit it can give you.

QOS will do NOTHING for you in regards to inbound traffic coming in to your router from the internet.

I don't think you realize how often a link becomes "fully loaded". Even at 1Gb/s, a link can get fully loaded for 100ms-250ms watching an 8Mb/s video stream. Imagine playing a game and getting periodic 100ms-250ms ping spikes because someone is watching Youtube or Netflix., even though you have a 1Gb/s internet connection. remember, they're bursting 40Gb-80Gb/s at you.

I found this out while investigating why I was getting small bursts of packetloss when I spam-jumped around video timelines in either Youtube or Netflix. I have a 150Mb/s dedicated connection over a 1Gb/s rate-limited line.

You don't think I realize how often a link becomes fully loaded???

Video is bursty and greedy, yes, but even a 4K video stream eating 15Mbps isn't going to burst to saturate a 1Gbps link and impact other users. Come on man. On a 150Mbps internet link, yeah, I can definitely see a slew of traffic maxing out that link.

Netflix can max out a 40Gb link. I've wiresharked Netflix on my WAN when my connection was idle, and I was getting the ~250KiB requests in 2ms. aka, 1Gb/s.

The problem is a lack of packet pacing. When they send you traffic over TCP, the entirety of the response fits in your network buffer. Since the same TCP connection gets reused for multiple requests, at no point do any packets get dropped to signal TCP to back-off because the entire request fits in the buffers. The TCP window keeps increasing and dumps the entire response at their line rate, which is 40Gb for most of their new servers.

Not everyone gets their problem because their ISP doesn't have a very good route to Netflix or YouTube. My ISP exclusively uses Level 3, which has excellent peering with Netlix and YouTube. I get 1Gb/s even from YouTube in Europe.

As of Dec '16, YouTube and Google in general has been deploying BBR TCP which does have packet pacing and should get rid of this burst issue. I haven't checked wireshark in over 6 months, but it's possibly this is partially or fully deployed already.