Gigabit Ethernet: Dude, Where's My Bandwidth?

[scook9]

I have to say, you all complain WAY to much. I personally found it quite an informative article. The title was not "how to push your gigabit LAN to the max" after all. This really explained a lot to me about how network file transfers occur and what is involved.

I do believe that a few months back THG said that they were opening up to user submitted articles, so PLEASE, show them how it is supposed to be done....oh wait, you just talk shit, so go sit in your cubicle and shut up please well these writers actually take the time to try helping and explaining these topics to readers. You complain like you are actually paying for this service. Get over yourselves.

 

[lejay]

The ram to ram test is very strange. Using 8b/10b encoding the maximum speed should be 100 MB/s.

 

[GlacierFreeze]

[citation][nom]jankee[/nom]The article does not make any sense and created from an rookie. Remember you will not see a big difference when transfer small amount of data due to some transfer negotiating between network. Try to transfer some 8GB file or folder across, you then see the difference. The same concept like you are trying to race between a honda civic and a ferrari just in a distance of 20 feet away.Hope this is cleared out.[/citation]
Wow, what a dumbass. Even worse is he tries to compare it to cars. lol

 

[que3jxp]

Look... The fact is that they screwed up a number of fairly important FACTS. Because of that, the entirety of the article is corrupt.

Gb Ethernet is 119MB/s NOT 125MB/s

PCI is NOT 133MB/s. It is 125.8MB/s(33MHz PCI) or 251.7MB/s (66MHz PCI). The last revision of PCI is typically the 66MHz variant and is on more PCs than you would think. But then, PCI is shared bus so if you have a slower 33MHz card and a faster 66MHz card installed at the same time, the lower speed card sets the pace.

Ethernet max length, as pointed out several times, is 328 feet NOT 100 feet. Also Cat5 IS Gigabit certified so long as your cabling infrastructure was IBDN certified (Obviously not something you will see in a house but is much less an issue for short runs.) Cat6 was a scam and 5e is more than enough for client use. The newer 10Gb over copper standards are not for PCs, they are for server racks. Eventually, this will get to the consumer level but not for a LONG time so newer versions of Cat6 or even Cat7 are a moot point.

And yes, copying a few LARGE files compared to backing up your pictures across the network is a HUGE difference. The exact same issue is seen even going from one drive to another in the same machine. The i/o penalty for opening a file and negotiating and then closing the file that frequently comes with a massive penalty.

All this said, anecdotally, I can say with great assurance, that the average achieved performance with 100Mb Ethernet is about 5MB/s. The average performance on 1000Mb Ethernet is about 30MB/s with large file copies achieving up to 55MB/s. The biggest impact, as pointed out in the article, is the hard drive performance. Which makes me also wish they did a test with an SSD or with an SSD array so that they can show any possible bottlenecks from the SATA interface.

All around, a poorly written/thought out article.

 

[que3jxp]

I should note that when I say "large file copies achieving up to 55MB/s", I am referring to common household use and not lab conditions.

 

[bunz_of_steel]

good article and comparisons Mr Woligroski, I would like to see a follow up with SSD's and maybe comparing different routers. See if the El cheapo's perform just as well as name brands. Would be interesting to see. I liked the article.

 

[lamorpa]

And information on things to do to make transfers faster (as the article intro stated)???

 

[icepick314]

i never had to move large files before i added gigabit equipment on my home network...

my file server is just a simple Ubuntu box with bunch of hard drives in it with gigabit ethernet card...i use it for downloading tv episodes on bittorrent network and backing up gaming files...

i have D-Link DGL-4300 linking between my gaming pc with onboard gigabit card...

also i have gigabit switch from Linksys/Cisco to connect Xbox 360, PS3, and Popcorn Hour to the router for online gaming, file sharing, and watching downloaded video files and playing my music collection...

the slowest file transfer is between Popcorn Hour and file server since Popcorn Hour only has 100Mbps interface...moving a large movie file, say DVD iso to play movies from other region, takes about 45 minutes to more than an hour...

but file transfer between my gaming pc and file server is FAST...once i moved a backup DVD iso in a few minutes...

 

[bunz_of_steel]

Enough about anal specs of ethernet guyz, who really cares which way you twist the wires? or 119 vs 125 bla bla bla. I just wanted to add one more thing Mr Woligroski if and when a follow up article is decided on. Can you also include various MTU/jumbo packet frames sizes and if they make a diff too? Thanks for a job well done sir!

 

[cybernard]

I've got a 2 relatively high end units. One with a RAID 6, capable of faster than 200mb/s write and faster than 400mb/s write. Vista 64 to linux 64. CAT 10gx wire. Mainly,Intel Server PCI-e nics. Latest version of software(2.6.30) and patches. From server to Vista 6mb/s and Vista to server 15mb/s. Any suggestions?? (Besides switching linux to Windows?) BTW Vista unit has 3 SSD's.

 

[nobrainer]

I found it to be an informative article. Just last week I ran Cat5e to the living room and was a little disappointed by the overall speed. Granted night and day better than 54g for streaming media - but still not as fast as I would have hoped. I will probably do a bit of "testing" on my own just for curiosity sake.

 

[mikepaul]

I set up gigabit between my two PCs a few years back, and even after a lot of tweaking I found it was faster to use cold-swappable hard drives (the cheap kits from CompUSA) instead. The shutdown/restart time was faster than the transfers.

I'd certainly go back to gigabit if I wired my house, but 802.11N is doing just fine and avoids paying someone to crawl through my attic. Meanwhile, my new Linksys WRT610N does show the value of gigabit routers in that the 'speed boost' thing that RoadRunner offers only showed up when I had a router fast enough to catch it. SHORT burts of 1.2mbps downlaods are better than none at all...

 

[geofelt]

Good article. Ignore the nitpicking of details. It explained the following:
1) Wny you might not get 10x transfer speeds with a gigabit network.
2) Identified the hard drive as the primary limiting factor.
3) Showed that cabling has no real impact on performance.

 

[GenKhan2]

I would like to have seen the importance of flow control in routers/switches and its impact on throughput mentioned. Device firmware is generally written to ensure availability of the network and not throughput. They will not allow any single client to destroy network performance for everyone else by using all the bandwidth. As a result, it is extraordinarily difficult to get full gigabit throughput through any device. It also turns out there is no standardized behavior when flow control is invoked. That is, the amount of time spent waiting depends solely on the whim of the firmware engineer. That means the throughput varies dramatically based vendor and even combination of devices on a network.

I have personally tested around a dozen consumer gigabit switches for throughput. The best performance thus far has been the DGS-2205 or DGS-2208. Both will allow for full gigabit throughput using jumbo frames. Ironically, it's probably a bug in their firmware as that behavior is detrimental to the network at large. Anyway, buyers beware. Gigabit is rarely gigabit.

 

[wathman]

The guys over at Small Net Builder did some great testing to see how fast of a small network you could build in regards to file transfer speed. They explored more hardware and OS options, though the final answer was basically the same. System RAM plays a large part in file transfer efficiency. If I remember right, they found that Vista will buffer a file transfer up to the amount of available system RAM, so for a 32-bit OS, that's usually 2 GB of available (NOT total). 64-bit OS you can obviously go much higher. 100 MB/sec might be achievable as long as your file transfer isn't larger than free RAM. Once you go over that, it drops down to the bandwidth the disks can supply, just as this article says.

 

[raptor550]

Wait, I thought this was common knowledge.. though judging by some of the questions and comments people ask, I guess it wasn't. I agree with most of the harsher comments, though I do appreciate and value the results.

But seriously, a router isn't better than a switch because of the processing overhead. At my office we have about 4 routers, mostly IP assignment, but all the processing is done by 12 Catalyst 6500's. That's because each one has 2 terabytes per second in throughput. A switch is designed just to transfer data and it is the most efficient path of doing so. You want a fast network? Get a managed switch.

 

[wathman]

raptor550 makes a great point, and I've been looking for ways to speed up my own home network. I use a router for DHCP assignment, and to supply wireless access. Under high traffic, it lights up like a Christmas tree and speeds start to suffer. I just picked up a small 8 port gigabit Smart switch, though my question now is what to do for DHCP? Use the old router, or dedicate a server to manage it?

 

[suddenstop]

OK - I found this article to be good and informative at the level it was intended. I also am very interested in a deeper dive on the gigabit networking.

I have regular cat5 run throughout my house up walls through the attic down walls etc. Do a cabling test like another poster recommended at 50 foot intervals to 400 feet (yes I know 328 is a max). Be interesting to see what happens. A lot of us want to know what our old cat5 layer is really doing to us.

I'd also like to see some nice fat drive arrays and ssds in the picture.

Overall nice article, now let's get the in depth geek test!

 

[bounty]

Good article to get things started. Sometimes you want a technical manual, other times you look at the quick start guide. This was the quick start guide.

"A switch is designed just to transfer data and it is the most efficient path of doing so. You want a fast network? Get a managed switch."

Nah, if you're that hard core between your PC and your server, just get a 15ft Cat5e (or 6) crossover cable, 3xRaid SSD's or new HDD's and go nuts tweaking your dedicated NICs.

 

[Computer_Lots]

Wathman... I use a little gigabit switch and a PC running Clarkconnect for my Router/Firewall/DHCP Server/File Server/Web Server/Print Server. I noticed a farily significant increase in my internet throughput after going with this setup as apposed to my little cheapie router. My speakeasy.net speedtest went from around 1.5mbps to around 12mbps. I'm sure this wouldn't be an issue if I had a better router to begin with since most of the good ones can throughput over 100mbps anyway. You're probably ok to use your old router for DHCP and internet access, but a Linux router would give you more options and might be a little faster.

 

[zodiacfml]

i'm pretty neutral for this article since i've read (forgot where, here at TH probably) long ago when gigabit products came out, it is found that it was overkill for home networks. from then on, i thought gigabit ethernet was for SOHO networks with busy multiple PCs, add to that the cost of the switch.
my home network is mostly internet and media streaming traffic.
anyways, i agree with comments to include some technical aspects so regular readers can learn something new.
i know, uninterested readers can easily skip those pages.

i have a question though, what happens if an ordinary router serves internet to a gigabit switch? PC to PC file transfer remain gigabit speeds, right???

 

[Guest]

I'm wondering why they used the smaller and slower hdds and yet the conclusion page show the same WD 640GB hdd I have that tests out on my system at about 105MB/s. That dive is about 50% full. My new WD 1TB hdd averages 108 MB/s and my Velociraptor tops 120 MB/s so I hope they re-run these tests with newer and faster hdd.

 

[KyleSTL]

Guys, get off the cleeve's back. He did a test of real-world results and even if every technical detail he wrote might not be 100% correct, he demonstrated what a home network might consist of, and how it would perform. I appreciate the simplistic approach of actual performance rather than the nitty gritty details of network interaction.

 

[Guest]

What about error correction and maybe and enginering channel takes up the last 14mb's stoping the tests from hitting 125?

 

[wathman]

Computer_Lots, thanks for your benchmarks. After I get to tweaking, I'll have a better idea of what I should be getting for performance. I also have a PC running Untangle that I could bring back up online to use in a similar fashion as you use your Clarkconnect PC. Now that I have a new smart switch coming in, I might have better luck with the Untangle server.

zodiacfml, for the most part that is correct. Also keep in mind you could tweak the performance a bit if you use hardware that supports jumbo frames, though jumbo frames is not a feature that will work miracles. Expect performance gains in the neighborhood of 5-10%. As for internet connections through routers, don't worry about gigabit connections between your modem and router. Unless you have a brand new DOCSIS 3.0 cable modem, or whatever hardware needed for Verizon FIOS, your modem is only capable of a 100 Mbit ethernet link to the router.