Gigabit Ethernet: Dude, Where's My Bandwidth?

[Guest]

only a real time OS would provide accurate results.

the server box has very poor hardware. why not to use a high end off-board ethernet card?

the conclusion could be much better. people with no technical knowledge wants a real solution to the problem while people with experience would rather focus on reasons for it. the conclusion answers the initial question but something more could be explored. mainly because the bottleneck can be removed without a RAM drive. a server was included to the test but without a SCSI drive. would a 15k drive keep up with the network? what about SSD? latency x jumbo packets? netbios overhead test?

 

[cleeve]

I'm seeing a lot of complaints about the disk speed aspect of the article.

Tell me guys, how exactly would you demonstrate the bottlenecks without demonstrating disk speed? We achieved a 111 MB/s transfer RAM-TO-RAM, so what were we supposed to do... spend 8 pages on investigating the miniscule overhead preventing us from getting to the theoretical maximum in this beginner's gigabit network primer?

For the average joe, disk-speed &*IS* the bottleneck. Hell, for pretty much everyone else it is, too. it's impossible to discuss maximum gigabit network performance without focusing on the lowest common denominator.

What were we supposed to do, ignore the disk speed and tell everyone that their networks are running just fine, not to worry about what's causing it? Or spend pages on frame sizes and othert technical minutia which have no impact on the home user experience?

I've seen a lot of complaints here that don't stand up when the target audience of the article is understood. I think it's pretty obvious who the article is for. If you want an in-depth analysis for experts, like I said - suggest a specific topic you'd like to see analyzed and we'll have an in-depth look.

But really, I'm not impressed with the snobbery. Newbs deserve information, too.
I have trouble taking anyone seriously who read the article and maintains that they can't understand how it was written for folks without a knowledgable technical beckground.

 

[cleeve]

[citation][nom]tdwebste[/nom]Networking is an area where Linux excells. Windows simply CANNOT match the networking performance of Linux.
Nice article Tom's but leaving out the performance driving OS, Linux and BSD as in Apple is a MISTAKE.[/citation]

I hear you, but not a lot of average users are knowledagable enough to be playing with Linux distros. This article targets a lower level of technical expertise than that, an average home-user who is looking for basic expectations and a short primer.

 

[neiroatopelcc]

[citation][nom]Cleeve[/nom]If you want an in-depth analysis for experts, like I said - suggest a specific topic you'd like to see analyzed and we'll have an in-depth look.[/citation]
Suggestions:
The cable issue I was talking about earlier. Perhaps even the result of mixing different standards. For instance ccna dictates a specific minimum distance between data and power cables, while the german system dictates a much smaller one, but require more. This is in part because european cables are of a better quality (and less cheap) than american ones, and thus reduce crosstalk. What would happen if one was to just follow the cheapest of both worlds? ie. use poor cables and keep a shorter minimum distance? In the real world it appears at least with cat5e and cat6 cables the performance isn't affected at all, but perhaps my place of work is seeing different results as other places?

Another suggestion:
Investigate why wireless communication so often breaks down even when signal strength is high and there is only one client. Specificly draft n seems very prone to dropping clients. My guess is the need for usb2 when many older client pcs only support usb1 speeds. (assuming the use of usb draft n adapters in computers with old p4 chipsets etc).

 

[Guest]

Its amazing how these tech sites, Toms Hardware and others, stay up when professional jack asses show up to try to ruin it for everyone else who isn't experience or just hadn't thought `that just MIGHT be why its not working as I expected.`

If you want to talk latency, if you want to talk about jumbo frames and packets, or you want to talk about squeezing the most bits per sec out of home network, write the flip'n document and post it yourself before ya start yank'n on someone elses time and effort, you stupid hicks.

Myself, I went through the upgrade of 100meg to GigE once it became affordable, and I did my experimentation and deductions with less detail to what the author did here. He went a few steps further. I just wanted to know why I wasn't getting what I was expecting, and it came down to the drives, just as he said.

*MY* personal opinion is that GigE is for everyone simply because no matter WHERE the bottleneck is going to be in ANY modern (Lets say 5-7 year old) equipment, data transfer is GOING to be better no-matter-what, with one reservation, and that is QoS (Quality of Service - Something these punks know nothing about, and I'm not talking networking).

Another thing to consider when dealing with a LAN and you want to get `default` results out of your network is consider how much data you're throwing around. I'm assuming the author did single file transfers from point A to point B, and ended up with a result. Thats fine and dandy if your network is quiet. However, if you start throwing two transfers between point A and point B, with QoS turned on, you may go from a low-point of 30meg/sec transfers (I routinely get 50meg/sec on the SATA based machines) you'll note that transfers drop to 5meg/sec PER TRANSFER. So if you throw two transfers, you're capped to 10meg/sec. This is the fault of the SWITCH. I also learned that there are GigE hubs? Don't bother with them on a noisy/busy network, where there is more than two computers yak'n.

In my machines case, I'm able to get into the network cards configuration page and turn OFF the 802.1p protocol, which is QoS. I also UNINSTALLED the Quality Of Service drivers from all my machines as well. Now I can throw as many large files around on the LAN and still get a total of anywhere from 30-50 meg/sec transfers.

Makes me ill seeing how someone spends his time, writing up a quality document for people who have specialties outside of computers, but still fiddle with them because its new and interesting to them, only to have shit heads come along and state that its a low-grade document. Give it a rest, jerks. You're not paying for the service, so go read somewhere else if you don't like it, or submit your own crap to have people rip it to shreds.

 

[old guy]

Second try ... use wsttcp ... here is why

First, the article is excellent and accurate ... if not a little incomplete. I experienced this problem with HD limits when I switched my home network over to GigE ... my home has 8 computers and a dozen or so network appliances connected with four GigE network switches. All computers have or were upgraded to GibE. Two of the computers have raid 0 arrays, and a third has a raid 0 array of flash drives. I can copy full GigE if the computers have raid 0 arrays (except for the MCE computer, below)

Here are my lessons learned moving to GibE:

1.) HD is typically the bandwidth limiter, need a raid 0 arrays or flash to do better.
2.) My home builder nicked a cable somewhere and that outlet would not connect at GibE speed (replaced cable to fix the problem)
3.) One computer would not connect full duplex with the given network switch (only half duplex), changing switch solved the problem.
4.) Laptop lacked the performance to hit full GibE bandwidth
5.) Vista bug severely limits bandwidth when the computer has multiple network adapters (NDIS throttling) and which is really bad for us that use DCTs
6.) Xbox MCE Extender + GibE do not mix well, leave Xbox and MCE on 100MB switch
7.) Seems like switches can talk to switches a GibE speeds better then computers to switches (or the computers have poor quality GibE h/w)

wsttcp takes the HD affects out of the equation and is the only way to figure out what is really going on with your home network(performance wise). Wsttcp is not hard to use, though it might be slightly above beginner level.

 

[Guest]

[citation][nom]rundll32[/nom]@MartenKL"Gbit is actually 10^9 bits per second, ie about 119 MB/s."Nope.kilo = 1024 (2^10)mega = 1024x1024 (2^20)giga = 1020x1024x1024 = 1 073 741 824 bits (2^30)=> 1 Gbit is actually 2^30 bits.Also1 Gbit = 1024 Mbit => 1 Gbit/s translates to 128 MB/s.[/citation]
You're full of crap, not even knowing the difference between a KiloBIT and a kiloBYTE!!
a kilobit is exactly 1.000 bits!
a Kilobyte is exactly 1024 bytes!

A gigabit network like I calculated IS around 119MB/S sucker!

 

[Guest]

[citation][nom]flinxsl[/nom]do you have any engineers on your staff that understand how this stuff works?? when you transfer some bits of data over a network, you don't just shoot the bits directly, they are sent in something called packets. Each packet contains control bits as overhead, which count toward the 125 Mbps limit, but don't count as data bits.11% loss due to negotiation and overhead on a network link is about ballpark for a home test.[/citation]

Full of crap!
It's 119MB/s and take overhead off it will result in something like 109MB/s (like tests show 111MB/s).
It also depends how many files are being transferred over the network at a time, how many computers are connected (that use the network to send and receive ping packets), and I agree fully what someone else said: Sending folders with many smaller files will decrease speeds rapidly!
It's better to send one or two big files at a time, than sending hundreds of smaller files. In such case the length of the CAT cable will matter somewhat; but regardless people always try to buy the shortest possible CAT cable anyways for their network. Noone is going to buy a cable that's several tens of feet too long, only to create a long loop to connect a device only a few feet away!

 

[mutantmagnet]

[citation][nom]eyemaster[/nom]Perfect article for the general population and me included. I loved to see how it's actually the HDD causing the problem and not the typical home network.If someone is a network administrator or wants more in dept knowledge on this issue, you either know more already and don't need any of this research or you're a poor network administrator.[/citation]

Or you could be interested in understanding what it takes to be a network admin because someone else thrusts you into the job because you are the only "tech guy" in the group :|

Anyway this was a good start Cleeve. I do agree with many of your detractors only on the principle I'm hungry for more knowledge.

 

[razor512]

this doesn't explain all of the overhead

when using a network monitor, while transferring a large file over a gigabit network, the receiving system is also sending a lot of data back to the sending system (probably ack packets)
but the numbers still don't add up,

I am thinking that network hardware also plays a roll, have you tried different gigabit networking hardware.

I have tried some cheap $30 trendnet gigabit networking hardware in the bast and they did not perform any better than a 100mbit network because the router it's self was too slow to handle it

the router needs a fast CPU to handle a network transfer

and transfers are not very linear, if you can transfer data across a network at 20MB/s and you try to have 2 systems send to 2 other systems, each wont go at 10MB/s, they will go at round 7-8 MB/s because it is harder for the router to handle the 2 connections

try consumer gigabit routers and try professional ones that are designed for use server rooms and other highly demanding situations

 

[royalcrown]

[citation][nom]wathman[/nom]raptor550 makes a great point, and I've been looking for ways to speed up my own home network. I use a router for DHCP assignment, and to supply wireless access. Under high traffic, it lights up like a Christmas tree and speeds start to suffer. I just picked up a small 8 port gigabit Smart switch, though my question now is what to do for DHCP? Use the old router, or dedicate a server to manage it?[/citation]

Dont use dhcp, give your nics static addresses and no dhcp needed.

 

[Guest]

You bunch of "holier than thous" need to shut it and move on. He isn't writing an engineering paper on this. He made it clear throughout the entire article that this was a typical home network. So with that in mind, him taking that data, copying the data, and recording it's throughput is what was needed.

Frame types, overhead, burst rates, all of that technical jargon that a typical home user would know nothing about means nothing to this article. He could of added about 1500 other test scenarios and made this a work to be published in one of your geeky engineer magazines using all that silly jargon you show offs are spouting. However, that was not the intent of this article so get over yourselves or move on.

Note to reviewer: It would of been nice to see this using SSD drives though.

 

[steiner666]

I don't see the why some of the ppl that are complaining about this article being more in-depth, detailed, and technical are even reading it. If you're so familiar with the technical aspects of networking, you should already be aware of what is required to set up a gigabit network and what actual transfer rates you can expect from it with disk performance and overhead factored in.

IMO this is a very good article because it addresses a very common question. Almost every person i know who has switched over to gigabit in their home network has asked me why they aren't getting 10x the speed. I tell them that, as long as every part of their network is gigabit capable, their hard drives are more than likely the limiting factor.

On my home network I see average sustained transfer rates of about 60-70MB/s between my RAID-1 media server and RAID-0 clients. I know I could get the speed faster if i replaced the lower-rpm WD green drives with black editions, but I'm very content with my balance between performance and cool-running redundancy. As the article says, being able to copy over a dvd-5 .iso in ~1 minute is more than acceptable and worth the upgrade from 100Mb networks. Since all my computers had integrated gigabit, the cost for me to move to gigabit was the $40 price tag for a gigabit switch. Money well spent.

 

[tdwebste]

I disagree that the average user cannot use Linux.

I converting brand new laptops/netbooks almost full time now. For guess: Home users who have never used linux before, but want the fast suspend and extra battery life.

 

[Narg]

I learned this 7 years ago. Hard drives are the lowest common speed factor for high speed data transmission. Hands down. It's simple to understand once you realize that the platters can only spin data into the read head of a hard drive at one speed. It's purely a physical thing, and the ATA interface and PCI bus can't do anything about it.

Also, Cat 6 shows it's real purpose when used in electrically rough situations. I ran two cable between floors of a building, one cat5, the other cat6. The cat5 cable suffered a lot of errors. Turns out the wall I used also had a LOT of electrical conduit. This interferance messed with the data cables. But, the cat6 cable had no problems at all. Of course cat6 cable is 4 times the cost of cat5. So unless you need it, don't bother.

 

[j2b2y2k]

There is another missing component to the analysis. That is the protocol needs and packet restrictions. The packet size for 1000 baseT is the same size (in time) as 100baseT. Which is to say it can hold 10 times as many bits. But to send a ACK or NAK back through the protocol exchange it still takes as long as under 100baseT. It has has to do with the speed of light, distance and the collision detection of baseT networks. I could write volumes on the subject, but have to move on.

 

[steiner666]

Yeah, i was frustrated for the longest time because i would only get gigabit transfer rates sometimes and a slow 12MB/s or so others. Ends up Vista likes to throttle network transfers whenever you open anything in your browser that involves streaming, not matter how minor and even after its closed. A simple registry fix changed that stupid "feature".

 

[Guest]

There are many mistakes in the article...
1. Author mentioned about GigabitEthernet hub - because GbE is not based on Colission Detection (like 10 and 100Mbps Eth) it's not posible to build such device as GbE hub
2. Cable quality and electromagnetic interferences are not so important - for cable enght not longer as 20m is enugh cat.5 and transmission will be 100% error free
3. 1Gigabit per second is equal 125 000 000 bytes per secon but it's equal ~118Megabytes per second (because in telecommunication 1Kilo is 1000 but in IT world it's 1024
4. Network speed in the context of GbE means how many bits are transmitted through the network in the time period (1s) but (what is cruicial) not on application layer but much more lower - ethernet layer. Please note that there are many overheads, preamble, crc controls fields, interframe gaps etc. which must be transmitted so it's definitelly decrease the "room" (bandwidth) for user's traffic.
5. Of course, considering desktops the avrage performance of drives are below the GbE speed but time changes and in one or two years the performance SSDs will be the standard equipment and then even GbE wuldn't be enugh. On the other hand - even if your file transfer is faster "only" five times (in comparison to 100Mbps) it's reallly worth to consider such investment.

 

[Guest]

I do not think gig-e came be found on a hub device. If that is the case the following quote is invalid: "is a gigabit-capable hub. Of course, a switch or router would be preferable, since a hub is a somewhat "dumb" device that shares all of the network traffic bandwidth at every one of its ports, which can cause a lot of data collisions and, therefore, slower network speeds."

 

[hagi607]

I work for a startup called Teranetics, one of the things we do is make 10G-BaseT phy’s. A phy device does not have multiple speeds like a DSL modem, it will always operate at its full rated speed be it at 1 meter or 100 meters. Pretty much any time you get a link you’ll be able to test a phy’s speed. That’s why the graphs didn’t show much difference between the different cable length and with/without EMI. Also, the MAC side of a NIC/switch could have major bottlenecks. Usually the phy and mac vendor are different.

The cable is only important if you are trying to link at unrealistic lengths of 100+ meters, so long as you get a link the speed should be set though. Phy’s go through training after Auto-negotiation to see if they can support a link or not. For 10G you'll need a Cat6A or Cat7 cable to go 100 meters; with Cat5e and Cat6 you’ll get only around 55meters but it will still operate at the 10G speed.

EMI can cause CRC errors, if you have good hardware your phy should be able to cancel out these signals and link even beyond 100+ meters. In general, if you have a really bad cable and/or really bad EMI you should expect packet errors and can even dropped links.

For more accurate benchmarks you need to use special network traffic programs like Lanconf from Intel or if you got ~$60k lying around you can buy a Spirent or Ixia packet generator. To generate real EMI you’ll need a signal generator and a scope would be good to confirm the EMI.

You probably were getting the full 125MB rate but you did not take into account the packet transfer protocol around the bytes you were sending. Some possible better tests would be to see how far you could link and to see how much throughput you can get on different switches. These are all just for bragging rights though and don’t really have any real world advantages.

 

[Mosswalker]

@spectrewind ... he wrote 100 meter's. Learn to read before you let your pocket protector twist into a knot.

 

[Guest]

@all article apologist

Either this is an enthusiast website or not.

This was essentially an entire article just to say the transfer speed of a standard hard drive is slower than the speed of a gigabit network card.

Next up "Dude why is my DVD drive slower than my hard drive"

But don't worry, if the "noob" articles keep coming, the negative comments will eventually go away (taking the enthusiast with them)

 

[entrepemenour]

yal' such negative nancy's just offer corrections and insight help a feller out'

 

[shurcooL]

Isn't there an error in the article?

A gigabit network implies 1'000'000'000 bits per second transfer rate. 1'000'000'000 bits / 8 = 125'000'000 bytes.

125'000'000 bytes / 1024 = 122'070.3 kilobytes.

122'070.3 kilobytes / 1024 = 119.2 megabytes.

In other words, a gigabit network's theoretical speed limit is 119.2 megabytes per second, not 125.

 

[Milleman]

Should be cool to test different network card and switches, with the RAM-to-RAM disk test. That would give some more indications on what actually limits the theretical limits from the physical. Please make that as a follow up to this very interesting article!