Question File Copying in Windows stops and starts, V-shaped ramps In transfer speed ?

[jtyubv]

I have two Windows 10 computers directly networked together, but when I copy a single, huge file (20GB) between them, I get performance that looks like this:

For some reason, after a certain period of time, the copy speed falls off a cliff and then ramps back up. I can't really tell what's happening. When I copy the file over a gigabit or 2.5 gigabit connection, I never observe this behavior. But when the connection is 750 MB/sec (first example) or 2 GB/sec (second example), I see this.

I guess it could be the SSD write caches filling up, but I thought that would look different, something like this for example:

Given that's not what I'm seeing, and considering I see the same behavior no matter which pair of SSDs I'm copying between (Samsung 980 Pro, Samsung 970 Evo +, TeamGroup A440 Pro, Samsung MZVLQ512HBLU), I suspect it's something else. So maybe it's a Windows problem, or maybe it's related to how my network adapters are configured. I'm not sure.

I've seen this with two networking setups; two Intel XL710-XDA2 cards connected with a QSFP+ cable, and an OWC 10G Ethernet card connected to a Mellanox Connect-X2 card using an SFP+ to ethernet adapter. Both setups gave me file copies with V shaped dips like that.

Maybe there is a similarity in how the adapters are configured by default. Maybe there's a certain setting that could cause this? Any of these, maybe?

One computer has a Ryzen 7950X CPU and the other has an Intel i7-9750H CPU, in case that's relevant. Both are running Windows 10 21H2 (build 19044).

Any ideas what causes this behavior, and what I should try changing to fix it?

EDIT:

I tested copying the same 20GB file directly between the SSDs on the same system (so not over the network). They copied at a constant 1.8 to 2.2 GB/sec (depending on which pair of SSDs I was testing) the entire time, over the entire transfer. So I'm pretty sure this is networking related, and not SSD cache related.

 

[Darkbreeze]

Most likely it is multiple things. For one it is the caches filling up and emptying. For another, it may also be the controller getting hot and slowing down, then cooling off and speeding back up again. So in other words, controller throttling and DRAM caching.

How much RAM do these systems have in them? Have you monitored your memory usage to see if that might also be a chokepoint?

Not being straight SFP+, and funneling down to Ethernet speeds is probably having an effect as well. Consider that your NVME SSDs can do something like theoretical 1Gbps per lane, compared to Ethernet being just 1Gbps or perhaps 2Gbps if that's your hardware. Being straight 10Gbps SFP+ would certainly eliminate that aspect if at both ends. Unless I'm misunderstanding your configuration and hardware.

I'm sure one of our more network oriented guys like @bill001g, @kanewolf or @USAFRet would be able to give you more specific assistance but I'm betting it's likely one of those things if not a combination of them all.

Also, you say you don't see that when you use certain types of connections, but, when you do so is your maximum transfer speed the same peak or sustained as it is when using the hardware you are seeing that behavior happen with?

 

[dwd999]

I have two Windows 10 computers directly networked together, but when I copy a single, huge file (20GB) between them, I get performance that looks like this:

For some reason, after a certain period of time, the copy speed falls off a cliff and then ramps back up. I can't really tell what's happening. When I copy the file over a gigabit or 2.5 gigabit connection, I never observe this behavior. But when the connection is 750 MB/sec (first example) or 2 GB/sec (second example), I see this.

I guess it could be the SSD write caches filling up, but I thought that would look different, something like this for example:

Given that's not what I'm seeing, and considering I see the same behavior no matter which pair of SSDs I'm copying between (Samsung 980 Pro, Samsung 970 Evo +, TeamGroup A440 Pro, Samsung MZVLQ512HBLU), I suspect it's something else. So maybe it's a Windows problem, or maybe it's related to how my network adapters are configured. I'm not sure.

I've seen this with two networking setups; two Intel XL710-XDA2 cards connected with a QSFP+ cable, and an OWC 10G Ethernet card connected to a Mellanox Connect-X2 card using an SFP+ to ethernet adapter. Both setups gave me file copies with V shaped dips like that.

Maybe there is a similarity in how the adapters are configured by default. Maybe there's a certain setting that could cause this? Any of these, maybe?

One computer has a Ryzen 7950X CPU and the other has an Intel i7-9750H CPU, in case that's relevant. Both are running Windows 10 21H2 (build 19044).

Any ideas what causes this behavior, and what I should try changing to fix it?

Since you are using some Samsung drives, are you using Samsung's Magician software? If yes, what performance optimization and over provisioning options are you using, especially rapid mode?

 

[Darkbreeze]

I don't think over provisioning would have anything to do with that, but the use of Rapid Mode certainly might since that uses RAM and might be filling up just like any DRAM cache.

 

[jtyubv]

Most likely it is multiple things. For one it is the caches filling up and emptying. For another, it may also be the controller getting hot and slowing down, then cooling off and speeding back up again. So in other words, controller throttling and DRAM caching.

How much RAM do these systems have in them? Have you monitored your memory usage to see if that might also be a chokepoint?

Not being straight SFP+, and funneling down to Ethernet speeds is probably having an effect as well. Consider that your NVME SSDs can do something like theoretical 1Gbps per lane, compared to Ethernet being just 1Gbps or perhaps 2Gbps if that's your hardware. Being straight 10Gbps SFP+ would certainly eliminate that aspect if at both ends. Unless I'm misunderstanding your configuration and hardware.

I'm sure one of our more network oriented guys like @bill001g, @kanewolf or @USAFRet would be able to give you more specific assistance but I'm betting it's likely one of those things if not a combination of them all.

Both systems have 64 GB RAM (the AMD is DDR5 4800 MHz and the Intel is DDR4 3200 MHz). Using ethernet in the second networking setup (the OWC and Mellanox cards) could be part of it, but I still observe this even with a direct QSFP+ connection in the first setup (the intel cards).

I decided to test by just copying the same 20GB file directly between the SSDs on the same system (not over the network) and they all copy at a consistent 1.8 to 2.2 GB/sec; there is no v shape. So it is related to the networking setup somehow, and not the SSD write caches.

 

[jtyubv]

Since you are using some Samsung drives, are you using Samsung's Magician software? If yes, what performance optimization and over provisioning options are you using, especially rapid mode?

I do not have Samsung Magician installed on either system, so these would all be using their default configurations.

 

[Darkbreeze]

Both systems have 64 GB RAM (the AMD is DDR5 4800 MHz and the Intel is DDR4 3200 MHz). Using ethernet in the second networking setup (the OWC and Mellanox cards) could be part of it, but I still observe this even with a direct QSFP+ connection in the first setup (the intel cards).

I decided to test by just copying the same 20GB file directly between the SSDs on the same system (not over the network) and they all copy at a consistent 1.8 to 2.2 GB/sec; there is no v shape. So it is related to the networking setup somehow, and not the SSD write caches.

Yes, this was the next thing I was going to ask you to do was direct transfers between drives with no network between them. So, definitely related to the network hardware. I cannot tell you specifically what though, but I'm pretty sure one of the folks I mentioned earlier will be able to if and when they see the call out and are able to chime in.

 

[kanewolf]

Most likely it is multiple things. For one it is the caches filling up and emptying. For another, it may also be the controller getting hot and slowing down, then cooling off and speeding back up again. So in other words, controller throttling and DRAM caching.

How much RAM do these systems have in them? Have you monitored your memory usage to see if that might also be a chokepoint?

Not being straight SFP+, and funneling down to Ethernet speeds is probably having an effect as well. Consider that your NVME SSDs can do something like theoretical 1Gbps per lane, compared to Ethernet being just 1Gbps or perhaps 2Gbps if that's your hardware. Being straight 10Gbps SFP+ would certainly eliminate that aspect if at both ends. Unless I'm misunderstanding your configuration and hardware.

I'm sure one of our more network oriented guys like @bill001g, @kanewolf or @USAFRet would be able to give you more specific assistance but I'm betting it's likely one of those things if not a combination of them all.

Also, you say you don't see that when you use certain types of connections, but, when you do so is your maximum transfer speed the same peak or sustained as it is when using the hardware you are seeing that behavior happen with?

To eliminate the network as the limit, you should use iPerf between the two hosts. That uses only the network part of the transfer path.

 

[Darkbreeze]

See, I have no familiarity with that utility but I will be educating myself about it this evening.

 

[jtyubv]

To eliminate the network as the limit, you should use iPerf between the two hosts. That uses only the network part of the transfer path.

What iperf command do you suggest I use? I'm not sure how to test it continuously for an extended period of time. When I do the default tests (the ones the last only a couple seconds) the performance is fine, but I need it to go continuously for a few minutes to simulate the copying of a large file.

 

[kanewolf]

What iperf command do you suggest I use? I'm not sure how to test it continuously for an extended period of time. When I do the default tests (the ones the last only a couple seconds) the performance is fine, but I need it to go continuously for a few minutes to simulate the copying of a large file.

Maybe read this tutorial for measuring high performance networks -- https://www.datapacket.com/blog/10gbps-network-bandwidth-test-iperf-tutorial

 

[jtyubv]

Maybe read this tutorial for measuring high performance networks -- https://www.datapacket.com/blog/10gbps-network-bandwidth-test-iperf-tutorial

I see similar performance with iperf. Using the -t parameter to do a 60 second test, here are the results. Please note one PC is only running with x2 lanes due to a limitation on my motherboard; so the maximum of 760 MB/sec (rather than 40 gigabit) is expected. However, you can still see the wave pattern even in iperf, where the performance periodically rises and falls.

Spoiler: Iperf Results

PS C:\Users\JT> iperf3 -c 192.168.1.221 -P 2 -t 60
Connecting to host 192.168.1.221, port 5201
[ 4] local 192.168.1.210 port 51001 connected to 192.168.1.221 port 5201
[ 6] local 192.168.1.210 port 51002 connected to 192.168.1.221 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 384 MBytes 3.22 Gbits/sec
[ 6] 0.00-1.00 sec 378 MBytes 3.17 Gbits/sec
[SUM] 0.00-1.00 sec 763 MBytes 6.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 62.0 MBytes 519 Mbits/sec
[ 6] 1.00-2.00 sec 61.1 MBytes 512 Mbits/sec
[SUM] 1.00-2.00 sec 123 MBytes 1.03 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 19.9 MBytes 167 Mbits/sec
[ 6] 2.00-3.00 sec 17.6 MBytes 148 Mbits/sec
[SUM] 2.00-3.00 sec 37.5 MBytes 315 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 21.0 MBytes 176 Mbits/sec
[ 6] 3.00-4.00 sec 18.2 MBytes 153 Mbits/sec
[SUM] 3.00-4.00 sec 39.2 MBytes 330 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 4.00-5.00 sec 340 MBytes 2.85 Gbits/sec
[ 6] 4.00-5.00 sec 336 MBytes 2.82 Gbits/sec
[SUM] 4.00-5.00 sec 676 MBytes 5.67 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 5.00-6.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 5.00-6.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 5.00-6.00 sec 768 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 6.00-7.00 sec 376 MBytes 3.15 Gbits/sec
[ 6] 6.00-7.00 sec 372 MBytes 3.12 Gbits/sec
[SUM] 6.00-7.00 sec 749 MBytes 6.28 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 7.00-8.00 sec 387 MBytes 3.25 Gbits/sec
[ 6] 7.00-8.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 7.00-8.00 sec 768 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 8.00-9.00 sec 22.6 MBytes 189 Mbits/sec
[ 6] 8.00-9.00 sec 20.6 MBytes 173 Mbits/sec
[SUM] 8.00-9.00 sec 43.2 MBytes 362 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.01 sec 53.9 MBytes 449 Mbits/sec
[ 6] 9.00-10.01 sec 50.8 MBytes 423 Mbits/sec
[SUM] 9.00-10.01 sec 105 MBytes 871 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 10.01-11.01 sec 22.8 MBytes 190 Mbits/sec
[ 6] 10.01-11.01 sec 20.9 MBytes 175 Mbits/sec
[SUM] 10.01-11.01 sec 43.6 MBytes 365 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 11.01-12.00 sec 144 MBytes 1.22 Gbits/sec
[ 6] 11.01-12.00 sec 140 MBytes 1.18 Gbits/sec
[SUM] 11.01-12.00 sec 283 MBytes 2.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 12.00-13.00 sec 386 MBytes 3.23 Gbits/sec
[ 6] 12.00-13.00 sec 383 MBytes 3.21 Gbits/sec
[SUM] 12.00-13.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 13.00-14.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 13.00-14.00 sec 382 MBytes 3.21 Gbits/sec
[SUM] 13.00-14.00 sec 769 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 14.00-15.00 sec 387 MBytes 3.25 Gbits/sec
[ 6] 14.00-15.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 14.00-15.00 sec 769 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 15.00-16.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 15.00-16.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 15.00-16.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 16.00-17.00 sec 27.1 MBytes 227 Mbits/sec
[ 6] 16.00-17.00 sec 26.0 MBytes 217 Mbits/sec
[SUM] 16.00-17.00 sec 53.1 MBytes 444 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 17.00-18.01 sec 19.8 MBytes 165 Mbits/sec
[ 6] 17.00-18.01 sec 18.1 MBytes 151 Mbits/sec
[SUM] 17.00-18.01 sec 37.9 MBytes 316 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 18.01-19.00 sec 57.5 MBytes 485 Mbits/sec
[ 6] 18.01-19.00 sec 54.1 MBytes 456 Mbits/sec
[SUM] 18.01-19.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 19.00-20.00 sec 227 MBytes 1.91 Gbits/sec
[ 6] 19.00-20.00 sec 221 MBytes 1.86 Gbits/sec
[SUM] 19.00-20.00 sec 448 MBytes 3.77 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 20.00-21.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 20.00-21.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 20.00-21.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 21.00-22.00 sec 388 MBytes 3.25 Gbits/sec
[ 6] 21.00-22.00 sec 380 MBytes 3.19 Gbits/sec
[SUM] 21.00-22.00 sec 768 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 22.00-23.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 22.00-23.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 22.00-23.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 23.00-24.00 sec 176 MBytes 1.48 Gbits/sec
[ 6] 23.00-24.00 sec 174 MBytes 1.46 Gbits/sec
[SUM] 23.00-24.00 sec 350 MBytes 2.93 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 24.00-25.01 sec 116 MBytes 961 Mbits/sec
[ 6] 24.00-25.01 sec 112 MBytes 929 Mbits/sec
[SUM] 24.00-25.01 sec 228 MBytes 1.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 25.01-26.01 sec 88.9 MBytes 751 Mbits/sec
[ 6] 25.01-26.01 sec 85.2 MBytes 720 Mbits/sec
[SUM] 25.01-26.01 sec 174 MBytes 1.47 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 26.01-27.00 sec 20.2 MBytes 171 Mbits/sec
[ 6] 26.01-27.00 sec 17.5 MBytes 148 Mbits/sec
[SUM] 26.01-27.00 sec 37.8 MBytes 319 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 27.00-28.00 sec 352 MBytes 2.96 Gbits/sec
[ 6] 27.00-28.00 sec 346 MBytes 2.90 Gbits/sec
[SUM] 27.00-28.00 sec 698 MBytes 5.86 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 28.00-29.00 sec 205 MBytes 1.72 Gbits/sec
[ 6] 28.00-29.00 sec 201 MBytes 1.68 Gbits/sec
[SUM] 28.00-29.00 sec 406 MBytes 3.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 29.00-30.01 sec 21.1 MBytes 175 Mbits/sec
[ 6] 29.00-30.01 sec 19.8 MBytes 164 Mbits/sec
[SUM] 29.00-30.01 sec 40.9 MBytes 339 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 30.01-31.01 sec 21.5 MBytes 180 Mbits/sec
[ 6] 30.01-31.01 sec 19.2 MBytes 161 Mbits/sec
[SUM] 30.01-31.01 sec 40.8 MBytes 341 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 31.01-32.00 sec 138 MBytes 1.18 Gbits/sec
[ 6] 31.01-32.00 sec 134 MBytes 1.14 Gbits/sec
[SUM] 31.01-32.00 sec 273 MBytes 2.32 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 32.00-33.00 sec 373 MBytes 3.13 Gbits/sec
[ 6] 32.00-33.00 sec 370 MBytes 3.10 Gbits/sec
[SUM] 32.00-33.00 sec 743 MBytes 6.23 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 33.00-34.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 33.00-34.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 33.00-34.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 34.00-35.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 34.00-35.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 34.00-35.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 35.00-36.01 sec 303 MBytes 2.52 Gbits/sec
[ 6] 35.00-36.01 sec 298 MBytes 2.47 Gbits/sec
[SUM] 35.00-36.01 sec 601 MBytes 4.99 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 36.01-37.01 sec 22.9 MBytes 192 Mbits/sec
[ 6] 36.01-37.01 sec 21.0 MBytes 177 Mbits/sec
[SUM] 36.01-37.01 sec 43.9 MBytes 369 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 37.01-38.01 sec 22.0 MBytes 183 Mbits/sec
[ 6] 37.01-38.01 sec 20.5 MBytes 171 Mbits/sec
[SUM] 37.01-38.01 sec 42.5 MBytes 354 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 38.01-39.01 sec 22.6 MBytes 191 Mbits/sec
[ 6] 38.01-39.01 sec 20.5 MBytes 173 Mbits/sec
[SUM] 38.01-39.01 sec 43.1 MBytes 363 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 39.01-40.00 sec 272 MBytes 2.30 Gbits/sec
[ 6] 39.01-40.00 sec 266 MBytes 2.26 Gbits/sec
[SUM] 39.01-40.00 sec 538 MBytes 4.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 40.00-41.00 sec 387 MBytes 3.25 Gbits/sec
[ 6] 40.00-41.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 40.00-41.00 sec 769 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 41.00-42.00 sec 362 MBytes 3.03 Gbits/sec
[ 6] 41.00-42.00 sec 357 MBytes 2.99 Gbits/sec
[SUM] 41.00-42.00 sec 719 MBytes 6.02 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 42.00-43.00 sec 381 MBytes 3.20 Gbits/sec
[ 6] 42.00-43.00 sec 375 MBytes 3.15 Gbits/sec
[SUM] 42.00-43.00 sec 756 MBytes 6.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 43.00-44.01 sec 158 MBytes 1.32 Gbits/sec
[ 6] 43.00-44.01 sec 156 MBytes 1.30 Gbits/sec
[SUM] 43.00-44.01 sec 315 MBytes 2.62 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 44.01-45.00 sec 21.1 MBytes 178 Mbits/sec
[ 6] 44.01-45.00 sec 19.1 MBytes 161 Mbits/sec
[SUM] 44.01-45.00 sec 40.2 MBytes 339 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 45.00-46.00 sec 120 MBytes 1.01 Gbits/sec
[ 6] 45.00-46.00 sec 114 MBytes 963 Mbits/sec
[SUM] 45.00-46.00 sec 234 MBytes 1.97 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 46.00-47.00 sec 314 MBytes 2.63 Gbits/sec
[ 6] 46.00-47.00 sec 311 MBytes 2.61 Gbits/sec
[SUM] 46.00-47.00 sec 625 MBytes 5.24 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 47.00-48.01 sec 19.4 MBytes 161 Mbits/sec
[ 6] 47.00-48.01 sec 18.0 MBytes 149 Mbits/sec
[SUM] 47.00-48.01 sec 37.4 MBytes 310 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 48.01-49.00 sec 145 MBytes 1.23 Gbits/sec
[ 6] 48.01-49.00 sec 140 MBytes 1.19 Gbits/sec
[SUM] 48.01-49.00 sec 285 MBytes 2.42 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 49.00-50.00 sec 334 MBytes 2.80 Gbits/sec
[ 6] 49.00-50.00 sec 331 MBytes 2.78 Gbits/sec
[SUM] 49.00-50.00 sec 665 MBytes 5.58 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 50.00-51.01 sec 354 MBytes 2.95 Gbits/sec
[ 6] 50.00-51.01 sec 349 MBytes 2.91 Gbits/sec
[SUM] 50.00-51.01 sec 703 MBytes 5.87 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 51.01-52.01 sec 20.8 MBytes 173 Mbits/sec
[ 6] 51.01-52.01 sec 19.0 MBytes 158 Mbits/sec
[SUM] 51.01-52.01 sec 39.8 MBytes 331 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 52.01-53.01 sec 21.2 MBytes 178 Mbits/sec
[ 6] 52.01-53.01 sec 18.9 MBytes 158 Mbits/sec
[SUM] 52.01-53.01 sec 40.1 MBytes 336 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 53.01-54.01 sec 19.5 MBytes 164 Mbits/sec
[ 6] 53.01-54.01 sec 17.5 MBytes 147 Mbits/sec
[SUM] 53.01-54.01 sec 37.0 MBytes 311 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 54.01-55.00 sec 142 MBytes 1.21 Gbits/sec
[ 6] 54.01-55.00 sec 140 MBytes 1.19 Gbits/sec
[SUM] 54.01-55.00 sec 283 MBytes 2.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 55.00-56.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 55.00-56.00 sec 382 MBytes 3.21 Gbits/sec
[SUM] 55.00-56.00 sec 769 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 56.00-57.00 sec 171 MBytes 1.43 Gbits/sec
[ 6] 56.00-57.00 sec 167 MBytes 1.40 Gbits/sec
[SUM] 56.00-57.00 sec 338 MBytes 2.84 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 57.00-58.00 sec 26.0 MBytes 218 Mbits/sec
[ 6] 57.00-58.00 sec 24.4 MBytes 205 Mbits/sec
[SUM] 57.00-58.00 sec 50.4 MBytes 423 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 58.00-59.00 sec 383 MBytes 3.21 Gbits/sec
[ 6] 58.00-59.00 sec 378 MBytes 3.17 Gbits/sec
[SUM] 58.00-59.00 sec 761 MBytes 6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 59.00-60.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 59.00-60.00 sec 382 MBytes 3.21 Gbits/sec
[SUM] 59.00-60.00 sec 768 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-60.00 sec 12.0 GBytes 1.72 Gbits/sec sender
[ 4] 0.00-60.00 sec 12.0 GBytes 1.72 Gbits/sec receiver
[ 6] 0.00-60.00 sec 11.8 GBytes 1.69 Gbits/sec sender
[ 6] 0.00-60.00 sec 11.8 GBytes 1.69 Gbits/sec receiver
[SUM] 0.00-60.00 sec 23.9 GBytes 3.42 Gbits/sec sender
[SUM] 0.00-60.00 sec 23.9 GBytes 3.42 Gbits/sec receiver

 

[kanewolf]

I see similar performance with iperf. Using the -t parameter to do a 60 second test, here are the results. Please note one PC is only running with x2 lanes due to a limitation on my motherboard; so the maximum of 760 MB/sec (rather than 40 gigabit) is expected. However, you can still see the wave pattern even in iperf, where the performance periodically rises and falls.

Spoiler: Iperf Results

PS C:\Users\JT> iperf3 -c 192.168.1.221 -P 2 -t 60
Connecting to host 192.168.1.221, port 5201
[ 4] local 192.168.1.210 port 51001 connected to 192.168.1.221 port 5201
[ 6] local 192.168.1.210 port 51002 connected to 192.168.1.221 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 384 MBytes 3.22 Gbits/sec
[ 6] 0.00-1.00 sec 378 MBytes 3.17 Gbits/sec
[SUM] 0.00-1.00 sec 763 MBytes 6.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 62.0 MBytes 519 Mbits/sec
[ 6] 1.00-2.00 sec 61.1 MBytes 512 Mbits/sec
[SUM] 1.00-2.00 sec 123 MBytes 1.03 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 19.9 MBytes 167 Mbits/sec
[ 6] 2.00-3.00 sec 17.6 MBytes 148 Mbits/sec
[SUM] 2.00-3.00 sec 37.5 MBytes 315 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 21.0 MBytes 176 Mbits/sec
[ 6] 3.00-4.00 sec 18.2 MBytes 153 Mbits/sec
[SUM] 3.00-4.00 sec 39.2 MBytes 330 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 4.00-5.00 sec 340 MBytes 2.85 Gbits/sec
[ 6] 4.00-5.00 sec 336 MBytes 2.82 Gbits/sec
[SUM] 4.00-5.00 sec 676 MBytes 5.67 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 5.00-6.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 5.00-6.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 5.00-6.00 sec 768 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 6.00-7.00 sec 376 MBytes 3.15 Gbits/sec
[ 6] 6.00-7.00 sec 372 MBytes 3.12 Gbits/sec
[SUM] 6.00-7.00 sec 749 MBytes 6.28 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 7.00-8.00 sec 387 MBytes 3.25 Gbits/sec
[ 6] 7.00-8.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 7.00-8.00 sec 768 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 8.00-9.00 sec 22.6 MBytes 189 Mbits/sec
[ 6] 8.00-9.00 sec 20.6 MBytes 173 Mbits/sec
[SUM] 8.00-9.00 sec 43.2 MBytes 362 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.01 sec 53.9 MBytes 449 Mbits/sec
[ 6] 9.00-10.01 sec 50.8 MBytes 423 Mbits/sec
[SUM] 9.00-10.01 sec 105 MBytes 871 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 10.01-11.01 sec 22.8 MBytes 190 Mbits/sec
[ 6] 10.01-11.01 sec 20.9 MBytes 175 Mbits/sec
[SUM] 10.01-11.01 sec 43.6 MBytes 365 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 11.01-12.00 sec 144 MBytes 1.22 Gbits/sec
[ 6] 11.01-12.00 sec 140 MBytes 1.18 Gbits/sec
[SUM] 11.01-12.00 sec 283 MBytes 2.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 12.00-13.00 sec 386 MBytes 3.23 Gbits/sec
[ 6] 12.00-13.00 sec 383 MBytes 3.21 Gbits/sec
[SUM] 12.00-13.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 13.00-14.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 13.00-14.00 sec 382 MBytes 3.21 Gbits/sec
[SUM] 13.00-14.00 sec 769 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 14.00-15.00 sec 387 MBytes 3.25 Gbits/sec
[ 6] 14.00-15.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 14.00-15.00 sec 769 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 15.00-16.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 15.00-16.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 15.00-16.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 16.00-17.00 sec 27.1 MBytes 227 Mbits/sec
[ 6] 16.00-17.00 sec 26.0 MBytes 217 Mbits/sec
[SUM] 16.00-17.00 sec 53.1 MBytes 444 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 17.00-18.01 sec 19.8 MBytes 165 Mbits/sec
[ 6] 17.00-18.01 sec 18.1 MBytes 151 Mbits/sec
[SUM] 17.00-18.01 sec 37.9 MBytes 316 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 18.01-19.00 sec 57.5 MBytes 485 Mbits/sec
[ 6] 18.01-19.00 sec 54.1 MBytes 456 Mbits/sec
[SUM] 18.01-19.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 19.00-20.00 sec 227 MBytes 1.91 Gbits/sec
[ 6] 19.00-20.00 sec 221 MBytes 1.86 Gbits/sec
[SUM] 19.00-20.00 sec 448 MBytes 3.77 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 20.00-21.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 20.00-21.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 20.00-21.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 21.00-22.00 sec 388 MBytes 3.25 Gbits/sec
[ 6] 21.00-22.00 sec 380 MBytes 3.19 Gbits/sec
[SUM] 21.00-22.00 sec 768 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 22.00-23.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 22.00-23.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 22.00-23.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 23.00-24.00 sec 176 MBytes 1.48 Gbits/sec
[ 6] 23.00-24.00 sec 174 MBytes 1.46 Gbits/sec
[SUM] 23.00-24.00 sec 350 MBytes 2.93 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 24.00-25.01 sec 116 MBytes 961 Mbits/sec
[ 6] 24.00-25.01 sec 112 MBytes 929 Mbits/sec
[SUM] 24.00-25.01 sec 228 MBytes 1.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 25.01-26.01 sec 88.9 MBytes 751 Mbits/sec
[ 6] 25.01-26.01 sec 85.2 MBytes 720 Mbits/sec
[SUM] 25.01-26.01 sec 174 MBytes 1.47 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 26.01-27.00 sec 20.2 MBytes 171 Mbits/sec
[ 6] 26.01-27.00 sec 17.5 MBytes 148 Mbits/sec
[SUM] 26.01-27.00 sec 37.8 MBytes 319 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 27.00-28.00 sec 352 MBytes 2.96 Gbits/sec
[ 6] 27.00-28.00 sec 346 MBytes 2.90 Gbits/sec
[SUM] 27.00-28.00 sec 698 MBytes 5.86 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 28.00-29.00 sec 205 MBytes 1.72 Gbits/sec
[ 6] 28.00-29.00 sec 201 MBytes 1.68 Gbits/sec
[SUM] 28.00-29.00 sec 406 MBytes 3.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 29.00-30.01 sec 21.1 MBytes 175 Mbits/sec
[ 6] 29.00-30.01 sec 19.8 MBytes 164 Mbits/sec
[SUM] 29.00-30.01 sec 40.9 MBytes 339 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 30.01-31.01 sec 21.5 MBytes 180 Mbits/sec
[ 6] 30.01-31.01 sec 19.2 MBytes 161 Mbits/sec
[SUM] 30.01-31.01 sec 40.8 MBytes 341 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 31.01-32.00 sec 138 MBytes 1.18 Gbits/sec
[ 6] 31.01-32.00 sec 134 MBytes 1.14 Gbits/sec
[SUM] 31.01-32.00 sec 273 MBytes 2.32 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 32.00-33.00 sec 373 MBytes 3.13 Gbits/sec
[ 6] 32.00-33.00 sec 370 MBytes 3.10 Gbits/sec
[SUM] 32.00-33.00 sec 743 MBytes 6.23 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 33.00-34.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 33.00-34.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 33.00-34.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 34.00-35.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 34.00-35.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 34.00-35.00 sec 768 MBytes 6.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 35.00-36.01 sec 303 MBytes 2.52 Gbits/sec
[ 6] 35.00-36.01 sec 298 MBytes 2.47 Gbits/sec
[SUM] 35.00-36.01 sec 601 MBytes 4.99 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 36.01-37.01 sec 22.9 MBytes 192 Mbits/sec
[ 6] 36.01-37.01 sec 21.0 MBytes 177 Mbits/sec
[SUM] 36.01-37.01 sec 43.9 MBytes 369 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 37.01-38.01 sec 22.0 MBytes 183 Mbits/sec
[ 6] 37.01-38.01 sec 20.5 MBytes 171 Mbits/sec
[SUM] 37.01-38.01 sec 42.5 MBytes 354 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 38.01-39.01 sec 22.6 MBytes 191 Mbits/sec
[ 6] 38.01-39.01 sec 20.5 MBytes 173 Mbits/sec
[SUM] 38.01-39.01 sec 43.1 MBytes 363 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 39.01-40.00 sec 272 MBytes 2.30 Gbits/sec
[ 6] 39.01-40.00 sec 266 MBytes 2.26 Gbits/sec
[SUM] 39.01-40.00 sec 538 MBytes 4.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 40.00-41.00 sec 387 MBytes 3.25 Gbits/sec
[ 6] 40.00-41.00 sec 382 MBytes 3.20 Gbits/sec
[SUM] 40.00-41.00 sec 769 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 41.00-42.00 sec 362 MBytes 3.03 Gbits/sec
[ 6] 41.00-42.00 sec 357 MBytes 2.99 Gbits/sec
[SUM] 41.00-42.00 sec 719 MBytes 6.02 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 42.00-43.00 sec 381 MBytes 3.20 Gbits/sec
[ 6] 42.00-43.00 sec 375 MBytes 3.15 Gbits/sec
[SUM] 42.00-43.00 sec 756 MBytes 6.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 43.00-44.01 sec 158 MBytes 1.32 Gbits/sec
[ 6] 43.00-44.01 sec 156 MBytes 1.30 Gbits/sec
[SUM] 43.00-44.01 sec 315 MBytes 2.62 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 44.01-45.00 sec 21.1 MBytes 178 Mbits/sec
[ 6] 44.01-45.00 sec 19.1 MBytes 161 Mbits/sec
[SUM] 44.01-45.00 sec 40.2 MBytes 339 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 45.00-46.00 sec 120 MBytes 1.01 Gbits/sec
[ 6] 45.00-46.00 sec 114 MBytes 963 Mbits/sec
[SUM] 45.00-46.00 sec 234 MBytes 1.97 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 46.00-47.00 sec 314 MBytes 2.63 Gbits/sec
[ 6] 46.00-47.00 sec 311 MBytes 2.61 Gbits/sec
[SUM] 46.00-47.00 sec 625 MBytes 5.24 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 47.00-48.01 sec 19.4 MBytes 161 Mbits/sec
[ 6] 47.00-48.01 sec 18.0 MBytes 149 Mbits/sec
[SUM] 47.00-48.01 sec 37.4 MBytes 310 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 48.01-49.00 sec 145 MBytes 1.23 Gbits/sec
[ 6] 48.01-49.00 sec 140 MBytes 1.19 Gbits/sec
[SUM] 48.01-49.00 sec 285 MBytes 2.42 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 49.00-50.00 sec 334 MBytes 2.80 Gbits/sec
[ 6] 49.00-50.00 sec 331 MBytes 2.78 Gbits/sec
[SUM] 49.00-50.00 sec 665 MBytes 5.58 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 50.00-51.01 sec 354 MBytes 2.95 Gbits/sec
[ 6] 50.00-51.01 sec 349 MBytes 2.91 Gbits/sec
[SUM] 50.00-51.01 sec 703 MBytes 5.87 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 51.01-52.01 sec 20.8 MBytes 173 Mbits/sec
[ 6] 51.01-52.01 sec 19.0 MBytes 158 Mbits/sec
[SUM] 51.01-52.01 sec 39.8 MBytes 331 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 52.01-53.01 sec 21.2 MBytes 178 Mbits/sec
[ 6] 52.01-53.01 sec 18.9 MBytes 158 Mbits/sec
[SUM] 52.01-53.01 sec 40.1 MBytes 336 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 53.01-54.01 sec 19.5 MBytes 164 Mbits/sec
[ 6] 53.01-54.01 sec 17.5 MBytes 147 Mbits/sec
[SUM] 53.01-54.01 sec 37.0 MBytes 311 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 54.01-55.00 sec 142 MBytes 1.21 Gbits/sec
[ 6] 54.01-55.00 sec 140 MBytes 1.19 Gbits/sec
[SUM] 54.01-55.00 sec 283 MBytes 2.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 55.00-56.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 55.00-56.00 sec 382 MBytes 3.21 Gbits/sec
[SUM] 55.00-56.00 sec 769 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 56.00-57.00 sec 171 MBytes 1.43 Gbits/sec
[ 6] 56.00-57.00 sec 167 MBytes 1.40 Gbits/sec
[SUM] 56.00-57.00 sec 338 MBytes 2.84 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 57.00-58.00 sec 26.0 MBytes 218 Mbits/sec
[ 6] 57.00-58.00 sec 24.4 MBytes 205 Mbits/sec
[SUM] 57.00-58.00 sec 50.4 MBytes 423 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 58.00-59.00 sec 383 MBytes 3.21 Gbits/sec
[ 6] 58.00-59.00 sec 378 MBytes 3.17 Gbits/sec
[SUM] 58.00-59.00 sec 761 MBytes 6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 59.00-60.00 sec 386 MBytes 3.24 Gbits/sec
[ 6] 59.00-60.00 sec 382 MBytes 3.21 Gbits/sec
[SUM] 59.00-60.00 sec 768 MBytes 6.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-60.00 sec 12.0 GBytes 1.72 Gbits/sec sender
[ 4] 0.00-60.00 sec 12.0 GBytes 1.72 Gbits/sec receiver
[ 6] 0.00-60.00 sec 11.8 GBytes 1.69 Gbits/sec sender
[ 6] 0.00-60.00 sec 11.8 GBytes 1.69 Gbits/sec receiver
[SUM] 0.00-60.00 sec 23.9 GBytes 3.42 Gbits/sec sender
[SUM] 0.00-60.00 sec 23.9 GBytes 3.42 Gbits/sec receiver

First question I might have is the power profile set to high performance on the i7 ? An "H" CPU is a low power version, so maybe there is some power saving settings that are biting you.

I think, until you can get your iPerf data more consistent, you can focus exclusively on that problem.

Have you tried iPerf the opposite direction to see if it has the same inconsistencies ?

 

[Darkbreeze]

I have to admit, this is interesting to me, and relevant, because I have seen the same issues between my primary PC and my NAS, both of which have 10Gbps SFP+ hardware, so I'm going to continue following along because it feels like there is something to learn here along the way.

 

[bill001g]

This is very strange since it can't just be a simple driver issue if it happens on multiple cards.
Although you likely will not see a difference try using the UDP option with a -b of 10000. I have never tried this on 10gbit nics.
There is a lot of overhead in TCP but it is still tiny compared to a cpu ability to process. Most times you use the UDP test when you are concerned about the latency since UDP does not acknowledge packets. Not really a issue with a direct in house fiber connection.

With IPERF you have eliminated the most common things that cause issue with transfers. The disk systems and all the cpu overhead. I doubt you have a cpu or memory bottleneck with iperf. I also assume you are only running IPERF and nothing else on the machines.

It has to be one of the very many setting buried deep in windows. These cards are more commonly run in a linux os or on microsoft server versions of the OS. If it caused this issue in a data centers you would see lots of screaming.

I guess to a point you could try linux boot image but all this would do is confirm it was some setting in windows. If linux has a similar issue you would then be looking at even more bizarre limitations related to things like PCIE lanes and which slots you are using.

I suspect you do not find more on this because it is only recently has become somewhat affordable to run fast nic cards. Most people using them are putting them in server class machines and running server based OS.

 

[Ralston18]

May also be worth doing some observing via Process Explorer (Microsoft, free).

https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer

Start with the Summary view and then look for a process behaving in the same manner or pattern.

Just a thought....

 

[Darkbreeze]

I guess to a point you could try linux boot image but all this would do is confirm it was some setting in windows.

Yes, but confirming it was some setting in Windows would go a long way towards being able to identify what is causing it though, so that's probably a pretty decent idea, because if nothing changes then you know it's not a Windows setting.