NFS vs CIFS speeds

[ouroborus]

I have a Linux server and a Win7 client with 1GbE between. I get about 39MB/s over NFS and 58MB/s over CIFS. Do these numbers look right? Is there something I can do to improve them outside of installing a fatter pipe?

 

[RealBeast]

It depends on a number of factors beyond your cabling. I expect that both adapters are gigabit capable to hit the speeds that you have now, but the storage device at each end is also a potential bottleneck.

From SSD to SSD over gigabit with two Win7 machines (using NTFS) I average around 880Mbps, pretty much at the limit of gigabit Ethernet once you deduct overhead. When I write to a RAID 6 array the speed is much slower due to the poor write performance of RAID 6.

 

[ouroborus]

It depends on a number of factors beyond your cabling. I expect that both adapters are gigabit capable to hit the speeds that you have now, but the storage device at each end is also a potential bottleneck.

From SSD to SSD over gigabit with two Win7 machines (using NTFS) I average around 880Mbps, pretty much at the limit of gigabit Ethernet once you deduct overhead. When I write to a RAID 6 array the speed is much slower due to the poor write performance of RAID 6.

The sustained speeds for the drive is listed at 171 MB/s (WD WD4003FZEX, specs PDF). I'm not sure how much of that is actually achievable, but still it's over 3x what I'm getting over the wire. And I'm still only getting about half your read speed.

 

[bill001g]

It is unlikely it is your network connection causing a delay and there really is no way to increase it over 1g unless you want to spend the huge money to buy equipment with 10g ports.

You can test your network speed with a old line mode tool called IPERF. This is a very light weight tool that runs totally from memory so your disk,cpu, memory etc have little to no impact on the speed.

What I suspect you will find is you get good numbers with iperf which does not fix your problem but it does point to something in the file system. Be very sure you test correctly you will see massively different transfer rates if you copy a directory of many tiny files than if you copy 1 large file.

 

[ouroborus]

It is unlikely it is your network connection causing a delay and there really is no way to increase it over 1g unless you want to spend the huge money to buy equipment with 10g ports.

You can test your network speed with a old line mode tool called IPERF. This is a very light weight tool that runs totally from memory so your disk,cpu, memory etc have little to no impact on the speed.

What I suspect you will find is you get good numbers with iperf which does not fix your problem but it does point to something in the file system. Be very sure you test correctly you will see massively different transfer rates if you copy a directory of many tiny files than if you copy 1 large file.

I've used iperf previously as you describe. I don't remember the numbers off the top of my head but I do recall that they were different depending on which direction I was testing in. (I'll run the tests again and update with those numbers.)

Update:
Windows to Linux: 400Mbit/s (Fluctuates a bit, between 350 and 500, probably due to other things running)
Linux to Windows: 480Mbit/s (Pretty steady, between 462 and 481)

These seem inline with what I get with CIFS.

 

[bill001g]

You generally should be able to get more than that. It normally gets over 800m and you get push it over 900m by messing with things like jumbo frames.

Unless you happen to have a CPU bottleneck which I doubt it has to be something strange in the drivers for the nic

 

[Astara]

I have a Linux server and a Win7 client with 1GbE between. I get about 39MB/s over NFS and 58MB/s over CIFS. Do these numbers look right? Is there something I can do to improve them outside of installing a fatter pipe?

I can't say much for NFS other than I found it slower than CIFS on more than one occasion when I tested it.

For CIFS, my max write was 125M(10**6)B/s and 119MB/s read between Win7x64 and a linux samba server. With XP, (both R/W were under 85MB/s (but foget the numbers).

One thing that might have be helping me is using 9k (jumbo ethernet frames).

I upgraded the connection between my win desktop and my server to use a 10Gb Intel card on each end (having no router between them saves on equipment costs). On those machines, I'm getting:

/h> bin/iotest
Using bs=16.0M, count=128, iosize=2.0G
R:2147483648 bytes (2.0GB) copied, 3.53869 s, 579MB/s
W:2147483648 bytes (2.0GB) copied, 7.4431 s, 275MB/s

That's using cygwin's 'dd' to test. The throughput varies greatly by blocksize and tends to run toward being CPU bound on one end or the other based on blocksize. I used a bash script that runs on cygwin that measures the network protocol speed separate from file-io using character devices for source and target. It has a few other bells and whistles I've added over the years (I'll include it at the end).
I created the standard /dev/zero and /dev/null in my mounted home directory.
from the cygwin machine:

>ssh suse-linux ls -lgG {/dev,/home/law}/{zero,null} 
crw-rw-rw- 1 1, 3 Jun 16 14:53 /dev/null
crw-rw-rw- 1 1, 5 Jun 16 14:53 /dev/zero
crwxrw-rw- 1 1, 3 May 22 17:09 /home/ast/null
crw-rw-rw- 1 1, 5 Jun 15  2015 /home/ast/zero

As you can see, the major,minor codes of the devices are the same for the devices in my directory and in /dev.

For the write test, I copy from cygwin's /dev/zero to the file 'null' in my mounted home directory (mounted as H: on windows (or /h on my cygwin installation). For the read test I read from the file 'zero' in my remote home dir and write to cygwin's local /dev/null.

Anyway, I find it better to measure network speeds separately, as the file speeds vary by many factors.

The bash script that runs on cygwin (with current directory holding the two devs: 'zero' and 'null')

The script:

#!/bin/bash 
# iotest v0.2 - lawalsh(at)tlinx.org : open usage allowed
# (c) 2014-2018
# vim=:SetNumberAndWidth

_prgpth="${0:?}"; _prg="${_prgpth##*/}"; _prgdr="${_prgpth%/$_prg}"
[[ -z $_prgdr || $_prg == $_prgdr ]] && $_prgdr="$PWD"
export PATH="$_prgdr:$_prgdr/lib:$PATH"
shopt -s expand_aliases extglob sourcepath ; set -o pipefail

#include stdalias
#include Types

Dd=$(type -P dd)

[[ $Dd ]] || { echo "Cannot find dd.  Cannot proceed.";  exit 1; }

alias my=declare sub=function int='my -i' array='my -a' 
alias map='my -A' intConst='int -x' string=my

# 1 num = block size
# num-num = range of block sizes to test; w/increment = "2x", so
# 4M-16M tests 4M, 8M, 16M
# 4M-12M test 4M, 8M, 12M
# count adjusted to xfer 4G, rounding up
#----

#all xfers are using 'devices' (/dev/zero for source, /dev/null for target)
# remote filenames "zero" and "null" should be setup to be remote devices

intConst K=1024 M=K*K G=M*K T=G*K
my sfxs="KMGT"

##
# Defaults for BS count and IOSIZE
##
check_parm () {
  my param=$1
  if [[ ${!param} =~ ^[0-9]+[$sfxs]$ ]]; then
    my unit=${!param:0-1}
    eval $param="${!param:0:0-1}*$unit"
  elif [[ ${!param} =~ ^[0-9]+$ ]]; then
    : # ok as is
  elif [[ ! ${!param} =~ ^[0-9]+\*[$sfxs]$ ]]; then
    printf >&2 "Invalid number format \"%s\", for \"%s\"\n" "${!param}" "$param"
    exit 1
  fi
}

my BS COUNT IOSIZE
for p in BS COUNT IOSIZE; do
  if [[ ${!p} ]]; then
   check_parm $p
   int $p=${!p}
   echo "${p}=${!p}"
 fi
done

int params=0
array -i coef=(0 0 0)
#: ${BS:=16*M}  ${COUNT:=64}
if ((${#BS})); then int BS=$BS; params+=1; coef[0]=1; fi
if ((${#COUNT})); then int COUNT=$COUNT; params+=1; coef[1]=1; fi
if ((${#IOSIZE})); then int IOSIZE=$IOSIZE; params+=1 coef[2]=1;  fi
int BS=${BS:-16*M}
int COUNT=${COUNT:-64}
int IOSIZE=${IOSIZE:-2*G}
if ((params>=3)); then 
  printf >&2 "Error: can only specify 2 params; 3rd is calculated\n"; exit 1;
elif ((params==2)); then
  if ((!coef[2])); then int IOSIZE=COUNT*BS
  elif ((!coef[1])); then int COUNT=IOSIZE/BS
  elif ((!coef[0])); then int BS=IOSIZE/COUNT
  fi
elif ((params==1)); then
  if ((coef[2])) ; then   # only size spec'd, use default for BS unless too big
    if ((BS>IOSIZE)); then BS=IOSIZE/COUNT
    else COUNT=IOSIZE/BS
    fi
  elif ((coef[1])); then BS=IOSIZE/COUNT
  else COUNT=IOSIZE/BS
  fi
fi

int IOSIZE=COUNT*BS
echo "BS=$BS, COUNT=$COUNT, IOSIZE=$IOSIZE"

# desuffix  arg1 [arg2]
# desuffix  arg1 - num + suffix -> convert to int
#           arg2 - optional buff name (else print to stdout)
#           return 0 if no error
desuffix () {            #convert num+Suff => int store in optional Buff
  my str="${1:?}" ; shift
  my bufnam=""; (($#)) && bufnam=$1
  if [[ $str =~ ^([0-9]+)([KMGT])$ ]]; then 
    int num=${BASH_REMATCH[1]}*${BASH_REMATCH[2]}
    ((num)) || return 1
    if [[ $bufnam ]] ; then printf -v $bufnam "%d" "$num"
    else printf "%d" "$num" ; fi
  else 
    return $p 
  fi
}


hdisp () {
  my parm=${1:?}
  if [[ ! $parm =~ ^[0-9]+$ ]]; then echo >&2 "hdisp: parm1 is not num: \"$parm\""; fi
  int num=${1:?}; shift
  export PERL5OPT="$PERL5OPT -I/h/bin/lib"
  my buf="$(perl -e 'use Hout; use P; P "%s", human($ARGV[0]);' "$num")"
  #printf "hdisp-buf=%s\n" "$buf"
  string bufnam=""; (($#)) && bufnam=$1
  if [[ $bufnam ]] ; then printf -v $bufnam "%s" "$buf"
  else printf "%s" "$buf" ; fi
}

map args=([b]=setblocksize [i]=iosize )

setblocksize () { BS=$1 ; }

iosize () { IOSIZE=$1 ; }

check_params () {
  int num=0
  while (($#)) ; do
    my arg=$1 ; shift
    if [[ ${arg:0:1} == - ]]; then arg=${arg:1}; fi
    my switch=${args["$arg"]:-""}
    int iswitch=0
    if [[ ${switch:-""} ]]; then
      my val=$1; shift
      my nval=""
      int ival=0
      desuffix "$val" nval
      if [[ $nval ]]; then ival=0+nval; fi
      $switch $ival
    fi
  done
  COUNT=IOSIZE/BS
}

(($#)) && check_params "$@"

string testdir=/h

array reada=($testdir/zero /dev/null )
array writea=(/dev/zero $testdir/null oflag=direct conv=nocreat,notrunc)

sub dd_io  {
  local if="$1" of="$2"; shift 2
  nice --19 $Dd if="$if" of="$of" bs="$BS" count="$COUNT" iflag=fullblock\
      conv=nocreat "$@"
}

dd () {
  local if="$1" of="$2" ; parms="$3"; shift 2
  array out=()
  my ignore_RE='^[^ \t]+ records\ (in|out)$'
  readarray out < <(  dd_io "$if" "$of" "$@" |& { int s=$?
                        if ((s)); then echo >&2 "stat:$s"; else cat; fi ; }
                    )
  printf "%s\n" "${out[@]}"
  return 0
}
  
dd_format () { 
  my bytes btxt pnum1 suffp1 pnum2 suffp2 copt time rest
  while read bytes btxt pnum1 suffp1 pnum2 suffp2 copt time rest; do
    if ! [[ $time =~ ^[0-9] ]]; then
      copt=$pnum2; time=$suffp2;
    fi
    [[ $btxt == records ]] && continue
    [[ $bytes && $time ]] || continue
    my sizeb="" rateb=""
    my bps="$(echo -E "$bytes/$time" | bc)"
    hdisp "$bytes" sizeb
    hdisp "$bps" rateb
    my fmt="%d bytes (%sB) copied, %s s, %sB/s\n"
    printf "$fmt" "$bytes" "$sizeb" "$time" "$rateb"
  done 
}

onecycle () {
  echo -n "R:"; { dd "${reada[@]}" || exit $?; } | dd_format
  sync;sleep .1;sync
  echo -n "W:"; { dd "${writea[@]}" || exit $?; } | dd_format 
}

my bsbuf="" ios_buf=""

hdisp "$BS" bsbuf
hdisp "$IOSIZE" ios_buf

#if (($#)) ; then
# if [[ $1 == -h || $1 == -\? ]]; then
#   printf "Defaults: BS=16M, COUNT=%d, IOSIZE=%s" $Count $IOSize
#   printf "BlkSize, cnt & Total can be overriden with BS, COUNT & IOSIZE:\n"
#   printf "BS=16M (uses 1G as Total IOSIZE), or\n" "$_prg" :
#   printf "BS=1M COUNT=1K %s (uses 1G as Total IOSIZE), or\n" "$_prg" 
#   printf ""
# fi
#fi

printf "Using bs=%s, count=%s, iosize=%s\n" "$bsbuf" "$COUNT" "$ios_buf"


onecycle