nfsv4 profile

The web site gives full documentation of the parameters, but the specific options used above are:

Tunables variations

The benchmark has been ran once for each variation of the following network parameters:

Results analysis

Since the cache size is 1MB, the record sizes that have been studied are 1, 2, 4, 8 and 16 MB. The other record sizes are lesser than the cache size: thus we can consider that they only test the CPU cache performance.

The measurements results (details can be found at http://libtune.sourceforge.net/doc/nfsv4) show that:

write test (writing a new file): the 5th variation (standard value * 3) is the one that looks the most interesting. This can be seen from the average throughput, as well as from the detailed results.
re-write test (writing to an already existing file):

For the 4 MB record size, no variation seems to be interesting.
For the other record sizes, the 5th variation (standard value * 3) is globally the most interesting one, event though the detailed results show that for some records / file sizes, the 6th (standard values * 4) variation is more interesting.

read test (reading an existing file): no variation seems to be globally interesting (see average throughput). But looking at the detailed results, we can conclude the following:

no variation is interesting (whatever the record size) for files between 16 MB and 512 MB.
for 8 MB and 1 GB files, the 2nd variation (standard values + 50%) is gobally the most interesting.
for 2 GB files, the 6th variation (standard values * 4) is the most interesting

re-read test (reading a recently read file):

no variation is interesting for 16 MB record sizes, whatever the file size.
no variation is interesting (whatever the record size) for files between 16 MB and 512 MB.
for 8 MB files, the 2nd variation (standard values + 50%) is globally the most interesting.
for 1 GB files, the 2nd variation is the most interesting if we only consider 2 MB and 4 MB record sizes
for 2 GB files, the 3rd variation (standard values *2) is globally the most interesting

fwrite test:

the 5th variation is globally the most interesting (up to 32 MB files)
for 64 MB files the 5th variation remains interesting only for 4 and 8 MB record sizes
for 256 MB files, no variation is interesting, unless when using 16 MB records (in that case, the 5th variation is the most interesting)
for 512 MB files, and up to 4 MB records, the 6th variation is the most interesting
for 1GB files, the 5th variation becomes again globally more interesting
for 2 GB files, it is the 2nd variation that is the most interesting

fread test:

No variation is interesting (whatever the record size) for files between 16 MB and 256 MB.
For 4 MB files, the 5th variation is globally the most interesting
For 8 MB files, the 6th variation is interesting for 4 and 8 MB records, while the 2nd one is the most interesting for 2 MB records
For 512MB files, the variations are interesting only for 4MB and 16MB records: the 6th variation for both record sizes
For 1GB files, the 4th variation is globally the most interesting
for 2 GB files, the 4th variation is globally the most interesting.

These conclusions should be applied "carefully": nothing but iozone was running on the client side and nothing was running on the server side. Thus all the memory was dedicated to the benchmark.

Synthetic results

First of all, let's name the different variations:

`var1`	`standard values + 10%`
`var2`	`standard values + 50%`
`var3`	`standard values * 2`
`var4`	`standard values * 2.5`
`var5`	`standard values * 3`
`var6`	`standard values * 4`

The following arrays synthetize what have been said in the previous chapter (only the file / record sizes for which non-standard values are more interesting are reported):

write / re-write operations:

	`write`		`re-write`
	`write`		`record sizes != 4MB`
	`var5`		`var5`
	`client`	`server`	`client`	`server`
`rmem_max`	`393215`	`393215`	`393215`	`393215`
`rmem_default`	`319488`	`304128`	`319488`	`304128`
`wmem_max`	`393215`	`393215`	`393215`	`393215`
`wmem_default`	`319488`	`304128`	`319488`	`304128`
`tcp_rmem`	`4096 262140 524280`	`4096 262140 524280`	`4096 262140 524280`	`4096 262140 524280`
`tcp_wmem`	`4096 49152 393216`	`4096 49152 393216`	`4096 49152 393215`	`4096 49152 393216`
`tcp_mem`	`179648 1572864 2359296`	`294912 393216 589824`	`179648 1572864 2359296`	`294912 393216 589824`

read / re-read operations:

	`read`				`re-read (!= 16MB records)`
	`8MB files 1GB files`		`2GB files`		`8MB files 1Gb files (2 and 4 MB records)`		`2Gb files`
	`var2`		`var6`		`var2`		`var3`
	`client`	`server`	`client`	`server`	`client`	`server`	`client`	`server`
`rmem_max`	`196607`	`196607`	`524287`	`524287`	`196607`	`196607`	`262143`	`262143`
`rmem_default`	`159744`	`152576`	`425984`	`405504`	`159744`	`152576`	`212992`	`202752`
`wmem_max`	`196607`	`196607`	`524287`	`524287`	`196607`	`196607`	`262143`	`262143`
`wmem_default`	`159744`	`152576`	`425984`	`405504`	`159744`	`152576`	`212992`	`202752`
`tcp_rmem`	`4096 131070 262140`	`4096 131070 262140`	`16384 349520 699040`	`16384 349520 699040`	`4096 131070 262140`	`4096 131070 262140`	`8192 174760 349520`	`8192 174760 349520`
`tcp_wmem`	`4096 24576 196608`	`4096 24576 196608`	`16384 65536 524288`	`16384 65536 524288`	`4096 24576 196608`	`4096 24576 196608`	`8192 32768 262144`	`8192 32768 262144`
`tcp_mem`	`589824 786432 1179648`	`147456 196608 294912`	`1572864 2097152 3145728`	`393216 524288 786432`	`589824 786432 1179648`	`147456 196608 294912`	`786432 1048576 1572864`	`196608 262144 393216`

fwrite operations:

	`read`
	`up to 32MB files 64MB files (4 and 8MB records) 256MB files (16MB records) 1GB files`		`512MB files (up to 4MB records)`		`2GB files`
	`var5`		`var6`		`var3`
	`client`	`server`	`client`	`server`	`client`	`server`
`rmem_max`	`393215`	`393215`	`524287`	`524287`	`262143`	`262143`
`rmem_default`	`319488`	`304128`	`425984`	`405504`	`212992`	`202752`
`wmem_max`	`393215`	`393215`	`524287`	`524287`	`262143`	`262143`
`wmem_default`	`319488`	`304128`	`425984`	`405504`	`212992`	`202752`
`tcp_rmem`	`4096 262140 524280`	`4096 262140 524280`	`16384 349520 699040`	`16384 349520 699040`	`8192 174760 349520`	`8192 174760 349520`
`tcp_wmem`	`4096 49152 393215`	`4096 49152 393216`	`16384 65536 524288`	`16384 65536 524288`	`8192 32768 262144`	`8192 32768 262144`
`tcp_mem`	`179648 1572864 2359296`	`294912 393216 589824`	`1572864 2097152 3145728`	`393216 524288 786432`	`786432 1048576 1572864`	`196608 262144 393216`

fread operations:

	`read`
	`4MB files`		`8MB files (4 and 8MB records) 512MB files (4 and 16MB records)`		`8MB files (2MB records)`		`1 and 2GB files`
	`var5`		`var6`		`var2`		`var4`
	`client`	`server`	`client`	`server`	`client`	`server`	`client`	`server`
`rmem_max`	`393215`	`393215`	`524287`	`524287`	`262143`	`262143`	`327679`	`327679`
`rmem_default`	`319488`	`304128`	`425984`	`405504`	`212992`	`202752`	`266240`	`253440`
`wmem_max`	`393215`	`393215`	`524287`	`524287`	`262143`	`262143`	`327679`	`327679`
`wmem_default`	`319488`	`304128`	`425984`	`405504`	`212992`	`202752`	`266240`	`253440`
`tcp_rmem`	`4096 262140 524280`	`4096 262140 524280`	`16384 349520 699040`	`16384 349520 699040`	`8192 174760 349520`	`8192 174760 349520`	`10240 218450 436900`	`10240 218450 436900`
`tcp_wmem`	`4096 49152 393215`	`4096 49152 393216`	`16384 65536 524288`	`16384 65536 524288`	`8192 32768 262144`	`8192 32768 262144`	`10240 40960 327680`	`10240 40960 327680`
`tcp_mem`	`179648 1572864 2359296`	`294912 393216 589824`	`1572864 2097152 3145728`	`393216 524288 786432`	`786432 1048576 1572864`	`196608 262144 393216`	`983040 1310720 1966080`	`245760 327680 491520`

Note

While running the iozone tests, we noticed that the following scenario doesn't work:

stop nfs on the server
apply some tunables settings on both client and server machines
start nfs on the server
nfs-mount the tested directory
run iozone
unmount the tested directory
stop nfs on the server
apply some other tunables setting
restart nfs on the server
nfs-mount the tested directory

Step 10 leads to a hang.

`parameter`	`usage`	`default value on the server`	`default value on the client`
`rmem_max`	`Maximum receive buffer size. If ipv4/tcp_rmem max value is >, rmem_max overrides it`	`131071`	`131071`
`rmem_default`	`Default receive buffer size. Overriden by ipv4/tcp_rmem default value`	`101376`	`106496`
`wmem_max`	`Maximum send buffer size. If ipv4 value is >, wmem_max overrides it`	`131071`	`131071`
`wmem_default`	`Default send buffer size. Overriden by ipv4/tcp_wmem default value`	`101376`	`106496`

`parameter`	`usage`	`default value on the server`	`default value on the client`
`tcp_rmem`	`Memory size of the TCP receive buffers (for a single TCP socket):`
	`minimum`	`4096`	`4096`
	`default`	`87380`	`87380`
	`maximum`	`174760`	`174760`
`tcp_wmem`	`Memory size of the TCP send buffers (for a single TCP socket):`
	`minimum`	`4096`	`4096`
	`default`	`16384`	`16384`
	`maximum`	`131072`	`131072`
`tcp_mem`	`TCP overall memory`
	`low: below this number of pages TCP is not bothered about its memory consumption.`	`98304`	`393216`
	`pressure: when amount of memory allocated by TCP exceeds this number of pages, TCP moderates its memory consumption and enters memory pressure mode, which is exited when memory consumption falls under "low".`	`131072`	`524288`
	`high: number of pages allowed for queuing by all TCP sockets. When this value is reached TCp streams and packets start getting dropped until a lower memory usage is reached again`	`196608`	`786432`

Tuning Linux kernel for NFSv4

Overview

Testing machines

Benchmark

Tunables variations

Results analysis

Synthetic results

Note

End of Document