Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
fio(1)			    General Commands Manual			fio(1)

NAME
       fio - flexible I/O tester

SYNOPSIS
       fio [options] [jobfile]...

DESCRIPTION
       fio  is a tool that will	spawn a	number of threads or processes doing a
       particular type of I/O action as	specified by the  user.	  The  typical
       use  of	fio  is	to write a job file matching the I/O load one wants to
       simulate.

OPTIONS
       --debug=type
	      Enable verbose tracing of	various	fio actions. May be `all'  for
	      all  types  or  individual  types	separated by a comma (eg --de-
	      bug=io,file). `help' will	list all available tracing options.

       --output=filename
	      Write output to filename.

       --output-format=format
	      Set the reporting	format to normal, terse, json, or json+.  Mul-
	      tiple  formats  can be selected, separate	by a comma. terse is a
	      CSV based	format.	json+ is like json, except it adds a full dump
	      of the latency buckets.

       --runtime=runtime
	      Limit run	time to	runtime	seconds.

       --bandwidth-log
	      Generate aggregate bandwidth logs.

       --minimal
	      Print statistics in a terse, semicolon-delimited format.

       --append-terse
	      Print statistics in selected mode	AND terse, semicolon-delimited
	      format.  Deprecated, use --output-format instead to select  mul-
	      tiple formats.

       --version
	      Display version information and exit.

       --terse-version=version
	      Set terse	version	output format (default 3, or 2 or 4)

       --help Display usage information	and exit.

       --cpuclock-test
	      Perform test and validation of internal CPU clock

       --crctest[=test]
	      Test  the	speed of the builtin checksumming functions. If	no ar-
	      gument is	given, all of them are tested. Or  a  comma  separated
	      list can be passed, in which case	the given ones are tested.

       --cmdhelp=command
	      Print  help  information for command.  May be `all' for all com-
	      mands.

       --enghelp=ioengine[,command]
	      List all commands	defined	by ioengine, or	print help for command
	      defined by ioengine.

       --showcmd=jobfile
	      Convert jobfile to a set of command-line options.

       --eta=when
	      Specifies	 when  real-time ETA estimate should be	printed.  when
	      may be one of `always', `never' or `auto'.

       --eta-newline=time
	      Force an ETA newline for every `time` period passed.

       --status-interval=time
	      Report full output status	every `time` period passed.

       --readonly
	      Turn on safety read-only checks, preventing any attempted	write.

       --section=sec
	      Only run section sec from	job file. This option can be used mul-
	      tiple times to add more sections to run.

       --alloc-size=kb
	      Set the internal smalloc pool size to kb kilobytes.

       --warnings-fatal
	      All  fio	parser warnings	are fatal, causing fio to exit with an
	      error.

       --max-jobs=nr
	      Set the maximum allowed number of	 jobs  (threads/processes)  to
	      support.

       --server=args
	      Start  a backend server, with args specifying what to listen to.
	      See client/server	section.

       --daemonize=pidfile
	      Background a fio server, writing the pid to the given pid	file.

       --client=host
	      Instead of running the jobs locally, send	and run	 them  on  the
	      given host or set	of hosts.  See client/server section.

       --idle-prof=option
	      Report  cpu  idleness  on	 a system or percpu basis (option=sys-
	      tem,percpu) or run  unit	work  calibration  only	 (option=cali-
	      brate).

JOB FILE FORMAT
       Job  files are in `ini' format. They consist of one or more job defini-
       tions, which begin with a job name in square brackets and extend	to the
       next  job  name.	 The job name can be any ASCII string except `global',
       which has a special meaning.  Following the job name is a  sequence  of
       zero  or	more parameters, one per line, that define the behavior	of the
       job.  Any line starting with a `;' or `#'  character  is	 considered  a
       comment and ignored.

       If jobfile is specified as `-', the job file will be read from standard
       input.

   Global Section
       The global section contains default parameters for  jobs	 specified  in
       the job file.  A	job is only affected by	global sections	residing above
       it, and there may be any	number of global sections.  Specific job defi-
       nitions may override any	parameter set in global	sections.

JOB PARAMETERS
   Types
       Some  parameters	may take arguments of a	specific type.	Anywhere a nu-
       meric value is required,	an arithmetic expression may be	used, provided
       it is surrounded	by parentheses.	Supported operators are:

		     addition (+)

		     subtraction (-)

		     multiplication (*)

		     division (/)

		     modulus (%)

		     exponentiation (^)

       For time	values in expressions, units are microseconds by default. This
       is different than for time values not in	expressions (not  enclosed  in
       parentheses). The types used are:

       str    String: a	sequence of alphanumeric characters.

       int    Integer. A whole number value, which may contain an integer pre-
	      fix and an integer suffix.

	      [integer prefix]number[integer suffix]

	      The optional integer prefix specifies the	number's base. The de-
	      fault is decimal.	0x specifies hexadecimal.

	      The  optional  integer  suffix specifies the number's units, and
	      includes an optional unit	prefix	and  an	 optional  unit.   For
	      quantities of data, the default unit is bytes. For quantities of
	      time, the	default	unit is	seconds.

	      With kb_base=1000, fio follows international standards for  unit
	      prefixes.	  To specify power-of-10 decimal values	defined	in the
	      International System of Units (SI):
	      ki means kilo (K)	or 1000
	      mi means mega (M)	or 1000**2
	      gi means giga (G)	or 1000**3
	      ti means tera (T)	or 1000**4
	      pi means peta (P)	or 1000**5

	      To specify power-of-2 binary values defined in IEC 80000-13:
	      k	means kibi (Ki)	or 1024
	      m	means mebi (Mi)	or 1024**2
	      g	means gibi (Gi)	or 1024**3
	      t	means tebi (Ti)	or 1024**4
	      p	means pebi (Pi)	or 1024**5

	      With kb_base=1024	(the default), the unit	prefixes are  opposite
	      from  those  specified  in  the SI and IEC 80000-13 standards to
	      provide compatibility with old scripts.  For example,  4k	 means
	      4096.

	      Examples with kb_base=1000:
	      4	KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
	      1	MiB: 1048576, 1m, 1024k
	      1	MB: 1000000, 1mi, 1000ki
	      1	TiB: 1073741824, 1t, 1024m, 1048576k
	      1	TB: 1000000000,	1ti, 1000mi, 1000000ki

	      Examples with kb_base=1024 (default):
	      4	KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
	      1	MiB: 1048576, 1m, 1024k
	      1	MB: 1000000, 1mi, 1000ki
	      1	TiB: 1073741824, 1t, 1024m, 1048576k
	      1	TB: 1000000000,	1ti, 1000mi, 1000000ki

	      For  quantities of data, an optional unit	of 'B' may be included
	      (e.g.,  'kb' is the same as 'k').

	      The integer suffix  is  not  case	 sensitive  (e.g.,  m/mi  mean
	      mebi/mega, not milli). 'b' and 'B' both mean byte, not bit.

	      To specify times (units are not case sensitive):
	      D	means days
	      H	means hours
	      M	mean minutes
	      s	or sec means seconds (default)
	      ms or msec means milliseconds
	      us or usec means microseconds

       bool   Boolean:	a  true	or false value.	`0' denotes false, `1' denotes
	      true.

       irange Integer range: a range  of  integers  specified  in  the	format
	      lower:upper or lower-upper. lower	and upper may contain a	suffix
	      as described above.  If an option	allows	two  sets  of  ranges,
	      they  are	 separated  with  a `,'	or `/' character. For example:
	      `8-8k/8M-4G'.

       float_list
	      List of floating numbers:	A list of floating numbers,  separated
	      by a ':' character.

   Parameter List
       name=str
	      May be used to override the job name.  On	the command line, this
	      parameter	has the	special	purpose	of signalling the start	 of  a
	      new job.

       wait_for=str
	      Specifies	 the name of the already defined job to	wait for. Sin-
	      gle waitee name only may be specified. If	set, the job won't  be
	      started  until all workers of the	waitee job are done.  Wait_for
	      operates on the job name basis, so there are a few  limitations.
	      First, the waitee	must be	defined	prior to the waiter job	(mean-
	      ing no forward references). Second, if a job is being referenced
	      as a waitee, it must have	a unique name (no duplicate waitees).

       description=str
	      Human-readable  description  of  the job.	It is printed when the
	      job is run, but otherwise	has no special purpose.

       directory=str
	      Prefix filenames with this directory.  Used to place files in  a
	      location	other than `./'.  You can specify a number of directo-
	      ries by separating the names with	a ':' character. These	direc-
	      tories  will  be assigned	equally	distributed to job clones cre-
	      ates with	numjobs	as long	as they	are using generated filenames.
	      If  specific  filename(s)	 are set fio will use the first	listed
	      directory, and thereby matching  the   filename  semantic	 which
	      generates	a file each clone if not specified, but	let all	clones
	      use the same if set. See filename	for  considerations  regarding
	      escaping certain characters on some platforms.

       filename=str
	      fio  normally makes up a file name based on the job name,	thread
	      number, and file number. If you  want  to	 share	files  between
	      threads in a job or several jobs,	specify	a filename for each of
	      them to override the default.  If	the I/O	engine is  file-based,
	      you can specify a	number of files	by separating the names	with a
	      `:' character. `-' is a reserved name, meaning stdin or  stdout,
	      depending	 on the	read/write direction set. On Windows, disk de-
	      vices are	accessed as \.PhysicalDrive0  for  the	first  device,
	      \.PhysicalDrive1	for  the second	etc. Note: Windows and FreeBSD
	      prevent write access to areas of the disk	containing in-use data
	      (e.g.  filesystems). If the wanted filename does need to include
	      a	colon, then escape that	with a '\' character. For instance, if
	      the  filename  is	"/dev/dsk/foo@3,0:c", then you would use file-
	      name="/dev/dsk/foo@3,0\:c".

       filename_format=str
	      If sharing multiple files	between	jobs, it is usually  necessary
	      to  have fio generate the	exact names that you want. By default,
	      fio will name a file based on the	default	file format specifica-
	      tion of jobname.jobnumber.filenumber. With this option, that can
	      be customized. Fio will recognize	and replace the	following key-
	      words in this string:

		     $jobname
			    The	name of	the worker thread or process.

		     $jobnum
			    The	 incremental  number  of  the worker thread or
			    process.

		     $filenum
			    The	incremental number of the file for that	worker
			    thread or process.

	      To  have dependent jobs share a set of files, this option	can be
	      set to have fio generate filenames that are shared  between  the
	      two. For instance, if testfiles.$filenum is specified, file num-
	      ber 4 for	any job	will be	 named	testfiles.4.  The  default  of
	      $jobname.$jobnum.$filenum	will be	used if	no other format	speci-
	      fier is given.

       unique_filename=bool
	      To avoid collisions between networked clients, fio  defaults  to
	      prefixing	 any  generated	filenames (with	a directory specified)
	      with the source of the client connecting.	To disable this	behav-
	      ior, set this option to 0.

       lockfile=str
	      Fio defaults to not locking any files before it does IO to them.
	      If a file	or file	descriptor is shared, fio can serialize	IO  to
	      that  file  to make the end result consistent. This is usual for
	      emulating	real workloads that share files.  The lock modes are:

		     none   No locking.	This is	the default.

		     exclusive
			    Only one thread or process may do IO  at  a	 time,
			    excluding all others.

		     readwrite
			    Read-write	locking	 on the	file. Many readers may
			    access the file at the same	time, but  writes  get
			    exclusive access.

       opendir=str Recursively open any	files below directory str.

       readwrite=str, rw=str
	      Type of I/O pattern.  Accepted values are:

		     read   Sequential reads.

		     write  Sequential writes.

		     trim   Sequential trims (Linux block devices only).

		     randread
			    Random reads.

		     randwrite
			    Random writes.

		     randtrim
			    Random trims (Linux	block devices only).

		     rw, readwrite
			    Mixed sequential reads and writes.

		     randrw Mixed random reads and writes.

		     trimwrite
			    Sequential	trim  and write	mixed workload.	Blocks
			    will be trimmed first, then	the same  blocks  will
			    be written to.

	      Fio  defaults to read if the option is not specified.  For mixed
	      I/O, the default split is	50/50. For certain types of io the re-
	      sult  may	 still be skewed a bit,	since the speed	may be differ-
	      ent. It is possible to specify a number of  IO's	to  do	before
	      getting  a new offset, this is done by appending a `:_nr_	to the
	      end of the string	given. For a random read, it would  look  like
	      rw=randread:8  for passing in an offset modifier with a value of
	      8. If the	postfix	is used	with a sequential IO pattern, then the
	      value  specified	will be	added to the generated offset for each
	      IO. For instance,	using  rw=write:4k  will  skip	4k  for	 every
	      write. It	turns sequential IO into sequential IO with holes. See
	      the rw_sequencer option.

       rw_sequencer=str
	      If an offset modifier is given by	 appending  a  number  to  the
	      rw=<str>	line,  then this option	controls how that number modi-
	      fies the IO offset being generated. Accepted values are:

		     sequential
			    Generate sequential	offset

		     identical
			    Generate the same offset

	      sequential is only useful	for random IO, where  fio  would  nor-
	      mally  generate  a new random offset for every IO. If you	append
	      eg 8 to randread,	you would get a	new random offset for every  8
	      IO's.  The result	would be a seek	for only every 8 IO's, instead
	      of for every IO. Use rw=randread:8 to specify that.  As  sequen-
	      tial IO is already sequential, setting sequential	for that would
	      not result in any	differences.  identical	behaves	in  a  similar
	      fashion,	except	it sends the same offset 8 number of times be-
	      fore generating a	new offset.

       kb_base=int
	      The base unit for	a kilobyte. The	defacto	base  is  2^10,	 1024.
	      Storage  manufacturers  like  to	use 10^3 or 1000 as a base ten
	      unit instead, for	obvious	reasons. Allowed values	 are  1024  or
	      1000, with 1024 being the	default.

       unified_rw_reporting=bool
	      Fio  normally  reports statistics	on a per data direction	basis,
	      meaning that reads, writes, and trims are	accounted and reported
	      separately.  If  this option is set fio sums the results and re-
	      ports them as "mixed" instead.

       randrepeat=bool
	      Seed the random number generator used for	random I/O patterns in
	      a	predictable way	so the pattern is repeatable across runs.  De-
	      fault: true.

       allrandrepeat=bool
	      Seed all random number generators	in a predictable  way  so  re-
	      sults are	repeatable across runs.	 Default: false.

       randseed=int
	      Seed  the	 random	number generators based	on this	seed value, to
	      be able to control what sequence of output is  being  generated.
	      If  not  set, the	random sequence	depends	on the randrepeat set-
	      ting.

       fallocate=str
	      Whether pre-allocation is	performed when laying down files.  Ac-
	      cepted values are:

		     none   Do not pre-allocate	space.

		     posix  Pre-allocate via posix_fallocate(3).

		     keep   Pre-allocate    via	   fallocate(2)	   with	  FAL-
			    LOC_FL_KEEP_SIZE set.

		     0	    Backward-compatible	alias for 'none'.

		     1	    Backward-compatible	alias for 'posix'.

	      May not be available on all supported platforms. 'keep' is  only
	      available	 on Linux. If using ZFS	on Solaris this	must be	set to
	      'none' because ZFS doesn't support it. Default: 'posix'.

       fadvise_hint=str
	      Use posix_fadvise(2) to advise the kernel	what I/O patterns  are
	      likely to	be issued. Accepted values are:

		     0	    Backwards compatible hint for "no hint".

		     1	    Backwards  compatible  hint	 for  "advise with fio
			    workload type". This uses FADV_RANDOM for a	random
			    workload,  and  FADV_SEQUENTIAL  for  a sequential
			    workload.

		     sequential
			    Advise using FADV_SEQUENTIAL

		     random Advise using FADV_RANDOM

       fadvise_stream=int
	      Use posix_fadvise(2) to advise the kernel	 what  stream  ID  the
	      writes issued belong to. Only supported on Linux.	Note, this op-
	      tion may change going forward.

       size=int
	      Total size of I/O	for this job.  fio will	run  until  this  many
	      bytes  have  been	 transferred,  unless limited by other options
	      (runtime,	for instance, or increased/descreased by io_size). Un-
	      less nrfiles and filesize	options	are given, this	amount will be
	      divided between the available files for the job. If not set, fio
	      will  use	 the  full  size of the	given files or devices.	If the
	      files do not exist, size must be given. It is also  possible  to
	      give  size  as  a	 percentage  between 1 and 100.	If size=20% is
	      given, fio will use 20% of the full size of the given  files  or
	      devices.

       io_size=int, io_limit =int
	      Normally fio operates within the region set by size, which means
	      that the size option sets	both the region	and size of IO	to  be
	      performed.   Sometimes  that is not what you want. With this op-
	      tion, it is possible to define just the amount of	 IO  that  fio
	      should  do.  For instance, if size is set	to 20G and io_limit is
	      set to 5G, fio will perform IO within the	 first	20G  but  exit
	      when  5G have been done. The opposite is also possible - if size
	      is set to	20G, and io_size is set	to 40G,	then fio will  do  40G
	      of IO within the 0..20G region.

       fill_device=bool, fill_fs=bool
	      Sets  size  to  something	 really	large and waits	for ENOSPC (no
	      space left on device) as the terminating condition.  Only	 makes
	      sense  with  sequential  write.	For a read workload, the mount
	      point will be filled first then IO started on the	 result.  This
	      option  doesn't  make  sense  if operating on a raw device node,
	      since the	size of	that is	already	known by the file system.  Ad-
	      ditionally,  writing beyond end-of-device	will not return	ENOSPC
	      there.

       filesize=irange
	      Individual file sizes. May be a range, in	which  case  fio  will
	      select sizes for files at	random within the given	range, limited
	      to size in total (if that	is given). If filesize is  not	speci-
	      fied, each created file is the same size.

       file_append=bool
	      Perform  IO after	the end	of the file. Normally fio will operate
	      within the size of a file. If this option	is set,	then fio  will
	      append  to the file instead. This	has identical behavior to set-
	      ting offset to the size of a file. This  option  is  ignored  on
	      non-regular files.

       blocksize=int[,int][,int], bs=int[,int][,int]
	      The block	size in	bytes for I/O units.  Default: 4096.  A	single
	      value applies to reads, writes, and trims.  Comma-separated val-
	      ues may be specified for reads, writes, and trims.  Empty	values
	      separated	by commas use the default value. A  value  not	termi-
	      nated in a comma applies to subsequent types.
	      Examples:
	      bs=256k	 means 256k for	reads, writes and trims
	      bs=8k,32k	 means 8k for reads, 32k for writes and	trims
	      bs=8k,32k, means 8k for reads, 32k for writes, and default for trims
	      bs=,8k	 means default for reads, 8k for writes	and trims
	      bs=,8k,	 means default for reads, 8k for writes, and default for trims

       blocksize_range=irange[,irange][,irange],
       bsrange=irange[,irange][,irange]
	      A	range of block sizes in	bytes for I/O units.  The  issued  I/O
	      unit  will  always  be  a	 multiple  of the minimum size,	unless
	      blocksize_unaligned is set.  Comma-separated ranges may be spec-
	      ified for	reads, writes, and trims as described in blocksize.
	      Example: bsrange=1k-4k,2k-8k.

       bssplit=str[,str][,str]
	      This option allows even finer grained control of the block sizes
	      issued, not just even splits between them. With this option, you
	      can  weight  various block sizes for exact control of the	issued
	      IO for a job that	has mixed block	sizes. The format of  the  op-
	      tion  is bssplit=blocksize/percentage, optionally	adding as many
	      definitions as needed  separated	by  a  colon.	Example:  bss-
	      plit=4k/10:64k/50:32k/40	would  issue  50%  64k	blocks,	10% 4k
	      blocks and 40% 32k blocks. bssplit also supports giving separate
	      splits  to reads,	writes,	and trims.  Comma-separated values may
	      be specified for reads, writes, and trims	as described in	block-
	      size.

       blocksize_unaligned, bs_unaligned
	      If  set,	fio  will  issue I/O units with	any size within	block-
	      size_range, not just multiples of	the minimum size.  This	 typi-
	      cally won't work with direct I/O,	as that	normally requires sec-
	      tor alignment.

       bs_is_seq_rand=bool
	      If this option is	set, fio will use the normal read,write	block-
	      size  settings  as sequential,random blocksize settings instead.
	      Any random read or write will use	the WRITE blocksize  settings,
	      and  any	sequential  read  or write will	use the	READ blocksize
	      settings.

       blockalign=int[,int][,int], ba=int[,int][,int]
	      Boundary to which	fio will  align	 random	 I/O  units.  Default:
	      blocksize.  Minimum alignment is typically 512b for using	direct
	      IO, though it usually depends on the hardware block size.	  This
	      option  is mutually exclusive with using a random	map for	files,
	      so it will turn off that option.	Comma-separated	values may  be
	      specified	 for  reads,  writes, and trims	as described in	block-
	      size.

       zero_buffers
	      Initialize buffers with all zeros. Default:  fill	 buffers  with
	      random data.

       refill_buffers
	      If this option is	given, fio will	refill the IO buffers on every
	      submit. The default is to	only fill it at	init  time  and	 reuse
	      that  data.  Only	 makes	sense if zero_buffers isn't specified,
	      naturally. If data verification is  enabled,  refill_buffers  is
	      also automatically enabled.

       scramble_buffers=bool
	      If  refill_buffers  is  too  costly and the target is using data
	      deduplication, then setting this option will slightly modify the
	      IO  buffer  contents  to defeat normal de-dupe attempts. This is
	      not enough to defeat more	clever block compression attempts, but
	      it will stop naive dedupe	of blocks. Default: true.

       buffer_compress_percentage=int
	      If  this is set, then fio	will attempt to	provide	IO buffer con-
	      tent (on WRITEs) that compress to	the specified level. Fio  does
	      this  by providing a mix of random data and a fixed pattern. The
	      fixed pattern is either zeroes, or the pattern specified by buf-
	      fer_pattern.  If	the  pattern option is used, it	might skew the
	      compression ratio	slightly. Note that this  is  per  block  size
	      unit,  for  file/disk  wide  compression level that matches this
	      setting. Note that this is per block size	 unit,	for  file/disk
	      wide  compression	 level	that matches this setting, you'll also
	      want to set refill_buffers.

       buffer_compress_chunk=int
	      See buffer_compress_percentage. This setting allows fio to  man-
	      age  how big the ranges of random	data and zeroed	data is. With-
	      out this set, fio	 will  provide	buffer_compress_percentage  of
	      blocksize	 random	 data,	followed by the	remaining zeroed. With
	      this set to some chunk size smaller than the block size, fio can
	      alternate	random and zeroed data throughout the IO buffer.

       buffer_pattern=str
	      If  set, fio will	fill the I/O buffers with this pattern or with
	      the contents of a	file. If not set, the contents of I/O  buffers
	      are defined by the other options related to buffer contents. The
	      setting can be any pattern of bytes, and can be prefixed with 0x
	      for  hex	values.	It may also be a string, where the string must
	      then be wrapped with ``""``. Or it may also be a filename, where
	      the  filename must be wrapped with ``''``	in which case the file
	      is opened	and read. Note that not	all the	file contents will  be
	      read  if that would cause	the buffers to overflow. So, for exam-
	      ple:
		     buffer_pattern='filename'
			    or
		     buffer_pattern="abcd"
			    or
		     buffer_pattern=-12
			    or
		     buffer_pattern=0xdeadface

	      Also you can combine everything together in any order:

		     buffer_pattern=0xdeadface"abcd"-12'filename'

       dedupe_percentage=int
	      If set, fio will generate	this percentage	of  identical  buffers
	      when  writing.   These  buffers will be naturally	dedupable. The
	      contents of the buffers depend on	what other buffer  compression
	      settings	have  been  set.  It's possible	to have	the individual
	      buffers either fully compressible, or not	at  all.  This	option
	      only controls the	distribution of	unique buffers.

       nrfiles=int
	      Number of	files to use for this job.  Default: 1.

       openfiles=int
	      Number  of  files	 to  keep open at the same time.  Default: nr-
	      files.

       file_service_type=str
	      Defines how files	to service are selected.  The following	 types
	      are defined:

		     random Choose a file at random.

		     roundrobin
			    Round robin	over opened files (default).

		     sequential
			    Do each file in the	set sequentially.

		     zipf   Use	 a zipfian distribution	to decide what file to
			    access.

		     pareto Use	a pareto distribution to decide	what  file  to
			    access.

		     gauss  Use	 a  gaussian  (normal)	distribution to	decide
			    what file to access.

	      For random, roundrobin, and sequential, a	 postfix  can  be  ap-
	      pended  to tell fio how many I/Os	to issue before	switching to a
	      new file.	 For  example,	specifying  file_service_type=random:8
	      would  cause  fio	to issue 8 I/Os	before selecting a new file at
	      random. For the  non-uniform  distributions,  a  floating	 point
	      postfix  can  be	given  to  influence  how  the distribution is
	      skewed. See random_distribution for a description	 of  how  that
	      would work.

       ioengine=str
	      Defines  how  the	 job  issues I/O.  The following types are de-
	      fined:

		     sync   Basic read(2) or write(2) I/O.  fseek(2)  is  used
			    to position	the I/O	location.

		     psync  Basic  pread(2)  or	pwrite(2) I/O.	Default	on all
			    supported operating	systems	except for Windows.

		     vsync  Basic readv(2)  or	writev(2)  I/O.	 Will  emulate
			    queuing  by	 coalescing adjacent IOs into a	single
			    submission.

		     pvsync Basic preadv(2) or pwritev(2) I/O.

		     pvsync2
			    Basic preadv2(2) or	pwritev2(2) I/O.

		     libaio Linux native asynchronous I/O. This	 ioengine  de-
			    fines engine specific options.

		     posixaio
			    POSIX   asynchronous  I/O  using  aio_read(3)  and
			    aio_write(3).

		     solarisaio
			    Solaris native asynchronous	I/O.

		     windowsaio
			    Windows native asynchronous	I/O. Default  on  Win-
			    dows.

		     mmap   File is memory mapped with mmap(2) and data	copied
			    using memcpy(3).

		     splice splice(2) is used to transfer  the	data  and  vm-
			    splice(2)  to transfer data	from user-space	to the
			    kernel.

		     sg	    SCSI generic sg v3 I/O. May	be either  synchronous
			    using  the	SG_IO ioctl, or	if the target is an sg
			    character device, we use read(2) and write(2)  for
			    asynchronous I/O.

		     null   Doesn't  transfer  any  data,  just	 pretends  to.
			    Mainly used	to exercise fio	itself and for	debug-
			    ging and testing purposes.

		     net    Transfer  over  the	 network.   The	protocol to be
			    used can be	defined	with the  protocol  parameter.
			    Depending  on  the	protocol,  filename, hostname,
			    port, or listen must be specified.	This  ioengine
			    defines engine specific options.

		     netsplice
			    Like  net,	but  uses splice(2) and	vmsplice(2) to
			    map	data and send/receive. This  ioengine  defines
			    engine specific options.

		     cpuio  Doesn't  transfer  any  data, but burns CPU	cycles
			    according to cpuload and cpuchunks	parameters.  A
			    job	 never	finishes  unless there is at least one
			    non-cpuio job.

		     guasi  The	GUASI I/O  engine  is  the  Generic  Userspace
			    Asynchronous  Syscall  Interface approach to asyn-
			    chronous I/O.
			    See	<http://www.xmailserver.org/guasi-lib.html>.

		     rdma   The	RDMA I/O engine	supports both RDMA memory  se-
			    mantics  (RDMA_WRITE/RDMA_READ) and	channel	seman-
			    tics (Send/Recv)  for  the	InfiniBand,  RoCE  and
			    iWARP protocols.

		     external
			    Loads  an external I/O engine object file.	Append
			    the	engine filename	as `:enginepath'.

		     falloc
			       IO engine that does regular linux native	fallo-
			    cate  call	to  simulate  data transfer as fio io-
			    engine
			      DDIR_READ	   does	  fallocate(,mode    =	  FAL-
			    LOC_FL_KEEP_SIZE,)
			      DIR_WRITE	does fallocate(,mode = 0)
			      DDIR_TRIM	   does	   fallocate(,mode    =	  FAL-
			    LOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE)

		     e4defrag
			    IO	engine	that  does  regular  EXT4_IOC_MOVE_EXT
			    ioctls  to simulate	defragment activity request to
			    DDIR_WRITE event

		     rbd    IO engine supporting direct	access to  Ceph	 Rados
			    Block Devices (RBD)	via librbd without the need to
			    use	the kernel rbd driver. This  ioengine  defines
			    engine specific options.

		     gfapi  Using  Glusterfs libgfapi sync interface to	direct
			    access to Glusterfs	volumes	without	having	to  go
			    through  FUSE.  This  ioengine defines engine spe-
			    cific options.

		     gfapi_async
			    Using Glusterfs libgfapi async interface to	direct
			    access  to	Glusterfs volumes without having to go
			    through FUSE. This ioengine	 defines  engine  spe-
			    cific options.

		     libhdfs
			    Read  and  write through Hadoop (HDFS).  The file-
			    name option	is used	to specify  host,port  of  the
			    hdfs  name-node to connect.	This engine interprets
			    offsets a little differently. In HDFS, files  once
			    created  cannot be modified.  So random writes are
			    not	possible. To imitate this, libhdfs engine  ex-
			    pects  bunch  of  small  files  to be created over
			    HDFS, and engine will randomly pick	a file out  of
			    those  files  based	on the offset generated	by fio
			    backend. (see the example job file to create  such
			    files,  use	 rw=write  option).  Please  note, you
			    might want to set necessary	environment  variables
			    to work with hdfs/libhdfs properly.

		     mtd    Read,  write  and  erase  an  MTD character	device
			    (e.g., /dev/mtd0). Discards	are treated as erases.
			    Depending  on  the underlying device type, the I/O
			    may	have to	go in  a  certain  pattern,  e.g.,  on
			    NAND,  writing  sequentially  to  erase blocks and
			    discarding before overwriting. The trimwrite  mode
			    works well for this	constraint.

		     pmemblk
			    Read and write using filesystem DAX	to a file on a
			    filesystem mounted with DAX	on a persistent	memory
			    device through the NVML libpmemblk library.

		     dev-dax
			    Read  and  write  using device DAX to a persistent
			    memory device (e.g., /dev/dax0.0) through the NVML
			    libpmem library.

       iodepth=int
	      Number  of  I/O  units  to keep in flight	against	the file. Note
	      that increasing iodepth beyond 1 will not	affect synchronous io-
	      engines  (except for small degress when verify_async is in use).
	      Even async engines may impose OS restrictions  causing  the  de-
	      sired  depth  not	to be achieved.	 This may happen on Linux when
	      using libaio and not setting direct=1, since buffered IO is  not
	      async  on	 that  OS. Keep	an eye on the IO depth distribution in
	      the fio output to	verify that the	achieved depth is as expected.
	      Default: 1.

       iodepth_batch=int, iodepth_batch_submit=int
	      This  defines  how  many	pieces of IO to	submit at once.	It de-
	      faults to	1 which	means that we submit each IO as	soon as	it  is
	      available,  but  can be raised to	submit bigger batches of IO at
	      the time.	If it is set to	0 the iodepth value will be used.

       iodepth_batch_complete_min=int, iodepth_batch_complete=int
	      This defines how many pieces of IO to retrieve at	once.  It  de-
	      faults to	1 which
	       means  that  we'll  ask	for a minimum of 1 IO in the retrieval
	      process from the kernel. The IO retrieval	will go	 on  until  we
	      hit  the limit set by iodepth_low. If this variable is set to 0,
	      then fio will always check for completed events  before  queuing
	      more  IO.	 This helps reduce IO latency, at the cost of more re-
	      trieval system calls.

       iodepth_batch_complete_max=int
	      This defines maximum pieces of IO	 to  retrieve  at  once.  This
	      variable	 should	  be   used   along   with  iodepth_batch_com-
	      plete_min=int variable, specifying the  range  of	 min  and  max
	      amount  of  IO which should be retrieved.	By default it is equal
	      to iodepth_batch_complete_min value.

	      Example #1:
		     iodepth_batch_complete_min=1

		     iodepth_batch_complete_max=<iodepth>

	      which means that we will retrieve	at least 1 IO and  up  to  the
	      whole  submitted	queue  depth. If none of IO has	been completed
	      yet, we will wait.

	      Example #2:
		     iodepth_batch_complete_min=0

		     iodepth_batch_complete_max=<iodepth>

	      which means that we can retrieve up to the whole submitted queue
	      depth,  but  if  none  of	IO has been completed yet, we will NOT
	      wait and immediately exit	the system call. In  this  example  we
	      simply do	polling.

       iodepth_low=int
	      Low  watermark indicating	when to	start filling the queue	again.
	      Default: iodepth.

       io_submit_mode=str
	      This option controls how fio submits the IO to  the  IO  engine.
	      The default is inline, which means that the fio job threads sub-
	      mit and reap IO directly.	 If set	to offload,  the  job  threads
	      will  offload  IO	 submission to a dedicated pool	of IO threads.
	      This requires some coordination and thus	has  a	bit  of	 extra
	      overhead,	 especially  for lower queue depth IO where it can in-
	      crease latencies.	The benefit is that fio	can manage  submission
	      rates  independently of the device completion rates. This	avoids
	      skewed latency reporting if IO gets back up on the  device  side
	      (the coordinated omission	problem).

       direct=bool
	      If  true,	 use  non-buffered  I/O	 (usually O_DIRECT).  Default:
	      false.

       atomic=bool
	      If value is true,	attempt	to use atomic direct IO. Atomic	writes
	      are  guaranteed  to be stable once acknowledged by the operating
	      system. Only Linux supports O_ATOMIC right now.

       buffered=bool
	      If true, use buffered I/O.  This is the opposite of  the	direct
	      parameter.  Default: true.

       offset=int
	      Start  I/O at the	provided offset	in the file, given as either a
	      fixed size or a percentage. If a percentage is given,  the  next
	      blockalign-ed  offset will be used. Data before the given	offset
	      will not be touched. This	effectively  caps  the	file  size  at
	      (real_size - offset). Can	be combined with size to constrain the
	      start and	end range of the I/O workload.

       offset_increment=int
	      If this is provided, then	the real offset	becomes	the  offset  +
	      offset_increment	*  thread_number, where	the thread number is a
	      counter that starts at 0 and is  incremented  for	 each  sub-job
	      (i.e.  when  numjobs option is specified). This option is	useful
	      if there are several jobs	which are intended  to	operate	 on  a
	      file  in	parallel  disjoint segments, with even spacing between
	      the starting points.

       number_ios=int
	      Fio will normally	perform	IOs until it has exhausted the size of
	      the  region set by size, or if it	exhaust	the allocated time (or
	      hits an error condition).	With this setting, the range/size  can
	      be  set  independently of	the number of IOs to perform. When fio
	      reaches this number, it will exit	normally  and  report  status.
	      Note  that  this	does  not extend the amount of IO that will be
	      done, it will only stop fio if  this  condition  is  met	before
	      other end-of-job criteria.

       fsync=int
	      How  many	 I/Os  to  perform before issuing an fsync(2) of dirty
	      data.  If	0, don't sync.	Default: 0.

       fdatasync=int
	      Like fsync, but uses fdatasync(2)	instead	to only	sync the  data
	      parts of the file. Default: 0.

       write_barrier=int
	      Make every Nth write a barrier write.

       sync_file_range=str:int
	      Use sync_file_range(2) for every val number of write operations.
	      Fio will track range of writes that have happened	since the last
	      sync_file_range(2) call.	str can	currently be one or more of:

	      wait_before
		     SYNC_FILE_RANGE_WAIT_BEFORE

	      write  SYNC_FILE_RANGE_WRITE

	      wait_after
		     SYNC_FILE_RANGE_WRITE

	      So  if  you  do  sync_file_range=wait_before,write:8, fio
	      would use
	      SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE for every  8
	      writes.	Also see the sync_file_range(2)	man page.  This	option
	      is Linux specific.

       overwrite=bool
	      If writing, setup	the file first and  do	overwrites.   Default:
	      false.

       end_fsync=bool
	      Sync  file  contents when	a write	stage has completed.  Default:
	      false.

       fsync_on_close=bool
	      If true,	sync  file  contents  on  close.   This	 differs  from
	      end_fsync	in that	it will	happen on every	close, not just	at the
	      end of the job.  Default:	false.

       rwmixread=int
	      Percentage of a mixed workload that should  be  reads.  Default:
	      50.

       rwmixwrite=int
	      Percentage  of  a	 mixed	workload  that	should	be writes.  If
	      rwmixread	and rwmixwrite are given and do	not sum	to  100%,  the
	      latter of	the two	overrides the first. This may interfere	with a
	      given rate setting, if fio is asked to limit reads or writes  to
	      a	 certain  rate.	If that	is the case, then the distribution may
	      be skewed. Default: 50.

       random_distribution=str:float
	      By default, fio will use a completely uniform  random  distribu-
	      tion  when asked to perform random IO. Sometimes it is useful to
	      skew the distribution in specific	ways, ensuring that some parts
	      of the data is more hot than others.  Fio	includes the following
	      distribution models:

	      random Uniform random distribution

	      zipf   Zipf distribution

	      pareto Pareto distribution

	      gauss  Normal (gaussian) distribution

	      zoned  Zoned random distribution

	      When using a zipf	or pareto distribution,	an input  value
	      is also
	      needed  to define	the access pattern. For	zipf, this is the zipf
	      theta.  For pareto, it's the pareto power. Fio includes  a  test
	      program,	genzipf, that can be used visualize what the given in-
	      put values will yield in terms of	hit rates. If  you  wanted  to
	      use  zipf	 with  a  theta	of 1.2,	you would use random_distribu-
	      tion=zipf:1.2 as the option. If a	non-uniform model is used, fio
	      will  disable use	of the random map. For the gauss distribution,
	      a	normal deviation is supplied as	a value	between	0 and 100.

	      For a zoned distribution,	fio supports specifying	percentages of
	      IO  access that should fall within what range of the file	or de-
	      vice. For	example, given a criteria of:

		     60% of accesses should be to the first 10%
		     30% of accesses should be to the next 20%
		     8%	of accesses should be to to the	next 30%
		     2%	of accesses should be to the next 40%

	      we can define that through zoning	of the	random	accesses.  For
	      the above	example, the user would	do:

		     random_distribution=zoned:60/10:30/20:8/30:2/40

	      similarly	 to  how bssplit works for setting ranges and percent-
	      ages of block sizes. Like	bssplit, it's possible to specify sep-
	      arate  zones  for	 reads,	 writes, and trims. If just one	set is
	      given, it'll apply to all	of them.

       percentage_random=int[,int][,int]
	      For a random workload, set how big a percentage should  be  ran-
	      dom.  This defaults to 100%, in which case the workload is fully
	      random. It can be	set from anywhere from 0 to 100.   Setting  it
	      to 0 would make the workload fully sequential. It	is possible to
	      set different values for reads, writes, and trim.	To do so, sim-
	      ply use a	comma separated	list. See blocksize.

       norandommap
	      Normally	fio will cover every block of the file when doing ran-
	      dom I/O. If this parameter is given, a new offset	will be	chosen
	      without looking at past I/O history.  This parameter is mutually
	      exclusive	with verify.

       softrandommap=bool
	      See norandommap. If fio runs with	the random block  map  enabled
	      and  it fails to allocate	the map, if this option	is set it will
	      continue without a random	block map. As coverage will not	be  as
	      complete	as  with  random  maps,	this option is disabled	by de-
	      fault.

       random_generator=str
	      Fio supports the following engines for generating	IO offsets for
	      random IO:

	      tausworthe
		     Strong 2^88 cycle random number generator

	      lfsr   Linear feedback shift register generator

	      tausworthe64
		     Strong 64-bit 2^258 cycle random number generator

	      Tausworthe  is  a	 strong	random number generator, but it
	      requires tracking	on the
	      side if we want to ensure	that blocks are	only read  or  written
	      once.  LFSR  guarantees  that  we	never generate the same	offset
	      twice, and it's also less	computationally	expensive. It's	not  a
	      true random generator, however, though for IO purposes it's typ-
	      ically good enough. LFSR only works with single block sizes, not
	      with  workloads that use multiple	block sizes. If	used with such
	      a	workload, fio may read or write	some  blocks  multiple	times.
	      The  default  value is tausworthe, unless	the required space ex-
	      ceeds 2^32 blocks. If it does, then tausworthe64 is selected au-
	      tomatically.

       nice=int
	      Run job with given nice value.  See nice(2).

       prio=int
	      Set  I/O	priority  value	 of this job between 0 (highest) and 7
	      (lowest).	 See ionice(1).

       prioclass=int
	      Set I/O priority class.  See ionice(1).

       thinktime=int
	      Stall job	for given number of microseconds between issuing I/Os.

       thinktime_spin=int
	      Pretend to spend CPU time	 for  given  number  of	 microseconds,
	      sleeping	the  rest  of  the  time specified by thinktime.  Only
	      valid if thinktime is set.

       thinktime_blocks=int
	      Only valid if thinktime is set - control how many	blocks to  is-
	      sue, before waiting thinktime microseconds. If not set, defaults
	      to 1 which will make fio wait thinktime microseconds after every
	      block. This effectively makes any	queue depth setting redundant,
	      since no more than 1 IO will be queued before we	have  to  com-
	      plete  it	and do our thinktime. In other words, this setting ef-
	      fectively	caps the queue depth if	the  latter  is	 larger.   De-
	      fault: 1.

       rate=int[,int][,int]
	      Cap  bandwidth used by this job. The number is in	bytes/sec, the
	      normal postfix rules apply. You can use rate=500k	to limit reads
	      and  writes  to  500k each, or you can specify reads, write, and
	      trim limits separately.  Using rate=1m,500k would	limit reads to
	      1MiB/sec	and writes to 500KiB/sec. Capping only reads or	writes
	      can be done with rate=,500k or rate=500k,. The former will  only
	      limit writes (to 500KiB/sec), the	latter will only limit reads.

       rate_min=int[,int][,int]
	      Tell  fio	 to  do	whatever it can	to maintain at least the given
	      bandwidth.  Failing to meet this requirement will	cause the  job
	      to  exit.	 The  same format as rate is used for read vs write vs
	      trim separation.

       rate_iops=int[,int][,int]
	      Cap the bandwidth	to this	number of IOPS.	Basically the same  as
	      rate, just specified independently of bandwidth. The same	format
	      as rate is used for read vs write	vs trim	separation. If	block-
	      size is a	range, the smallest block size is used as the metric.

       rate_iops_min=int[,int][,int]
	      If this rate of I/O is not met, the job will exit. The same for-
	      mat as rate is used for read vs write vs trim separation.

       rate_process=str
	      This option controls how fio manages rated IO  submissions.  The
	      default  is  linear,  which  submits IO in a linear fashion with
	      fixed delays between IOs that gets adjusted based	on IO  comple-
	      tion  rates. If this is set to poisson, fio will submit IO based
	      on a more	real world random request flow,	known as  the  Poisson
	      process	(https://en.wikipedia.org/wiki/Poisson_process).   The
	      lambda will be 10^6 / IOPS for the given workload.

       rate_cycle=int
	      Average bandwidth	for rate and rate_min over this	number of mil-
	      liseconds.  Default: 1000ms.

       latency_target=int
	      If  set, fio will	attempt	to find	the max	performance point that
	      the given	workload will run at while maintaining a latency below
	      this  target.  The  values  is  given  in	 microseconds. See la-
	      tency_window and latency_percentile.

       latency_window=int
	      Used with	latency_target to specify the sample window  that  the
	      job  is run at varying queue depths to test the performance. The
	      value is given in	microseconds.

       latency_percentile=float
	      The percentage of	IOs that must fall within the criteria	speci-
	      fied  by latency_target and latency_window. If not set, this de-
	      faults to	100.0, meaning that all	IOs must be equal or below  to
	      the value	set by latency_target.

       max_latency=int
	      If  set,	fio  will  exit	the job	if it exceeds this maximum la-
	      tency. It	will exit with an ETIME	error.

       cpumask=int
	      Set CPU affinity for this	job. int is a bitmask of allowed  CPUs
	      the job may run on.  See sched_setaffinity(2).

       cpus_allowed=str
	      Same  as	cpumask, but allows a comma-delimited list of CPU num-
	      bers.

       cpus_allowed_policy=str
	      Set the policy of	how fio	 distributes  the  CPUs	 specified  by
	      cpus_allowed or cpumask. Two policies are	supported:

		     shared All	jobs will share	the CPU	set specified.

		     split  Each job will get a	unique CPU from	the CPU	set.

	      shared  is the default behaviour,	if the option isn't specified.
	      If split is specified, then fio will assign one cpu per job.  If
	      not  enough  CPUs	 are  given for	the jobs listed, then fio will
	      roundrobin the CPUs in the set.

       numa_cpu_nodes=str
	      Set this job running on specified	NUMA nodes'  CPUs.  The	 argu-
	      ments  allow comma delimited list	of cpu numbers,	A-B ranges, or
	      'all'.

       numa_mem_policy=str
	      Set this job's memory policy and corresponding NUMA nodes.  For-
	      mat of the arguments:

	      <mode>[:<nodelist>]

	      mode   is	one of the following memory policy:

	      default, prefer, bind, interleave, local

	      For default and local memory policy, no nodelist is
	      needed  to  be  specified. For prefer, only one node is allowed.
	      For bind and interleave, nodelist	allows comma delimited list of
	      numbers, A-B ranges, or 'all'.

       startdelay=irange
	      Delay start of job for the specified number of seconds. Supports
	      all time suffixes	to allow specification of hours, minutes, sec-
	      onds  and	 milliseconds  -  seconds are the default if a unit is
	      omitted.	Can be given as	a range	which causes  each  thread  to
	      choose randomly out of the range.

       runtime=int
	      Terminate	processing after the specified number of seconds.

       time_based
	      If  given,  run  for  the	specified runtime duration even	if the
	      files are	completely read	or written. The	same workload will  be
	      repeated as many times as	runtime	allows.

       ramp_time=int
	      If  set,	fio will run the specified workload for	this amount of
	      time before logging any performance numbers. Useful for  letting
	      performance  settle  before logging results, thus	minimizing the
	      runtime required for stable results. Note	that the ramp_time  is
	      considered lead in time for a job, thus it will increase the to-
	      tal runtime if a special timeout or runtime is specified.

       steadystate=str:float, ss=str:float
	      Define the criterion and limit for assessing steady  state  per-
	      formance.	 The  first parameter designates the criterion whereas
	      the second parameter sets	 the  threshold.  When	the  criterion
	      falls  below  the	 threshold for the specified duration, the job
	      will stop. For example, iops_slope:0.1% will direct fio to  ter-
	      minate the job when the least squares regression slope falls be-
	      low 0.1% of the mean IOPS. If group_reporting  is	 enabled  this
	      will apply to all	jobs in	the group. All assessments are carried
	      out using	only data from the rolling collection window.  Thresh-
	      old  limits can be expressed as a	fixed value or as a percentage
	      of the mean in the collection window. Below  are	the  available
	      steady state assessment criteria.

		     iops   Collect  IOPS data.	Stop the job if	all individual
			    IOPS measurements are within the  specified	 limit
			    of	the mean IOPS (e.g., iops:2 means that all in-
			    dividual IOPS values must be within	2 of the mean,
			    whereas  iops:0.2%	means that all individual IOPS
			    values must	be within 0.2% of  the	mean  IOPS  to
			    terminate the job).

		     iops_slope
			    Collect  IOPS data and calculate the least squares
			    regression slope. Stop the job if the slope	 falls
			    below the specified	limit.

		     bw	    Collect  bandwidth data. Stop the job if all indi-
			    vidual bandwidth measurements are within the spec-
			    ified limit	of the mean bandwidth.

		     bw_slope
			    Collect  bandwidth	data  and  calculate the least
			    squares regression slope.  Stop  the  job  if  the
			    slope falls	below the specified limit.

       steadystate_duration=time, ss_dur=time
	      A	 rolling window	of this	duration will be used to judge whether
	      steady state has been reached. Data will be collected  once  per
	      second. The default is 0 which disables steady state detection.

       steadystate_ramp_time=time, ss_ramp=time
	      Allow the	job to run for the specified duration before beginning
	      data collection for checking the steady  state  job  termination
	      criterion. The default is	0.

       invalidate=bool
	      Invalidate buffer-cache for the file prior to starting I/O.  De-
	      fault: true.

       sync=bool
	      Use synchronous I/O for buffered writes.	For  the  majority  of
	      I/O engines, this	means using O_SYNC.  Default: false.

       iomem=str, mem=str
	      Allocation method	for I/O	unit buffer.  Allowed values are:

		     malloc Allocate  memory  with  malloc(3).	Default	memory
			    type.

		     shm    Use	 shared	 memory	 buffers   allocated   through
			    shmget(2).

		     shmhuge
			    Same as shm, but use huge pages as backing.

		     mmap   Use	mmap(2)	for allocation.	 Uses anonymous	memory
			    unless a filename is given after the option	in the
			    format `:file'.

		     mmaphuge
			    Same as mmap, but use huge files as	backing.

		     mmapshared
			    Same as mmap, but use a MMAP_SHARED	mapping.

		     cudamalloc
			    Use	 GPU  memory as	the buffers for	GPUDirect RDMA
			    benchmark. The ioengine must be rdma.

	      The amount of memory allocated is	the maximum allowed  blocksize
	      for  the	job multiplied by iodepth.  For	shmhuge	or mmaphuge to
	      work, the	system must have free huge pages allocated.   mmaphuge
	      also needs to have hugetlbfs mounted, and	file must point	there.
	      At least on Linux, huge pages must be  manually  allocated.  See
	      /proc/sys/vm/nr_hugehages	 and  the documentation	for that. Nor-
	      mally you	just need to echo an appropriate number, eg echoing  8
	      will ensure that the OS has 8 huge pages ready for use.

       iomem_align=int,	mem_align=int
	      This  indicates  the  memory alignment of	the IO memory buffers.
	      Note that	the given alignment is applied to the  first  IO  unit
	      buffer,  if using	iodepth	the alignment of the following buffers
	      are given	by the bs used.	In other words,	if using a bs that  is
	      a	 multiple of the page sized in the system, all buffers will be
	      aligned to this value. If	using a	bs that	is not	page  aligned,
	      the  alignment of	subsequent IO memory buffers is	the sum	of the
	      iomem_align and bs used.

       hugepage-size=int
	      Defines the size of a huge page.	Must be	at least equal to  the
	      system setting.  Should be a multiple of 1MiB. Default: 4MiB.

       exitall
	      Terminate	 all  jobs  when one finishes.	Default: wait for each
	      job to finish.

       exitall_on_error	=bool
	      Terminate	all jobs if one	job finishes in	error.	Default:  wait
	      for each job to finish.

       bwavgtime=int
	      Average  bandwidth calculations over the given time in millisec-
	      onds.  If	 the  job  also	  does	 bandwidth   logging   through
	      write_bw_log,  then  the minimum of this option and log_avg_msec
	      will be used.  Default: 500ms.

       iopsavgtime=int
	      Average IOPS calculations	over the given time  in	 milliseconds.
	      If  the  job also	does IOPS logging through write_iops_log, then
	      the minimum of this option and log_avg_msec will be  used.   De-
	      fault: 500ms.

       create_serialize=bool
	      If true, serialize file creation for the jobs.  Default: true.

       create_fsync=bool
	      fsync(2) data file after creation.  Default: true.

       create_on_open=bool
	      If  true,	the files are not created until	they are opened	for IO
	      by the job.

       create_only=bool
	      If true, fio will	only run the setup phase of the	job. If	 files
	      need  to be laid out or updated on disk, only that will be done.
	      The actual job contents are not executed.

       allow_file_create=bool
	      If true, fio is permitted	to create files	as part	of  its	 work-
	      load.  This  is  the  default behavior. If this option is	false,
	      then fio will error out if the files it needs to use  don't  al-
	      ready exist. Default: true.

       allow_mounted_write=bool
	      If  this isn't set, fio will abort jobs that are destructive (eg
	      that write) to what appears to be	a mounted device or partition.
	      This should help catch creating inadvertently destructive	tests,
	      not realizing that the test will destroy	data  on  the  mounted
	      file system. Default: false.

       pre_read=bool
	      If  this	is  given,  files  will	be pre-read into memory	before
	      starting the given IO operation. This will also clear  the   in-
	      validate	flag,  since it	is pointless to	pre-read and then drop
	      the cache. This will only	work for IO engines that are seekable,
	      since  they allow	you to read the	same data multiple times. Thus
	      it will not work on eg network or	splice IO.

       unlink=bool
	      Unlink job files when done.  Default: false.

       unlink_each_loop=bool
	      Unlink job files after each iteration or loop.  Default: false.

       loops=int
	      Specifies	the number of iterations (runs of the  same  workload)
	      of this job.  Default: 1.

       verify_only=bool
	      Do  not  perform	the specified workload,	only verify data still
	      matches previous invocation of this workload. This option	allows
	      one  to  check data multiple times at a later date without over-
	      writing it. This option makes  sense  only  for  workloads  that
	      write  data,  and	does not support workloads with	the time_based
	      option set.

       do_verify=bool
	      Run the verify phase after a write phase.	 Only valid if	verify
	      is set.  Default:	true.

       verify=str
	      Method  of  verifying  file contents after each iteration	of the
	      job. Each	verification method also implies verification of  spe-
	      cial  header,  which  is written to the beginning	of each	block.
	      This header also includes	meta information, like offset  of  the
	      block,  block  number,  timestamp	 when  block was written, etc.
	      verify=str can be	combined with verify_pattern=str option.   The
	      allowed values are:

		     md5  crc16	 crc32	crc32c	crc32c-intel crc64 crc7	sha256
		     sha512 sha1 sha3-224 sha3-256 sha3-384 sha3-512 xxhash
			    Store appropriate checksum in the header  of  each
			    block. crc32c-intel	is hardware accelerated	SSE4.2
			    driven, falls back to regular crc32c if  not  sup-
			    ported by the system.

		     meta   This option	is deprecated, since now meta informa-
			    tion is included in	 generic  verification	header
			    and	meta verification happens by default.  For de-
			    tailed information see the description of the ver-
			    ify=str  setting.  This  option is kept because of
			    compatibility's sake with old  configurations.  Do
			    not	use it.

		     pattern
			    Verify  a  strict pattern. Normally	fio includes a
			    header with	some basic information	and  checksum-
			    ming, but if this option is	set, only the specific
			    pattern set	with verify_pattern is verified.

		     null   Pretend to verify.	Used for testing internals.

	      This option can be used for repeated burn-in tests of  a	system
	      to  make sure that the written data is also correctly read back.
	      If the data direction given is a read or random read,  fio  will
	      assume  that  it should verify a previously written file.	If the
	      data direction includes any form of write, the verify will be of
	      the newly	written	data.

       verifysort=bool
	      If  true,	written	verify blocks are sorted if fio	deems it to be
	      faster to	read them back in a sorted manner.  Default: true.

       verifysort_nr=int
	      Pre-load and sort	verify blocks for a read workload.

       verify_offset=int
	      Swap the verification header with	data  somewhere	 else  in  the
	      block before writing.  It	is swapped back	before verifying.

       verify_interval=int
	      Write  the  verification	header for this	number of bytes, which
	      should divide blocksize.	Default: blocksize.

       verify_pattern=str
	      If set, fio will fill the	io buffers with	this pattern. Fio  de-
	      faults  to filling with totally random bytes, but	sometimes it's
	      interesting to fill with a known	pattern	 for  io  verification
	      purposes.	 Depending  on the width of the	pattern, fio will fill
	      1/2/3/4 bytes of the buffer at the time(it can be	either a deci-
	      mal or a hex number). The	verify_pattern if larger than a	32-bit
	      quantity has to be a hex number that starts with either "0x"  or
	      "0X". Use	with verify=str. Also, verify_pattern supports %o for-
	      mat, which means that for	each block offset will be written  and
	      then verified back, e.g.:
		     verify_pattern=%o
	      Or use combination of everything:

		     verify_pattern=0xff%o"abcd"-21

       verify_fatal=bool
	      If  true,	 exit the job on the first observed verification fail-
	      ure.  Default: false.

       verify_dump=bool
	      If set, dump the contents	of both	the original  data  block  and
	      the  data	 block	we  read  off disk to files. This allows later
	      analysis to inspect just what kind of data corruption  occurred.
	      Off by default.

       verify_async=int
	      Fio  will	 normally verify IO inline from	the submitting thread.
	      This option takes	an integer describing how many	async  offload
	      threads  to  create  for IO verification instead,	causing	fio to
	      offload the duty of verifying IO contents	to one or  more	 sepa-
	      rate  threads.   If  using this offload option, even sync	IO en-
	      gines can	benefit	from using an iodepth setting higher  than  1,
	      as  it  allows them to have IO in	flight while verifies are run-
	      ning.

       verify_async_cpus=str
	      Tell fio to set the given	CPU affinity on	the async IO verifica-
	      tion threads.  See cpus_allowed for the format used.

       verify_backlog=int
	      Fio will normally	verify the written contents of a job that uti-
	      lizes verify once	that job has completed.	In other words,	every-
	      thing  is	written	then everything	is read	back and verified. You
	      may want to verify continually instead for a variety of reasons.
	      Fio  stores the meta data	associated with	an IO block in memory,
	      so for large verify workloads, quite a bit of  memory  would  be
	      used  up	holding	this meta data.	If this	option is enabled, fio
	      will write only N	blocks before verifying	these blocks.

       verify_backlog_batch=int
	      Control how many blocks fio will	verify	if  verify_backlog  is
	      set.  If	not  set,  will	default	to the value of	verify_backlog
	      (meaning the entire queue	is read	back and verified).   If  ver-
	      ify_backlog_batch	 is  less  than	 verify_backlog	 then  not all
	      blocks will be verified,	if verify_backlog_batch	is larger than
	      verify_backlog,  some blocks will	be verified more than once.

       trim_percentage=int
	      Number of	verify blocks to discard/trim.

       trim_verify_zero=bool
	      Verify that trim/discarded blocks	are returned as	zeroes.

       trim_backlog=int
	      Trim after this number of	blocks are written.

       trim_backlog_batch=int
	      Trim this	number of IO blocks.

       experimental_verify=bool
	      Enable experimental verification.

       verify_state_save=bool
	      When  a  job  exits during the write phase of a verify workload,
	      save its current state. This allows fio to replay	up until  that
	      point, if	the verify state is loaded for the verify read phase.

       verify_state_load=bool
	      If a verify termination trigger was used,	fio stores the current
	      write state of each thread. This can  be	used  at  verification
	      time  so	that  fio knows	how far	it should verify. Without this
	      information, fio will run	a full verification pass, according to
	      the settings in the job file used.

       stonewall , wait_for_previous
	      Wait  for	preceding jobs in the job file to exit before starting
	      this one.	 stonewall implies new_group.

       new_group
	      Start a new reporting group.  If not given, all jobs in  a  file
	      will  be part of the same	reporting group, unless	separated by a
	      stonewall.

       stats=bool
	      By default, fio collects and shows final output results for  all
	      jobs that	run.  If this option is	set to 0, then fio will	ignore
	      it in the	final stat output.

       numjobs=int
	      Number of	clones (processes/threads performing  the  same	 work-
	      load) of this job.  Default: 1.

       group_reporting
	      If  set,	display	 per-group  reports  instead  of  per-job when
	      numjobs is specified.

       thread Use threads created with pthread_create(3) instead of  processes
	      created with fork(2).

       zonesize=int
	      Divide  file  into  zones	 of  the specified size	in bytes.  See
	      zoneskip.

       zonerange=int
	      Give size	of an IO zone.	See zoneskip.

       zoneskip=int
	      Skip the specified number	of bytes when zonesize bytes  of  data
	      have been	read.

       write_iolog=str
	      Write  the issued	I/O patterns to	the specified file.  Specify a
	      separate file for	each job, otherwise the	iologs will be	inter-
	      spersed and the file may be corrupt.

       read_iolog=str
	      Replay  the  I/O patterns	contained in the specified file	gener-
	      ated by write_iolog, or may be a blktrace	binary file.

       replay_no_stall=int
	      While replaying I/O patterns using read_iolog the	default	behav-
	      ior  attempts  to	 respect timing	information between I/Os.  En-
	      abling replay_no_stall causes I/Os to be	replayed  as  fast  as
	      possible while still respecting ordering.

       replay_redirect=str
	      While replaying I/O patterns using read_iolog the	default	behav-
	      ior is to	replay the IOPS	onto the major/minor device that  each
	      IOP  was recorded	from.  Setting replay_redirect causes all IOPS
	      to be replayed onto the single specified	device	regardless  of
	      the device it was	recorded from.

       replay_align=int
	      Force  alignment	of  IO	offsets	and lengths in a trace to this
	      power of 2 value.

       replay_scale=int
	      Scale sector offsets down	by this	factor when replaying traces.

       per_job_logs=bool
	      If set, this generates bw/clat/iops log with  per	 file  private
	      filenames.  If not set, jobs with	identical names	will share the
	      log filename. Default: true.

       write_bw_log=str
	      If given,	write a	bandwidth log for this job.  Can  be  used  to
	      store  data  of the bandwidth of the jobs	in their lifetime. The
	      included fio_generate_plots script uses gnuplot  to  turn	 these
	      text  files into nice graphs. See	write_lat_log for behaviour of
	      given filename. For this option, the postfix is _bw.x.log, where
	      x	is the index of	the job	(1..N, where N is the number of	jobs).
	      If per_job_logs is false,	then the filename will not include the
	      job index.  See the LOG FILE FORMATS section.

       write_lat_log=str
	      Same  as	write_bw_log, but writes I/O completion	latencies.  If
	      no filename is given with	this option, the default  filename  of
	      "jobname_type.x.log"  is	used,  where x is the index of the job
	      (1..N, where N is	the number of jobs). Even if the  filename  is
	      given, fio will still append the type of log. If per_job_logs is
	      false, then the filename will not	include	the job	index. See the
	      LOG FILE FORMATS section.

       write_hist_log=str
	      Same  as	write_lat_log,	but writes I/O completion latency his-
	      tograms. If no filename is given with this option,  the  default
	      filename	of  "jobname_clat_hist.x.log"  is used,	where x	is the
	      index of the job (1..N, where N is the number of jobs). Even  if
	      the filename is given, fio will still append the type of log. If
	      per_job_logs is false, then the filename will  not  include  the
	      job index. See the LOG FILE FORMATS section.

       write_iops_log=str
	      Same  as	write_bw_log, but writes IOPS. If no filename is given
	      with this	option,	the default filename  of  "jobname_type.x.log"
	      is  used,	 where x is the	index of the job (1..N,	where N	is the
	      number of	jobs). Even if the filename is given, fio  will	 still
	      append the type of log. If per_job_logs is false,	then the file-
	      name will	not include the	job index. See the  LOG	 FILE  FORMATS
	      section.

       log_avg_msec=int
	      By  default,  fio	 will log an entry in the iops,	latency, or bw
	      log for every IO that completes. When writing to the  disk  log,
	      that  can	quickly	grow to	a very large size. Setting this	option
	      makes fio	average	the each log entry over	the  specified	period
	      of  time,	 reducing the resolution of the	log. See log_max_value
	      as well.	Defaults to 0, logging all entries.

       log_max_value=bool
	      If log_avg_msec is set, fio logs the average over	 that  window.
	      If you instead want to log the maximum value, set	this option to
	      1.  Defaults to 0, meaning that averaged values are logged.

       log_hist_msec=int
	      Same as log_avg_msec, but	logs entries  for  completion  latency
	      histograms.   Computing latency percentiles from averages	of in-
	      tervals using log_avg_msec is innacurate.	 Setting  this	option
	      makes  fio  log  histogram  entries over the specified period of
	      time, reducing log sizes for high	IOPS devices  while  retaining
	      percentile  accuracy.  See log_hist_coarseness as	well. Defaults
	      to 0, meaning histogram logging is disabled.

       log_hist_coarseness=int
	      Integer ranging from 0 to	6, defining the	coarseness of the res-
	      olution  of  the	histogram logs enabled with log_hist_msec. For
	      each increment in	coarseness, fio	outputs	half as	many bins. De-
	      faults to	0, for which histogram logs contain 1216 latency bins.
	      See the LOG FILE FORMATS section.

       log_offset=bool
	      If this is set, the iolog	options	will include the  byte	offset
	      for the IO entry as well as the other data values.

       log_compression=int
	      If  this	is  set,  fio will compress the	IO logs	as it goes, to
	      keep the memory footprint	lower. When a log reaches  the	speci-
	      fied  size,  that	 chunk	is removed and compressed in the back-
	      ground. Given that IO logs are fairly highly compressible,  this
	      yields  a	 nice  memory savings for longer runs. The downside is
	      that the compression will	consume	some background	CPU cycles, so
	      it  may  impact the run. This, however, is also true if the log-
	      ging ends	up consuming most of the system	memory.	So  pick  your
	      poison.  The  IO logs are	saved normally at the end of a run, by
	      decompressing the	chunks and storing them	in the	specified  log
	      file. This feature depends on the	availability of	zlib.

       log_compression_cpus=str
	      Define  the  set	of  CPUs that are allowed to handle online log
	      compression for the IO jobs. This	can provide  better  isolation
	      between  performance  sensitive jobs, and	background compression
	      work.

       log_store_compressed=bool
	      If set, fio will store the log files  in	a  compressed  format.
	      They  can	be decompressed	with fio, using	the --inflate-log com-
	      mand line	parameter.  The	files will be stored with a  .fz  suf-
	      fix.

       log_unix_epoch=bool
	      If  set,	fio will log Unix timestamps to	the log	files produced
	      by enabling write_type_log for each log type, instead of the de-
	      fault zero-based timestamps.

       block_error_percentiles=bool
	      If  set, record errors in	trim block-sized units from writes and
	      trims and	output a histogram of how many trims it	took to	get to
	      errors, and what kind of error was encountered.

       disable_lat=bool
	      Disable  measurements  of	total latency numbers. Useful only for
	      cutting back the number of calls	to  gettimeofday(2),  as  that
	      does impact performance at really	high IOPS rates.  Note that to
	      really get rid of	a large	amount of  these  calls,  this	option
	      must be used with	disable_slat and disable_bw as well.

       disable_clat=bool
	      Disable  measurements  of	 completion  latency numbers. See dis-
	      able_lat.

       disable_slat=bool
	      Disable measurements of submission  latency  numbers.  See  dis-
	      able_lat.

       disable_bw_measurement=bool
	      Disable  measurements  of	throughput/bandwidth numbers. See dis-
	      able_lat.

       lockmem=int
	      Pin the specified	amount of memory with mlock(2).	 Can  be  used
	      to  simulate a smaller amount of memory. The amount specified is
	      per worker.

       exec_prerun=str
	      Before running the job, execute the specified command with  sys-
	      tem(3).
	      Output is	redirected in a	file called jobname.prerun.txt

       exec_postrun=str
	      Same  as	exec_prerun, but the command is	executed after the job
	      completes.
	      Output is	redirected in a	file called jobname.postrun.txt

       ioscheduler=str
	      Attempt to switch	the device hosting the file to	the  specified
	      I/O scheduler.

       disk_util=bool
	      Generate	disk  utilization  statistics if the platform supports
	      it. Default: true.

       clocksource=str
	      Use the given clocksource	as the base of timing.	The  supported
	      options are:

	      gettimeofday
		     gettimeofday(2)

	      clock_gettime
		     clock_gettime(2)

	      cpu    Internal CPU clock	source

	      cpu is the preferred clocksource if it is	reliable, as it
	      is very fast
	      (and fio is heavy	on time	calls).	 Fio  will  automatically  use
	      this  clocksource	 if  it's supported and	considered reliable on
	      the system it is	running	 on,  unless  another  clocksource  is
	      specifically set.	For x86/x86-64 CPUs, this means	supporting TSC
	      Invariant.

       gtod_reduce=bool
	      Enable  all  of  the  gettimeofday(2)  reducing  options	 (dis-
	      able_clat,  disable_slat,	 disable_bw)  plus reduce precision of
	      the timeout somewhat to really shrink the	 gettimeofday(2)  call
	      count.  With  this  option enabled, we only do about 0.4%	of the
	      gtod() calls we would have done if all time keeping was enabled.

       gtod_cpu=int
	      Sometimes	it's cheaper to	dedicate a single thread of  execution
	      to  just	getting	 the current time. Fio (and databases, for in-
	      stance) are very intensive on gettimeofday(2) calls.  With  this
	      option,  you can set one CPU aside for doing nothing but logging
	      current time  to	a  shared  memory  location.  Then  the	 other
	      threads/processes	that run IO workloads need only	copy that seg-
	      ment, instead of entering	 the  kernel  with  a  gettimeofday(2)
	      call.  The  CPU set aside	for doing these	time calls will	be ex-
	      cluded from other	uses. Fio will manually	clear it from the  CPU
	      mask of other jobs.

       ignore_error=str
	      Sometimes	 you  want  to	ignore some errors during test in that
	      case you can specify error list for each error type.
	      ignore_error=READ_ERR_LIST,WRITE_ERR_LIST,VERIFY_ERR_LIST
	      errors for given error type is separated with ':'.  Error	may be
	      symbol ('ENOSPC',	'ENOMEM') or an	integer.
	      Example: ignore_error=EAGAIN,ENOSPC:122 .
	      This  option  will  ignore  EAGAIN  from	READ,  and  ENOSPC and
	      122(EDQUOT) from WRITE.

       error_dump=bool
	      If set dump every	error even if it is non	 fatal,	 true  by  de-
	      fault. If	disabled only fatal error will be dumped

       profile=str
	      Select a specific	builtin	performance test.

       cgroup=str
	      Add  job	to this	control	group. If it doesn't exist, it will be
	      created.	The system must	have  a	 mounted  cgroup  blkio	 mount
	      point  for this to work. If your system doesn't have it mounted,
	      you can do so with:

	      #	mount -t cgroup	-o blkio none /cgroup

       cgroup_weight=int
	      Set the weight of	the cgroup to this value. See  the  documenta-
	      tion that	comes with the kernel, allowed values are in the range
	      of 100..1000.

       cgroup_nodelete=bool
	      Normally fio will	delete the cgroups it has  created  after  the
	      job  completion.	To override this behavior and to leave cgroups
	      around after the job completion, set cgroup_nodelete=1. This can
	      be useful	if one wants to	inspect	various	cgroup files after job
	      completion. Default: false

       uid=int
	      Instead of running as the	invoking user, set the user ID to this
	      value before the thread/process does any work.

       gid=int
	      Set group	ID, see	uid.

       unit_base=int
	      Base unit	for reporting.	Allowed	values are:

	      0	     Use auto-detection	(default).

	      8	     Byte based.

	      1	     Bit based.

       flow_id=int
	      The  ID  of  the	flow. If not specified,	it defaults to being a
	      global flow. See flow.

       flow=int
	      Weight in	token-based flow control. If this value	is used,  then
	      there is a flow counter which is used to regulate	the proportion
	      of activity between two or more jobs. fio	attempts to keep  this
	      flow  counter  near zero.	The flow parameter stands for how much
	      should be	added or subtracted to the flow	counter	on each	itera-
	      tion  of	the  main I/O loop. That is, if	one job	has flow=8 and
	      another job has flow=-1, then there will be a roughly 1:8	 ratio
	      in how much one runs vs the other.

       flow_watermark=int
	      The maximum value	that the absolute value	of the flow counter is
	      allowed to reach before the job must wait	for a lower  value  of
	      the counter.

       flow_sleep=int
	      The  period of time, in microseconds, to wait after the flow wa-
	      termark has been exceeded	before retrying	operations

       clat_percentiles=bool
	      Enable the reporting of percentiles of completion	latencies.

       percentile_list=float_list
	      Overwrite	the default list of percentiles	for completion	laten-
	      cies  and	 the  block error histogram. Each number is a floating
	      number in	the range (0,100], and the maximum length of the  list
	      is  20.  Use  ':'	 to  separate the numbers. For example,	--per-
	      centile_list=99.5:99.9 will cause	fio to report  the  values  of
	      completion  latency  below which 99.5% and 99.9% of the observed
	      latencies	fell, respectively.

   Ioengine Parameters List
       Some parameters are only	valid when a  specific	ioengine  is  in  use.
       These  are  used	identically to normal parameters, with the caveat that
       when used on the	command	line, they must	come after the ioengine.

       (cpuio)cpuload=int
	      Attempt to use the specified percentage of CPU cycles.

       (cpuio)cpuchunks=int
	      Split the	load into cycles of the	given time. In microseconds.

       (cpuio)exit_on_io_done=bool
	      Detect when IO threads are done, then exit.

       (libaio)userspace_reap
	      Normally,	with the libaio	 engine	 in  use,  fio	will  use  the
	      io_getevents  system  call  to reap newly	returned events.  With
	      this flag	turned on, the AIO ring	will  be  read	directly  from
	      user-space to reap events. The reaping mode is only enabled when
	      polling for a minimum of 0 events	 (eg  when  iodepth_batch_com-
	      plete=0).

       (pvsync2)hipri
	      Set  RWF_HIPRI  on  IO,  indicating  to  the kernel that it's of
	      higher priority than normal.

       (net,netsplice)hostname=str
	      The host name or IP address to use for TCP or UDP	based IO.   If
	      the  job	is  a  TCP listener or UDP reader, the hostname	is not
	      used and must be omitted unless it is a valid UDP	multicast  ad-
	      dress.

       (net,netsplice)port=int
	      The  TCP	or  UDP	port to	bind to	or connect to. If this is used
	      with numjobs to spawn multiple instances of the same  job	 type,
	      then  this will be the starting port number since	fio will use a
	      range of ports.

       (net,netsplice)interface=str
	      The IP address of	the network interface used to send or  receive
	      UDP multicast packets.

       (net,netsplice)ttl=int
	      Time-to-live  value for outgoing UDP multicast packets. Default:
	      1

       (net,netsplice)nodelay=bool
	      Set TCP_NODELAY on TCP connections.

       (net,netsplice)protocol=str, proto=str
	      The network protocol to use. Accepted values are:

		     tcp    Transmission control protocol

		     tcpv6  Transmission control protocol V6

		     udp    User datagram protocol

		     udpv6  User datagram protocol V6

		     unix   UNIX domain	socket

	      When the protocol	is TCP or UDP, the port	must also be given, as
	      well as the hostname if the job is a TCP listener	or UDP reader.
	      For unix sockets,	the normal filename option should be used  and
	      the port is invalid.

       (net,netsplice)listen
	      For  TCP	network	 connections,  tell fio	to listen for incoming
	      connections rather than initiating an outgoing  connection.  The
	      hostname must be omitted if this option is used.

       (net,pingpong)=bool
	      Normally a network writer	will just continue writing data, and a
	      network reader will just consume packets.	If pingpong=1 is  set,
	      a	 writer	 will send its normal payload to the reader, then wait
	      for the reader to	send the same payload back.  This  allows  fio
	      to  measure network latencies. The submission and	completion la-
	      tencies then measure local time spent sending or receiving,  and
	      the  completion  latency measures	how long it took for the other
	      end to receive and send back. For	UDP  multicast	traffic	 ping-
	      pong=1  should  only  be	set  for a single reader when multiple
	      readers are listening to the same	address.

       (net,window_size)=int
	      Set the desired socket buffer size for the connection.

       (net,mss)=int
	      Set the TCP maximum segment size (TCP_MAXSEG).

       (e4defrag,donorname)=str
	      File will	be used	as a block donor (swap extents between files)

       (e4defrag,inplace)=int
	      Configure	donor file block allocation strategy
	      0(default): Preallocate donor's file on init

	      1:     allocate space immediately	inside defragment  event,  and
		     free right	after event

       (rbd)clustername=str
	      Specifies	the name of the	ceph cluster.

       (rbd)rbdname=str
	      Specifies	the name of the	RBD.

       (rbd)pool=str
	      Specifies	the name of the	Ceph pool containing the RBD.

       (rbd)clientname=str
	      Specifies	 the  username	(without the 'client.' prefix) used to
	      access the Ceph cluster. If the clustername  is  specified,  the
	      clientname  shall	be the full type.id string. If no type.	prefix
	      is given,	fio will add 'client.' by default.

       (mtd)skipbad=bool
	      Skip operations against known bad	blocks.

OUTPUT
       While running, fio will display the status of the  created  jobs.   For
       example:

	      Jobs:   1:   [_r]	  [24.8%  done]	 [  13509/   8334  kb/s]  [eta
	      00h:01m:31s]

       The characters in the first set of brackets denote the  current	status
       of each threads.	 The possible values are:

	      P	     Setup but not started.
	      C	     Thread created.
	      I	     Initialized, waiting.
	      R	     Running, doing sequential reads.
	      r	     Running, doing random reads.
	      W	     Running, doing sequential writes.
	      w	     Running, doing random writes.
	      M	     Running, doing mixed sequential reads/writes.
	      m	     Running, doing mixed random reads/writes.
	      F	     Running, currently	waiting	for fsync(2).
	      V	     Running, verifying	written	data.
	      E	     Exited, not reaped	by main	thread.
	      -	     Exited, thread reaped.

       The second set of brackets shows	the estimated completion percentage of
       the current group.  The third set shows the read	and  write  I/O	 rate,
       respectively. Finally, the estimated run	time of	the job	is displayed.

       When fio	completes (or is interrupted by	Ctrl-C), it will show data for
       each thread, each group of threads, and each disk, in that order.

       Per-thread statistics first show	the threads client  number,  group-id,
       and error code.	The remaining figures are as follows:

	      io     Number of megabytes of I/O	performed.

	      bw     Average data rate (bandwidth).

	      runt   Threads run time.

	      slat   Submission	latency	minimum, maximum, average and standard
		     deviation.	This is	the time it took to submit the I/O.

	      clat   Completion	latency	minimum, maximum, average and standard
		     deviation.	  This is the time between submission and com-
		     pletion.

	      bw     Bandwidth minimum,	maximum, percentage of aggregate band-
		     width received, average and standard deviation.

	      cpu    CPU usage statistics. Includes user and system time, num-
		     ber of context switches this thread went through and num-
		     ber  of  major and	minor page faults. The CPU utilization
		     numbers are averages  for	the  jobs  in  that  reporting
		     group, while the context and fault	counters are summed.

	      IO depths
		     Distribution  of  I/O depths.  Each depth includes	every-
		     thing less	than (or equal)	to it, but  greater  than  the
		     previous depth.

	      IO issued
		     Number of read/write requests issued, and number of short
		     read/write	requests.

	      IO latencies
		     Distribution of I/O completion  latencies.	  The  numbers
		     follow the	same pattern as	IO depths.

       The group statistics show:
	      io     Number of megabytes I/O performed.
	      aggrb  Aggregate bandwidth of threads in the group.
	      minb   Minimum average bandwidth a thread	saw.
	      maxb   Maximum average bandwidth a thread	saw.
	      mint   Shortest runtime of threads in the	group.
	      maxt   Longest runtime of	threads	in the group.

       Finally,	disk statistics	are printed with reads first:
	      ios    Number of I/Os performed by all groups.
	      merge  Number of merges in the I/O scheduler.
	      ticks  Number of ticks we	kept the disk busy.
	      io_queue
		     Total time	spent in the disk queue.
	      util   Disk utilization.

       It  is  also possible to	get fio	to dump	the current output while it is
       running,	without	terminating the	job. To	do that,  send	fio  the  USR1
       signal.

TERSE OUTPUT
       If  the	--minimal / --append-terse options are given, the results will
       be  printed/appended  in	 a  semicolon-delimited	 format	 suitable  for
       scripted	 use.	A job description (if provided)	follows	on a new line.
       Note that the first number in the line is the version  number.  If  the
       output  has  to	be changed for some reason, this number	will be	incre-
       mented by 1 to signify that change.  The	fields are:

	      terse version, fio version, jobname, groupid, error

	      Read status:
		     Total I/O (KiB), bandwidth	(KiB/s), IOPS, runtime (ms)

		     Submission	latency:
			    min, max, mean, standard deviation
		     Completion	latency:
			    min, max, mean, standard deviation
		     Completion	latency	percentiles (20	fields):
			    Xth	percentile=usec
		     Total latency:
			    min, max, mean, standard deviation
		     Bandwidth:
			    min, max, aggregate	 percentage  of	 total,	 mean,
			    standard deviation

	      Write status:
		     Total I/O (KiB), bandwidth	(KiB/s), IOPS, runtime (ms)

		     Submission	latency:
			    min, max, mean, standard deviation
		     Completion	latency:
			    min, max, mean, standard deviation
		     Completion	latency	percentiles (20	fields):
			    Xth	percentile=usec
		     Total latency:
			    min, max, mean, standard deviation
		     Bandwidth:
			    min,  max,	aggregate  percentage  of total, mean,
			    standard deviation

	      CPU usage:
		     user, system, context switches, major page	faults,	 minor
		     page faults

	      IO depth distribution:
		     <=1, 2, 4,	8, 16, 32, >=64

	      IO latency distribution:
		     Microseconds:
			    <=2, 4, 10,	20, 50,	100, 250, 500, 750, 1000
		     Milliseconds:
			    <=2,  4,  10,  20,	50,  100, 250, 500, 750, 1000,
			    2000, >=2000

	      Disk utilization (1 for each disk	used):
		     name, read	ios, write ios,	 read  merges,	write  merges,
		     read  ticks,  write  ticks, read in-queue time, write in-
		     queue time, disk utilization percentage

	      Error Info (dependent on continue_on_error, default off):
		     total # errors, first error code

	      text description (if provided in config -	appears	on newline)

       Below is	a single line containing short names for each of the fields in
       the minimal output v3, separated	by semicolons:

	      terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth;read_iops;read_runtime_ms;read_slat_min;read_slat_max;read_slat_mean;read_slat_dev;read_clat_max;read_clat_min;read_clat_mean;read_clat_dev;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min;read_lat_max;read_lat_mean;read_lat_dev;read_bw_min;read_bw_max;read_bw_agg_pct;read_bw_mean;read_bw_dev;write_kb;write_bandwidth;write_iops;write_runtime_ms;write_slat_min;write_slat_max;write_slat_mean;write_slat_dev;write_clat_max;write_clat_min;write_clat_mean;write_clat_dev;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min;write_lat_max;write_lat_mean;write_lat_dev;write_bw_min;write_bw_max;write_bw_agg_pct;write_bw_mean;write_bw_dev;cpu_user;cpu_sys;cpu_csw;cpu_mjf;pu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util

TRACE FILE FORMAT
       There  are two trace file format	that you can encounter.	The older (v1)
       format is unsupported since version  1.20-rc3  (March  2008).  It  will
       still  be described below in case that you get an old trace and want to
       understand it.

       In any case the trace is	a simple text file with	a  single  action  per
       line.

       Trace file format v1
	      Each line	represents a single io action in the following format:

	      rw, offset, length

	      where  rw=0/1  for read/write, and the offset and	length entries
	      being in bytes.

	      This format is not supported in Fio versions => 1.20-rc3.

       Trace file format v2
	      The second version of the	trace file format  was	added  in  Fio
	      version  1.17.   It  allows one to access	more then one file per
	      trace and	has a bigger set of possible file actions.

	      The first	line of	the trace file has to be:

	      fio version 2 iolog

	      Following	this can be lines in two different formats, which  are
	      described	below.	The file management format:

	      filename action

	      The filename is given as an absolute path. The action can	be one
	      of these:

		     add    Add	the given filename to the trace
		     open   Open the file with the given filename.  The	 file-
			    name  has  to  have	been previously	added with the
			    add	action.
		     close  Close the file with	the given filename.  The  file
			    must have previously been opened.

	      The file io action format:

	      filename action offset length

	      The  filename is given as	an absolute path, and has to have been
	      added and	opened before it can be	used  with  this  format.  The
	      offset  and  length are given in bytes. The action can be	one of
	      these:

		     wait   Wait for 'offset' microseconds.  Everything	 below
			    100	 is  discarded.	  The  time is relative	to the
			    previous wait statement.
		     read   Read length	bytes beginning	from offset
		     write  Write length bytes beginning from offset
		     sync   fsync() the	file
		     datasync
			    fdatasync()	the file
		     trim   trim the given file	 from  the  given  offset  for
			    length bytes

CPU IDLENESS PROFILING
       In  some	cases, we want to understand CPU overhead in a test. For exam-
       ple, we test patches for	the specific goodness of whether  they	reduce
       CPU  usage.   fio  implements a balloon approach	to create a thread per
       CPU that	runs at	idle priority, meaning that it only runs  when	nobody
       else  needs  the	cpu.  By measuring the amount of work completed	by the
       thread, idleness	of each	CPU can	be derived accordingly.

       An unit work is defined as touching a full page of unsigned characters.
       Mean  and  standard  deviation  of time to complete an unit work	is re-
       ported in "unit work" section. Options can be chosen to report detailed
       percpu idleness or overall system idleness by aggregating percpu	stats.

VERIFICATION AND TRIGGERS
       Fio  is usually run in one of two ways, when data verification is done.
       The first is a normal write job of some sort with verify	enabled.  When
       the  write  phase has completed,	fio switches to	reads and verifies ev-
       erything	it wrote. The second model is running just  the	 write	phase,
       and  then  later	 on  running  the  same	job (but with reads instead of
       writes) to repeat the same IO patterns and verify the contents. Both of
       these  methods depend on	the write phase	being completed, as fio	other-
       wise has	no idea	how much data was written.

       With verification triggers, fio	supports  dumping  the	current	 write
       state  to  local	files. Then a subsequent read verify workload can load
       this state and know exactly where to stop. This is useful  for  testing
       cases  where  power  is	cut  to	a server in a managed fashion, for in-
       stance.

       A verification trigger consists of two things:

	      Storing the write	state of each job

	      Executing	a trigger command

       The write state is relatively small, on the order of hundreds of	 bytes
       to  single  kilobytes. It contains information on the number of comple-
       tions done, the last X completions, etc.

       A trigger is invoked either through creation  (touch)  of  a  specified
       file  in	 the  system, or through a timeout setting. If fio is run with
       --trigger-file=/tmp/trigger-file, then it will  continually  check  for
       the  existence  of  /tmp/trigger-file.  When it sees this file, it will
       fire off	the trigger (thus saving state,	and executing the trigger com-
       mand).

       For client/server runs, there's both a local and	remote trigger.	If fio
       is running as a server backend, it will send the	job states back	to the
       client for safe storage,	then execute the remote	trigger, if specified.
       If a local trigger is specified,	the server will	still  send  back  the
       write state, but	the client will	then execute the trigger.

       Verification trigger example

	      Lets  say	 we  want to run a powercut test on the	remote machine
	      'server'.	 Our write workload is in write-test.fio. We  want  to
	      cut  power  to  'server' at some point during the	run, and we'll
	      run this test from the safety or our local machine,  'localbox'.
	      On the server, we'll start the fio backend normally:

	      server# fio --server

	      and on the client, we'll fire off	the workload:

	      localbox$	  fio  --client=server	--trigger-file=/tmp/my-trigger
	      --trigger-remote="bash -c	"echo b	> /proc/sysrq-triger""

	      We set /tmp/my-trigger as	the trigger file, and we tell  fio  to
	      execute

	      echo b > /proc/sysrq-trigger

	      on  the  server once it has received the trigger and sent	us the
	      write state. This	will work, but it's not	really	cutting	 power
	      to  the  server, it's merely abruptly rebooting it. If we	have a
	      remote way of cutting power to the server	through	IPMI or	 simi-
	      lar,  we	could do that through a	local trigger command instead.
	      Lets assume we have a script that	does IPMI reboot  of  a	 given
	      hostname,	 ipmi-reboot.  On localbox, we could then have run fio
	      with a local trigger instead:

	      localbox$	 fio  --client=server	--trigger-file=/tmp/my-trigger
	      --trigger="ipmi-reboot server"

	      For  this	 case,	fio  would  wait for the server	to send	us the
	      write state, then	execute	'ipmi-reboot server'  when  that  hap-
	      pened.

       Loading verify state
	      To  load store write state, read verification job	file must con-
	      tain the verify_state_load option. If that is set, fio will load
	      the previously stored state. For a local fio run this is done by
	      loading the files	directly, and  on  a  client/server  run,  the
	      server  backend  will  ask the client to send the	files over and
	      load them	from there.

LOG FILE FORMATS
       Fio supports a variety of log  file  formats,  for  logging  latencies,
       bandwidth,  and	IOPS. The logs share a common format, which looks like
       this:

       time (msec), value, data	direction, offset

       Time for	the log	entry is always	in milliseconds. The value logged  de-
       pends on	the type of log, it will be one	of the following:

       Latency log
	      Value is in latency in usecs
       Bandwidth log
	      Value is in KiB/sec
       IOPS log
	      Value is in IOPS

       Data direction is one of	the following:

       0      IO is a READ
       1      IO is a WRITE
       2      IO is a TRIM

       The  offset  is	the  offset, in	bytes, from the	start of the file, for
       that particular IO. The logging of  the	offset	can  be	 toggled  with
       log_offset.

       If  windowed  logging is	enabled	through	log_avg_msec, then fio doesn't
       log individual IOs. Instead of logs the average values over the	speci-
       fied period of time. Since data direction and offset are	per-IO values,
       they aren't applicable if windowed logging is enabled. If windowed log-
       ging  is	enabled	and log_max_value is set, then fio logs	maximum	values
       in that window instead of averages.

       For histogram logging the logs look like	this:

       time (msec), data direction, block-size,	bin 0, bin 1, ..., bin 1215

       Where 'bin i' gives the frequency of IO requests	with a latency falling
       in the i-th bin.	See log_hist_coarseness	for logging fewer bins.

CLIENT / SERVER
       Normally	 you would run fio as a	stand-alone application	on the machine
       where the IO workload should be generated. However, it is also possible
       to run the frontend and backend of fio separately. This makes it	possi-
       ble to have a fio server	running	on the machine(s) where	the  IO	 work-
       load should be running, while controlling it from another machine.

       To start	the server, you	would do:

       fio --server=args

       on  that	machine, where args defines what fio listens to. The arguments
       are of the form 'type:hostname or IP:port'. 'type' is either  'ip'  (or
       ip4) for	TCP/IP v4, 'ip6' for TCP/IP v6,	or 'sock' for a	local unix do-
       main socket. 'hostname' is either a hostname or IP address, and	'port'
       is  the	port to	listen to (only	valid for TCP/IP, not a	local socket).
       Some examples:

       1) fio --server

	  Start	a fio server, listening	on all interfaces on the default  port
       (8765).

       2) fio --server=ip:hostname,4444

	  Start	 a  fio	 server,  listening on IP belonging to hostname	and on
       port 4444.

       3) fio --server=ip6:::1,4444

	  Start	a fio server, listening	on IPv6	 localhost  ::1	 and  on  port
       4444.

       4) fio --server=,4444

	  Start	a fio server, listening	on all interfaces on port 4444.

       5) fio --server=1.2.3.4

	  Start	a fio server, listening	on IP 1.2.3.4 on the default port.

       6) fio --server=sock:/tmp/fio.sock

	  Start	a fio server, listening	on the local socket /tmp/fio.sock.

       When  a	server	is  running,  you can connect to it from a client. The
       client is run with:

       fio --local-args	--client=server	--remote-args <job file(s)>

       where --local-args are arguments	that are local to the client where  it
       is  running, 'server' is	the connect string, and	--remote-args and <job
       file(s)>	are sent to the	server.	The 'server' string follows  the  same
       format  as  it does on the server side, to allow	IP/hostname/socket and
       port strings.  You can connect to multiple clients as well, to do  that
       you could run:

       fio --client=server2 --client=server2 <job file(s)>

       If  the	job  file  is located on the fio server, then you can tell the
       server to load a	local file as well. This is done  by  using  --remote-
       config:

       fio --client=server --remote-config /path/to/file.fio

       Then fio	will open this local (to the server) job file instead of being
       passed one from the client.

       If you have many	servers	(example: 100 VMs/containers), you can input a
       pathname	of a file containing host IPs/names as the parameter value for
       the --client option.  For example, here is an example "host.list"  file
       containing 2 hostnames:

       host1.your.dns.domain
       host2.your.dns.domain

       The fio command would then be:

       fio --client=host.list <job file>

       In this mode, you cannot	input server-specific parameters or job	files,
       and all servers receive the same	job file.

       In order	to enable fio --client runs utilizing a	shared filesystem from
       multiple	 hosts,	fio --client now prepends the IP address of the	server
       to the filename.	For example, if	fio is	using  directory  /mnt/nfs/fio
       and is writing filename fileio.tmp, with	a --client hostfile containing
       two  hostnames  h1  and	h2  with  IP  addresses	  192.168.10.120   and
       192.168.10.121, then fio	will create two	files:

       /mnt/nfs/fio/192.168.10.120.fileio.tmp
       /mnt/nfs/fio/192.168.10.121.fileio.tmp

AUTHORS
       fio  was	 written by Jens Axboe <jens.axboe@oracle.com>,	now Jens Axboe
       <axboe@fb.com>.
       This man	page was written  by  Aaron  Carroll  <aaronc@cse.unsw.edu.au>
       based on	documentation by Jens Axboe.

REPORTING BUGS
       Report bugs to the fio mailing list <fio@vger.kernel.org>.  See README.

SEE ALSO
       For further documentation see HOWTO and README.
       Sample jobfiles are available in	the examples directory.
       These are typically located under /usr/share/doc/fio.

       HOWTO:  http://git.kernel.dk/cgit/fio/plain/HOWTO
       README: http://git.kernel.dk/cgit/fio/plain/README

User Manual			   May 2017				fio(1)

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | JOB FILE FORMAT | JOB PARAMETERS | OUTPUT | TERSE OUTPUT | TRACE FILE FORMAT | CPU IDLENESS PROFILING | VERIFICATION AND TRIGGERS | LOG FILE FORMATS | CLIENT / SERVER | AUTHORS | REPORTING BUGS | SEE ALSO

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=fio&manpath=FreeBSD+12.0-RELEASE+and+Ports>

home | help