Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
fio(1)			    General Commands Manual			fio(1)

       fio - flexible I/O tester

       fio [options] [jobfile]...

       fio  is a tool that will	spawn a	number of threads or processes doing a
       particular type of I/O action as	specified by the  user.	  The  typical
       use  of	fio  is	to write a job file matching the I/O load one wants to

	      Enable verbose tracing type of various fio actions. May be `all'
	      for  all	types  or  individual types separated by a comma (e.g.
	      `--debug=file,mem'  will	enable	file  and  memory  debugging).
	      `help' will list all available tracing options.

	      Parse options only, don't	start any I/O.

	      Merge blktraces only, don't start	any I/O.

	      Write output to filename.

	      Set  the	reporting  format  to  `normal',  `terse',  `json', or
	      `json+'. Multiple	formats	can be selected, separate by a	comma.
	      `terse' is a CSV based format. `json+' is	like `json', except it
	      adds a full dump of the latency buckets.

	      Generate aggregate bandwidth logs.

	      Print statistics in a terse, semicolon-delimited format.

	      Print statistics in selected mode	AND terse, semicolon-delimited
	      format.	Deprecated, use	--output-format	instead	to select mul-
	      tiple formats.

	      Set terse	version	output format (default `3', or `2', `4', `5').

	      Print version information	and exit.

       --help Print a summary of the command line options and exit.

	      Perform test and validation of internal CPU clock.

	      Test the speed of	the built-in checksumming functions. If	no ar-
	      gument  is given,	all of them are	tested.	Alternatively, a comma
	      separated	list can be passed, in which case the given  ones  are

	      Print  help  information	for command. May be `all' for all com-

	      List all commands	defined	by ioengine, or	print help for command
	      defined by ioengine. If no ioengine is given, list all available

	      Convert jobfile to a set of command-line options.

	      Turn on safety read-only checks, preventing  writes  and	trims.
	      The  --readonly option is	an extra safety	guard to prevent users
	      from accidentally	starting a write or trim workload when that is
	      not  desired.  Fio  will	only  modify  the device under test if
	      `rw=write/randwrite/rw/randrw/trim/randtrim/trimwrite' is	given.
	      This safety net can be used as an	extra precaution.

	      Specifies	 when  real-time  ETA estimate should be printed. when
	      may be `always', `never' or `auto'. `auto' is  the  default,  it
	      prints  ETA when requested if the	output is a TTY. `always' dis-
	      regards the output type, and prints ETA when requested.  `never'
	      never prints ETA.

	      By default, fio requests client ETA status roughly every second.
	      With this	option,	the interval is	configurable.  Fio  imposes  a
	      minimum  allowed	time  to avoid flooding	the console, less than
	      250 msec is not supported.

	      Force a new line for every time period passed. When the unit  is
	      omitted, the value is interpreted	in seconds.

	      Force  a	full status dump of cumulative (from job start)	values
	      at time intervals. This option  does  *not*  provide  per-period
	      measurements.  So	values such as bandwidth are running averages.
	      When the time unit is omitted, time is interpreted  in  seconds.
	      Note  that  using	 this  option with `--output-format=json' will
	      yield output that	technically isn't valid	json, since the	output
	      will  be	collated  sets of valid	json. It will need to be split
	      into valid sets of json after the	run.

	      Only run specified section name in job file.  Multiple  sections
	      can  be  specified.   The	--section option allows	one to combine
	      related jobs into	one file.  E.g.	 one  job  file	 could	define
	      light,  moderate,	 and  heavy sections. Tell fio to run only the
	      "heavy" section by giving	`--section=heavy' command line option.
	      One  can	also specify the "write" operations in one section and
	      "verify" operation in another section. The --section option only
	      applies to job sections. The reserved *global* section is	always
	      parsed and used.

	      Allocate additional internal smalloc pools of size  kb  in  KiB.
	      The  --alloc-size	 option	 increases shared memory set aside for
	      use by fio.  If running large jobs with randommap	 enabled,  fio
	      can  run	out  of	 memory.  Smalloc is an	internal allocator for
	      shared structures	from a fixed size memory pool and can grow  to
	      16  pools. The pool size defaults	to 16MiB.  NOTE: While running
	      `.fio_smalloc.*' backing store files are visible in `/tmp'.

	      All fio parser warnings are fatal, causing fio to	exit  with  an

	      Set  the	maximum	 number	of threads/processes to	support	to nr.
	      NOTE: On Linux, it may be	necessary to increase the  shared-mem-
	      ory  limit  (`/proc/sys/kernel/shmmax')  if fio runs into	errors
	      while creating jobs.

	      Start a backend server, with args	specifying what	to listen  to.
	      See CLIENT/SERVER	section.

	      Background  a  fio  server, writing the pid to the given pidfile

	      Instead of running the jobs locally, send	and run	 them  on  the
	      given hostname or	set of hostnames. See CLIENT/SERVER section.

	      Tell fio server to load this local file.

	      Report CPU idleness. option is one of the	following:

			    Run	unit work calibration only and exit.

		     system Show aggregate system idleness and unit work.

		     percpu As system but also show per	CPU idleness.

	      Inflate and output compressed log.

	      Execute trigger command when file	exists.

	      Execute trigger at this time.

	      Set this command as local	trigger.

	      Set this command as remote trigger.

	      Use  the	directory  specified by	path for generated state files
	      instead of the current working directory.

       Any parameters following	the options will be assumed to be  job	files,
       unless  they  match  a  job  file  parameter. Multiple job files	can be
       listed and each job file	will be	regarded as a separate group. Fio will
       stonewall execution between each	group.

       Fio accepts one or more job files describing what it is supposed	to do.
       The job file format is the classic ini file, where the  names  enclosed
       in  [] brackets define the job name. You	are free to use	any ASCII name
       you want, except	*global* which has special meaning. Following the  job
       name  is	 a sequence of zero or more parameters,	one per	line, that de-
       fine the	behavior of the	job. If	the first character in a line is a ';'
       or a '#', the entire line is discarded as a comment.

       A *global* section sets defaults	for the	jobs described in that file. A
       job may override	a *global* section parameter, and a job	file may  even
       have several *global* sections if so desired. A job is only affected by
       a *global* section residing above it.

       The --cmdhelp option also lists all options. If used  with  an  command
       argument, --cmdhelp will	detail the given command.

       See  the	 `examples/'  directory	 for  inspiration  on how to write job
       files. Note the copyright and license requirements currently  apply  to
       `examples/' files.

       Note that the maximum length of a line in the job file is 8192 bytes.

       Some parameters take an option of a given type, such as an integer or a
       string. Anywhere	a numeric value	is required, an	arithmetic  expression
       may be used, provided it	is surrounded by parentheses. Supported	opera-
       tors are:

	      addition (+)

	      subtraction (-)

	      multiplication (*)

	      division (/)

	      modulus (%)

	      exponentiation (^)

       For time	values in expressions, units are microseconds by default. This
       is  different  than for time values not in expressions (not enclosed in

       The following parameter types are used.

       str    String. A	sequence of alphanumeric characters.

       time   Integer with possible time suffix. Without a unit	value  is  in-
	      terpreted	 as seconds unless otherwise specified.	Accepts	a suf-
	      fix of 'd' for days, 'h' for hours, 'm'  for  minutes,  's'  for
	      seconds,	'ms' (or 'msec') for milliseconds and 'us' (or 'usec')
	      for microseconds.	For example, use 10m for 10 minutes.

       int    Integer. A whole number value, which may contain an integer pre-
	      fix and an integer suffix.

		     [*integer prefix*]	**number** [*integer suffix*]

	      The  optional  *integer prefix* specifies	the number's base. The
	      default is decimal. *0x* specifies hexadecimal.

	      The optional *integer suffix* specifies the number's units,  and
	      includes an optional unit	prefix and an optional unit. For quan-
	      tities of	data, the default unit is  bytes.  For	quantities  of
	      time, the	default	unit is	seconds	unless otherwise specified.

	      With  `kb_base=1000',  fio  follows  international standards for
	      unit prefixes. To	specify	power-of-10 decimal values defined  in
	      the International	System of Units	(SI):

		     K means kilo (K) or 1000
		     M means mega (M) or 1000**2
		     G means giga (G) or 1000**3
		     T means tera (T) or 1000**4
		     P means peta (P) or 1000**5

	      To specify power-of-2 binary values defined in IEC 80000-13:

		     Ki	means kibi (Ki)	or 1024
		     Mi	means mebi (Mi)	or 1024**2
		     Gi	means gibi (Gi)	or 1024**3
		     Ti	means tebi (Ti)	or 1024**4
		     Pi	means pebi (Pi)	or 1024**5

	      With  `kb_base=1024'  (the default), the unit prefixes are oppo-
	      site from	those specified	in the SI and IEC  80000-13  standards
	      to provide compatibility with old	scripts. For example, 4k means

	      For quantities of	data, an optional unit of 'B' may be  included
	      (e.g., 'kB' is the same as 'k').

	      The  *integer  suffix*  is  not  case sensitive (e.g., m/mi mean
	      mebi/mega, not milli). 'b' and 'B' both mean byte, not bit.

	      Examples with `kb_base=1000':

		     4 KiB: 4096, 4096b, 4096B,	4k, 4kb, 4kB, 4K, 4KB
		     1 MiB: 1048576, 1m, 1024k
		     1 MB: 1000000, 1mi, 1000ki
		     1 TiB: 1073741824,	1t, 1024m, 1048576k
		     1 TB: 1000000000, 1ti, 1000mi, 1000000ki

	      Examples with `kb_base=1024' (default):

		     4 KiB: 4096, 4096b, 4096B,	4k, 4kb, 4kB, 4K, 4KB
		     1 MiB: 1048576, 1m, 1024k
		     1 MB: 1000000, 1mi, 1000ki
		     1 TiB: 1073741824,	1t, 1024m, 1048576k
		     1 TB: 1000000000, 1ti, 1000mi, 1000000ki

	      To specify times (units are not case sensitive):

		     D means days
		     H means hours
		     M mean minutes
		     s or sec means seconds (default)
		     ms	or msec	means milliseconds
		     us	or usec	means microseconds

	      If the option accepts an upper and lower range, use a colon  ':'
	      or minus '-' to separate such values. See	irange parameter type.
	      If the lower value specified happens to be larger	than the upper
	      value the	two values are swapped.

       bool   Boolean.	Usually	parsed as an integer, however only defined for
	      true and false (1	and 0).

       irange Integer range with suffix. Allows	value range to be given,  such
	      as  1024-4096.  A	 colon may also	be used	as the separator, e.g.
	      1k:4k. If	the option allows two sets  of	ranges,	 they  can  be
	      specified	 with  a  ',' or '/' delimiter:	1k-4k/8k-32k. Also see
	      int parameter type.

	      A	list of	floating point numbers,	separated by a ':' character.

       With the	above in mind, here follows the	complete list of fio  job  pa-

	      Select the interpretation	of unit	prefixes in input parameters.

		     1000   Inputs  comply  with IEC 80000-13 and the Interna-
			    tional System of Units (SI). Use:

			    - power-of-2 values	with IEC prefixes (e.g., KiB)
			    - power-of-10 values with SI prefixes (e.g., kB)

		     1024   Compatibility mode (default).  To  avoid  breaking
			    old	scripts:

			    - power-of-2 values	with SI	prefixes
			    - power-of-10 values with IEC prefixes

	      See bs for more details on input parameters.

	      Outputs  always  use correct prefixes. Most outputs include both
	      side-by-side, like:

		     bw=2383.3kB/s (2327.4KiB/s)

	      If only one value	is reported, then kb_base selects the  one  to

		     1000 -- SI	prefixes
		     1024 -- IEC prefixes

	      Base unit	for reporting. Allowed values are:

		     0	    Use	auto-detection (default).

		     8	    Byte based.

		     1	    Bit	based.

   Job description
	      ASCII  name  of  the  job. This may be used to override the name
	      printed by fio for this job. Otherwise the job name is used.  On
	      the  command line	this parameter has the special purpose of also
	      signaling	the start of a new job.

	      Text description of the job. Doesn't  do	anything  except  dump
	      this text	description when this job is run. It's not parsed.

	      Run  the specified number	of iterations of this job. Used	to re-
	      peat the same workload a given number of times. Defaults to 1.

	      Create the specified number of clones of this job. Each clone of
	      job  is spawned as an independent	thread or process. May be used
	      to setup a larger	number of  threads/processes  doing  the  same
	      thing. Each thread is reported separately; to see	statistics for
	      all clones as a whole, use group_reporting in  conjunction  with
	      new_group.  See --max-jobs. Default: 1.

   Time	related	parameters
	      Tell  fio	 to terminate processing after the specified period of
	      time. It can be quite hard to determine for how long a specified
	      job  will	 run, so this parameter	is handy to cap	the total run-
	      time to a	given time. When the unit is omitted, the value	is in-
	      terpreted	in seconds.

	      If  set,	fio will run for the duration of the runtime specified
	      even if the file(s) are completely read or written. It will sim-
	      ply loop over the	same workload as many times as the runtime al-

	      Delay the	start of job for the specified amount of time. Can  be
	      a	 single	 value	or a range. When given as a range, each	thread
	      will choose a value randomly from	within the range. Value	is  in
	      seconds if a unit	is omitted.

	      If  set,	fio will run the specified workload for	this amount of
	      time before logging any performance numbers. Useful for  letting
	      performance  settle  before logging results, thus	minimizing the
	      runtime required for stable results. Note	that the ramp_time  is
	      considered lead in time for a job, thus it will increase the to-
	      tal runtime if a special timeout or runtime is  specified.  When
	      the unit is omitted, the value is	given in seconds.

	      Use  the	given clocksource as the base of timing. The supported
	      options are:



		     cpu    Internal CPU clock source

	      cpu is the preferred clocksource if it is	 reliable,  as	it  is
	      very  fast  (and fio is heavy on time calls). Fio	will automati-
	      cally use	this clocksource if it's supported and considered  re-
	      liable  on  the  system  it is running on, unless	another	clock-
	      source is	specifically set. For x86/x86-64 CPUs, this means sup-
	      porting TSC Invariant.

	      Enable   all  of	the  gettimeofday(2)  reducing	options	 (dis-
	      able_clat,  disable_slat,	 disable_bw_measurement)  plus	reduce
	      precision	 of the	timeout	somewhat to really shrink the gettime-
	      ofday(2) call count. With	this option enabled, we	only do	 about
	      0.4% of the gettimeofday(2) calls	we would have done if all time
	      keeping was enabled.

	      Sometimes	it's cheaper to	dedicate a single thread of  execution
	      to  just	getting	 the current time. Fio (and databases, for in-
	      stance) are very intensive on gettimeofday(2) calls.  With  this
	      option,  you can set one CPU aside for doing nothing but logging
	      current time  to	a  shared  memory  location.  Then  the	 other
	      threads/processes	 that  run  I/O	 workloads need	only copy that
	      segment, instead of entering the kernel with  a  gettimeofday(2)
	      call.  The  CPU set aside	for doing these	time calls will	be ex-
	      cluded from other	uses. Fio will manually	clear it from the  CPU
	      mask of other jobs.

   Target file/device
	      Prefix  filenames	 with this directory. Used to place files in a
	      different	location than `./'. You	can specify a number of	direc-
	      tories  by  separating the names with a ':' character. These di-
	      rectories	will be	assigned equally  distributed  to  job	clones
	      created  by  numjobs  as	long as	they are using generated file-
	      names. If	specific filename(s) are set fio will  use  the	 first
	      listed  directory,  and  thereby	matching the filename semantic
	      (which generates a file for each clone  if  not  specified,  but
	      lets all clones use the same file	if set).

	      See  the	filename  option  for information on how to escape ':'
	      characters within	the directory path itself.

	      Note: To control the directory fio will use for  internal	 state
	      files use	--aux-path.

	      Fio  normally  makes up a	filename based on the job name,	thread
	      number, and file number (see filename_format). If	 you  want  to
	      share  files between threads in a	job or several jobs with fixed
	      file paths, specify a filename for each of them to override  the
	      default. If the ioengine is file based, you can specify a	number
	      of files by separating the names with a ':'  colon.  So  if  you
	      wanted  a	job to open `/dev/sda' and `/dev/sdb' as the two work-
	      ing files, you would use `filename=/dev/sda:/dev/sdb'. This also
	      means  that  whenever  this  option is specified,	nrfiles	is ig-
	      nored. The size of regular files specified by this  option  will
	      be  size	divided	 by number of files unless an explicit size is
	      specified	by filesize.

	      Each colon in the	wanted path must be escaped with a '\' charac-
	      ter.  For	instance, if the path is `/dev/dsk/foo@3,0:c' then you
	      would use	`filename=/dev/dsk/foo@3,0\:c'	and  if	 the  path  is
	      `F:\filename' then you would use `filename=F\:\filename'.

	      On  Windows,  disk  devices are accessed as `\\.\PhysicalDrive0'
	      for the first device, `\\.\PhysicalDrive1' for the  second  etc.
	      Note:  Windows  and FreeBSD prevent write	access to areas	of the
	      disk containing in-use data (e.g.	filesystems).

	      The filename `-' is a reserved name, meaning  *stdin*  or	 *std-
	      out*. Which of the two depends on	the read/write direction set.

	      If  sharing multiple files between jobs, it is usually necessary
	      to have fio generate the exact names that	you want. By  default,
	      fio will name a file based on the	default	file format specifica-
	      tion of `jobname.jobnumber.filenumber'. With this	 option,  that
	      can  be customized. Fio will recognize and replace the following
	      keywords in this string:

			    The	name of	the worker thread or process.

			    The	incremental number of  the  worker  thread  or

			    The	incremental number of the file for that	worker
			    thread or process.

	      To have dependent	jobs share a set of files, this	option can  be
	      set  to  have fio	generate filenames that	are shared between the
	      two. For instance, if `testfiles.$filenum'  is  specified,  file
	      number 4 for any job will	be named `testfiles.4'.	The default of
	      `$jobname.$jobnum.$filenum' will be  used	 if  no	 other	format
	      specifier	is given.

	      If you specify a path then the directories will be created up to
	      the main directory for the file.	So for example if you  specify
	      `a/b/c/$jobnum`  then  the directories a/b/c will	be created be-
	      fore the file setup part of the job.  If you  specify  directory
	      then  the	 path will be relative that directory, otherwise it is
	      treated as the absolute path.

	      To avoid collisions between networked clients, fio  defaults  to
	      prefixing	 any  generated	filenames (with	a directory specified)
	      with the source of the client connecting.	To disable this	behav-
	      ior, set this option to 0.

	      Recursively open any files below directory str.

	      Fio  defaults  to	 not  locking  any files before	it does	I/O to
	      them. If a file or file descriptor is shared, fio	can  serialize
	      I/O  to  that  file  to  make the	end result consistent. This is
	      usual for	emulating real workloads that share  files.  The  lock
	      modes are:

		     none   No locking.	The default.

			    Only  one  thread or process may do	I/O at a time,
			    excluding all others.

			    Read-write locking on the file. Many  readers  may
			    access  the	 file at the same time,	but writes get
			    exclusive access.

	      Number of	files to use for this job. Defaults to 1. The size  of
	      files will be size divided by this unless	explicit size is spec-
	      ified by filesize. Files are created for each thread separately,
	      and  each	 file  will  have a file number	within its name	by de-
	      fault, as	explained in filename section.

	      Number of	files to keep open at the same time. Defaults  to  the
	      same as nrfiles, can be set smaller to limit the number simulta-
	      neous opens.

	      Defines how fio decides which file from a	job to	service	 next.
	      The following types are defined:

		     random Choose a file at random.

			    Round  robin  over	opened	files. This is the de-

			    Finish one file before moving on to	the next. Mul-
			    tiple  files  can still be open depending on open-

		     zipf   Use	a Zipf distribution to decide what file	to ac-

		     pareto Use	 a  Pareto distribution	to decide what file to

		     normal Use	a Gaussian  (normal)  distribution  to	decide
			    what file to access.

		     gauss  Alias for normal.

	      For  random,  roundrobin,	 and  sequential, a postfix can	be ap-
	      pended to	tell fio how many I/Os to issue	before switching to  a
	      new  file.  For example, specifying `file_service_type=random:8'
	      would cause fio to issue 8 I/Os before selecting a new  file  at
	      random.  For  the	 non-uniform  distributions,  a	floating point
	      postfix can be  given  to	 influence  how	 the  distribution  is
	      skewed.  See  random_distribution	 for a description of how that
	      would work.

	      Attempt to switch	the device hosting the file to	the  specified
	      I/O scheduler before running.

	      If  true,	 serialize the file creation for the jobs. This	may be
	      handy to avoid interleaving of data files, which may greatly de-
	      pend on the filesystem used and even the number of processors in
	      the system. Default: true.

	      fsync(2) the data	file after creation. This is the default.

	      If true, don't pre-create	files but allow	the  job's  open()  to
	      create  a	 file when it's	time to	do I/O.	Default: false -- pre-
	      create all necessary files when the job starts.

	      If true, fio will	only run the setup phase of the	job. If	 files
	      need  to	be laid	out or updated on disk,	only that will be done
	      -- the actual job	contents are not executed. Default: false.

	      If true, fio is permitted	to create files	as part	of  its	 work-
	      load.  If	 this  option is false,	then fio will error out	if the
	      files it needs to	use don't already exist. Default: true.

	      If this isn't set, fio will  abort  jobs	that  are  destructive
	      (e.g. that write)	to what	appears	to be a	mounted	device or par-
	      tition. This should help catch creating  inadvertently  destruc-
	      tive tests, not realizing	that the test will destroy data	on the
	      mounted file system. Note	that some platforms don't allow	 writ-
	      ing against a mounted device regardless of this option. Default:

	      If this is given,	files will  be	pre-read  into	memory	before
	      starting	the  given I/O operation. This will also clear the in-
	      validate flag, since it is pointless to pre-read and  then  drop
	      the  cache.  This	 will only work	for I/O	engines	that are seek-
	      able, since they allow you to read the same data multiple	times.
	      Thus it will not work on non-seekable I/O	engines	(e.g. network,
	      splice). Default:	false.

	      Unlink the job files when	done. Not  the	default,  as  repeated
	      runs  of	that job would then waste time recreating the file set
	      again and	again. Default:	false.

	      Unlink job files after each iteration or loop. Default: false.

	      Accepted values are:

		     none   The	zonerange, zonesize zonecapacity and  zoneskip
			    parameters are ignored.

			    I/O	 happens in a single zone until	zonesize bytes
			    have been transferred.  After that number of bytes
			    has	 been  transferred processing of the next zone
			    starts. The	zonecapacity parameter is ignored.

		     zbd    Zoned block	device mode. I/O happens  sequentially
			    in	each  zone,  even  if  random I/O has been se-
			    lected. Random I/O happens across  all  zones  in-
			    stead of being restricted to a single zone.

	      For  zonemode=strided,  this  is	the size of a single zone. See
	      also zonesize and	zoneskip.

	      For zonemode=zbd,	this parameter is ignored.

	      For zonemode=strided, this is the	number of  bytes  to  transfer
	      before  skipping	zoneskip  bytes.  If this parameter is smaller
	      than zonerange then only a fraction of each zone with  zonerange
	      bytes  will  be accessed.	 If this parameter is larger than zon-
	      erange then each zone will be  accessed  multiple	 times	before
	      skipping to the next zone.

	      For  zonemode=zbd,  this	is the size of a single	zone. The zon-
	      erange parameter is ignored in this mode.	For a job accessing  a
	      zoned block device, the specified	zonesize must be 0 or equal to
	      the device zone size. For	a regular block	device	or  file,  the
	      specified	zonesize must be at least 512B.

	      For  zonemode=zbd,  this	defines	the capacity of	a single zone,
	      which is the accessible area starting from the  zone  start  ad-
	      dress.  This  parameter  only applies when using zonemode=zbd in
	      combination with regular block devices.  If not specified	it de-
	      faults  to  the zone size. If the	target device is a zoned block
	      device, the zone capacity	is obtained from the  device  informa-
	      tion and this option is ignored.

	      For zonemode=strided, the	number of bytes	to skip	after zonesize
	      bytes of data have been transferred.

	      For zonemode=zbd,	the zonesize aligned number of bytes  to  skip
	      once  a  zone  is	fully written (write workloads)	or all written
	      data in the zone have been read (read workloads).	This parameter
	      is  valid	 only  for sequential workloads	and ignored for	random
	      workloads. For read workloads, see also read_beyond_wp.

	      This parameter applies to	zonemode=zbd only.

	      Zoned block devices are block devices that consist  of  multiple
	      zones.  Each zone	has a type, e.g. conventional or sequential. A
	      conventional zone	can be written at any offset that is a	multi-
	      ple  of the block	size. Sequential zones must be written sequen-
	      tially. The position at which a write must occur is  called  the
	      write  pointer.  A zoned block device can	be either host managed
	      or host aware. For host managed devices  the  host  must	ensure
	      that writes happen sequentially. Fio recognizes host managed de-
	      vices and	serializes writes to sequential	zones  for  these  de-

	      If  a  read occurs in a sequential zone beyond the write pointer
	      then the zoned block device will complete	the read without read-
	      ing  any	data from the storage medium. Since such reads lead to
	      unrealistically high bandwidth and IOPS numbers fio  only	 reads
	      beyond  the  write pointer if explicitly told to do so. Default:

	      When running a random write test across  an  entire  drive  many
	      more  zones will be open than in a typical application workload.
	      Hence this command line option that allows to limit  the	number
	      of open zones. The number	of open	zones is defined as the	number
	      of zones to which	write commands are issued by all  threads/pro-

	      Limit  on	 the  number of	simultaneously opened zones per	single

	      A	number between zero and	one that indicates the ratio of	 logi-
	      cal  blocks  with	 data to the total number of logical blocks in
	      the test above which zones should	be reset periodically.

	      A	number between zero and	one that indicates how	often  a  zone
	      reset  should be issued if the zone reset	threshold has been ex-
	      ceeded. A	zone reset is  submitted  after	 each  (1  /  zone_re-
	      set_frequency)  write  requests. This and	the previous parameter
	      can be used to simulate garbage collection activity.

   I/O type
	      If value is true,	use non-buffered I/O. This  is	usually	 O_DI-
	      RECT.  Note that OpenBSD and ZFS on Solaris don't	support	direct
	      I/O. On Windows the synchronous ioengines	don't  support	direct
	      I/O. Default: false.

	      If  value	 is  true,  attempt  to	 use atomic direct I/O.	Atomic
	      writes are guaranteed to be stable once acknowledged by the  op-
	      erating system. Only Linux supports O_ATOMIC right now.

	      If  value	is true, use buffered I/O. This	is the opposite	of the
	      direct option. Defaults to true.

       readwrite=str, rw=str
	      Type of I/O pattern. Accepted values are:

		     read   Sequential reads.

		     write  Sequential writes.

		     trim   Sequential trims (Linux  block  devices  and  SCSI
			    character devices only).

			    Random reads.

			    Random writes.

			    Random trims (Linux	block devices and SCSI charac-
			    ter	devices	only).

			    Sequential mixed reads and writes.

		     randrw Random mixed reads and writes.

			    Sequential trim+write sequences.  Blocks  will  be
			    trimmed  first, then the same blocks will be writ-
			    ten	to.

	      Fio defaults to read if the option is  not  specified.  For  the
	      mixed I/O	types, the default is to split them 50/50. For certain
	      types of I/O the result may still	be skewed  a  bit,  since  the
	      speed may	be different.

	      It  is  possible to specify the number of	I/Os to	do before get-
	      ting a new offset	by appending `:<nr>' to	the end	of the	string
	      given. For a random read,	it would look like `rw=randread:8' for
	      passing in an offset modifier with a value of 8. If  the	suffix
	      is  used	with  a	 sequential I/O	pattern, then the `<nr>' value
	      specified	will be	added to the generated	offset	for  each  I/O
	      turning  sequential I/O into sequential I/O with holes.  For in-
	      stance, using `rw=write:4k' will skip 4k for every  write.  Also
	      see the rw_sequencer option.

	      If  an  offset  modifier	is  given by appending a number	to the
	      `rw=str' line, then this option controls how that	 number	 modi-
	      fies the I/O offset being	generated. Accepted values are:

			    Generate sequential	offset.

			    Generate the same offset.

	      sequential  is  only useful for random I/O, where	fio would nor-
	      mally generate a new random offset for every I/O.	If you	append
	      e.g.  8 to randread, you would get a new random offset for every
	      8	I/Os. The result would be a seek for only every	 8  I/Os,  in-
	      stead  of	for every I/O. Use `rw=randread:8' to specify that. As
	      sequential I/O is	already	 sequential,  setting  sequential  for
	      that would not result in any differences.	identical behaves in a
	      similar fashion, except it sends the same	 offset	 8  number  of
	      times before generating a	new offset.

	      Fio  normally  reports statistics	on a per data direction	basis,
	      meaning that reads, writes, and trims are	accounted and reported
	      separately.  If  this option is set fio sums the results and re-
	      port them	as "mixed" instead.

	      Seed the random number generator used for	random I/O patterns in
	      a	 predictable way so the	pattern	is repeatable across runs. De-
	      fault: true.

	      Seed all random number generators	in a predictable  way  so  re-
	      sults are	repeatable across runs.	Default: false.

	      Seed  the	 random	number generators based	on this	seed value, to
	      be able to control what sequence of output is  being  generated.
	      If  not  set, the	random sequence	depends	on the randrepeat set-

	      Whether pre-allocation is	performed when laying down files.  Ac-
	      cepted values are:

		     none   Do not pre-allocate	space.

		     native Use	 a  platform's	native pre-allocation call but
			    fall back to none behavior if it fails/is not  im-

		     posix  Pre-allocate via posix_fallocate(3).

		     keep   Pre-allocate    via	   fallocate(2)	   with	  FAL-
			    LOC_FL_KEEP_SIZE set.

			    Extend file	to final size using ftruncate|(2)  in-
			    stead of allocating.

		     0	    Backward-compatible	alias for none.

		     1	    Backward-compatible	alias for posix.

	      May  not	be  available on all supported platforms. keep is only
	      available	on Linux. If using ZFS on Solaris this cannot  be  set
	      to  posix	 because  ZFS doesn't support pre-allocation. Default:
	      native if	any pre-allocation methods except truncate are	avail-
	      able, none if not.

	      Note  that  using	truncate on Windows will interact surprisingly
	      with non-sequential write	patterns. When writing to a file  that
	      has  been	 extended by setting the end-of-file information, Win-
	      dows will	backfill the unwritten portion of the file up to  that
	      offset with zeroes before	issuing	the new	write. This means that
	      a	single small write to the end of an extended file  will	 stall
	      until the	entire file has	been filled with zeroes.

	      Use  posix_fadvise(2)  or	 posix_madvise(2) to advise the	kernel
	      what I/O patterns	are likely to be issued. Accepted values are:

		     0	    Backwards compatible hint for "no hint".

		     1	    Backwards compatible hint  for  "advise  with  fio
			    workload type". This uses FADV_RANDOM for a	random
			    workload, and  FADV_SEQUENTIAL  for	 a  sequential

			    Advise using FADV_SEQUENTIAL.

		     random Advise using FADV_RANDOM.

	      Use  fcntl(2) to advise the kernel what life time	to expect from
	      a	write. Only supported on Linux,	as of version  4.13.  Accepted
	      values are:

		     none   No particular life time associated with this file.

		     short  Data written to this file has a short life time.

		     medium Data written to this file has a medium life	time.

		     long   Data written to this file has a long life time.

			    Data  written  to  this  file has a	very long life

	      The values are all relative to each other, and no	absolute mean-
	      ing should be associated with them.

	      Start  I/O at the	provided offset	in the file, given as either a
	      fixed size in bytes or a percentage. If a	percentage  is	given,
	      the generated offset will	be aligned to the minimum blocksize or
	      to the value of offset_align if provided.	Data before the	 given
	      offset  will not be touched. This	effectively caps the file size
	      at `real_size - offset'. Can be combined with size to  constrain
	      the  start  and end range	of the I/O workload.  A	percentage can
	      be specified by a	number between 1 and 100 followed by '%',  for
	      example, `offset=20%' to specify 20%.

	      If  set  to  non-zero value, the byte offset generated by	a per-
	      centage offset is	aligned	upwards	to this	value. Defaults	 to  0
	      meaning that a percentage	offset is aligned to the minimum block

	      If this is provided, then	the real offset	becomes	`offset	+ off-
	      set_increment  *	thread_number',	 where	the thread number is a
	      counter that starts at 0 and is  incremented  for	 each  sub-job
	      (i.e.  when  numjobs option is specified). This option is	useful
	      if there are several jobs	which are intended  to	operate	 on  a
	      file  in	parallel  disjoint segments, with even spacing between
	      the starting points. Percentages can be used  for	 this  option.
	      If  a  percentage	is given, the generated	offset will be aligned
	      to the minimum blocksize or to the value of offset_align if pro-

	      Fio  will	 normally perform I/Os until it	has exhausted the size
	      of the region set	by size, or if it exhaust the  allocated  time
	      (or  hits	an error condition). With this setting,	the range/size
	      can be set independently of the number of	I/Os to	perform.  When
	      fio  reaches  this number, it will exit normally and report sta-
	      tus. Note	that this does not extend the amount of	I/O that  will
	      be  done,	 it will only stop fio if this condition is met	before
	      other end-of-job criteria.

	      If writing to a file, issue an fsync(2) (or its  equivalent)  of
	      the dirty	data for every number of blocks	given. For example, if
	      you give 32 as a parameter, fio will sync	the file  after	 every
	      32  writes  issued. If fio is using non-buffered I/O, we may not
	      sync the file. The exception is the sg I/O  engine,  which  syn-
	      chronizes	 the disk cache	anyway.	Defaults to 0, which means fio
	      does not periodically issue and wait for	a  sync	 to  complete.
	      Also see end_fsync and fsync_on_close.

	      Like fsync but uses fdatasync(2) to only sync data and not meta-
	      data blocks. In Windows, FreeBSD,	DragonFlyBSD or	OSX  there  is
	      no  fdatasync(2) so this falls back to using fsync(2).  Defaults
	      to 0, which means	fio does not periodically issue	and wait for a
	      data-only	sync to	complete.

	      Make every N-th write a barrier write.

	      Use sync_file_range(2) for every int number of write operations.
	      Fio will track range of writes that have happened	since the last
	      sync_file_range(2) call. str can currently be one	or more	of:




	      So  if  you  do `sync_file_range=wait_before,write:8', fio would
	      every  8	writes.	Also see the sync_file_range(2)	man page. This
	      option is	Linux specific.

	      If true, writes to a file	will always overwrite  existing	 data.
	      If the file doesn't already exist, it will be created before the
	      write phase begins. If the file exists and is large  enough  for
	      the specified write phase, nothing will be done. Default:	false.

	      If  true,	 fsync(2)  file	 contents  when	a write	stage has com-
	      pleted.  Default:	false.

	      If true, fio will	fsync(2) a dirty file on close.	 This  differs
	      from  end_fsync  in that it will happen on every file close, not
	      just at the end of the job. Default: false.

	      Percentage of a mixed workload that should  be  reads.  Default:

	      Percentage  of  a	 mixed workload	that should be writes. If both
	      rwmixread	and rwmixwrite is given	and the	values do not  add  up
	      to  100%,	 the  latter  of  the two will be used to override the
	      first. This may interfere	with a given rate setting, if  fio  is
	      asked to limit reads or writes to	a certain rate.	If that	is the
	      case, then the distribution may be skewed. Default: 50.

	      By default, fio will use a completely uniform  random  distribu-
	      tion when	asked to perform random	I/O. Sometimes it is useful to
	      skew the distribution in specific	ways, ensuring that some parts
	      of the data is more hot than others.  fio	includes the following
	      distribution models:

		     random Uniform random distribution

		     zipf   Zipf distribution

		     pareto Pareto distribution

		     normal Normal (Gaussian) distribution

		     zoned  Zoned random distribution zoned_abs	Zoned absolute
			    random distribution

	      When using a zipf	or pareto distribution,	an input value is also
	      needed to	define the access pattern. For zipf, this is the `Zipf
	      theta'.	For  pareto,  it's  the	`Pareto	power'.	Fio includes a
	      test program, fio-genzipf, that can be used visualize  what  the
	      given  input  values  will  yield	 in terms of hit rates.	If you
	      wanted to	use zipf with a	`theta'	of 1.2,	you  would  use	 `ran-
	      dom_distribution=zipf:1.2' as the	option.	If a non-uniform model
	      is used, fio will	disable	use of the random map. For the	normal
	      distribution,  a	normal	(Gaussian)  deviation is supplied as a
	      value between 0 and 100.

	      For a zoned distribution,	fio supports specifying	percentages of
	      I/O access that should fall within what range of the file	or de-
	      vice. For	example, given a criteria of:

		     60% of accesses should be to the first 10%
		     30% of accesses should be to the next 20%
		     8%	of accesses should be to the next 30%
		     2%	of accesses should be to the next 40%

	      we can define that through zoning	of the	random	accesses.  For
	      the above	example, the user would	do:


	      A	 zoned_abs  distribution  works	 exactly like thezoned,	except
	      that it takes absolute sizes. For	example, let's say you	wanted
	      to define	access according to the	following criteria:

		     60% of accesses should be to the first 20G
		     30% of accesses should be to the next 100G
		     10% of accesses should be to the next 500G

	      we can define an absolute	zoning distribution with:


	      For  both	 zoned	and zoned_abs, fio supports defining up	to 256
	      separate zones.

	      Similarly	to how bssplit works for setting ranges	 and  percent-
	      ages of block sizes. Like	bssplit, it's possible to specify sep-
	      arate zones for reads, writes, and trims.	If  just  one  set  is
	      given, it'll apply to all	of them.

	      For  a  random workload, set how big a percentage	should be ran-
	      dom. This	defaults to 100%, in which case	the workload is	 fully
	      random. It can be	set from anywhere from 0 to 100. Setting it to
	      0	would make the workload	fully sequential. Any setting  in  be-
	      tween  will result in a random mix of sequential and random I/O,
	      at the given percentages.	Comma-separated	values may  be	speci-
	      fied for reads, writes, and trims	as described in	blocksize.

	      Normally	fio will cover every block of the file when doing ran-
	      dom I/O. If this option is given,	fio will just get a new	random
	      offset without looking at	past I/O history. This means that some
	      blocks may not be	read or	written, and that some blocks  may  be
	      read/written  more than once. If this option is used with	verify
	      and multiple blocksizes (via bsrange), only  intact  blocks  are
	      verified,	 i.e., partially-overwritten blocks are	ignored.  With
	      an async I/O engine and an I/O depth > 1,	it is possible for the
	      same  block  to be overwritten, which can	cause verification er-
	      rors.  Either do not use norandommap in this case, or  also  use
	      the lfsr random generator.

	      See  norandommap.	 If fio	runs with the random block map enabled
	      and it fails to allocate the map,	if this	option is set it  will
	      continue	without	a random block map. As coverage	will not be as
	      complete as with random maps, this option	 is  disabled  by  de-

	      Fio  supports  the  following engines for	generating I/O offsets
	      for random I/O:

			    Strong 2^88	cycle random number generator.

		     lfsr   Linear feedback shift register generator.

			    Strong 64-bit 2^258	cycle random number generator.

	      tausworthe is a strong random number generator, but it  requires
	      tracking	on  the	side if	we want	to ensure that blocks are only
	      read or written once. lfsr guarantees that we never generate the
	      same offset twice, and it's also less computationally expensive.
	      It's not a true random generator,	however, though	for  I/O  pur-
	      poses  it's  typically  good enough. lfsr	only works with	single
	      block sizes, not with workloads that use multiple	 block	sizes.
	      If  used with such a workload, fio may read or write some	blocks
	      multiple times. The default value	is tausworthe, unless the  re-
	      quired  space exceeds 2^32 blocks. If it does, then tausworthe64
	      is selected automatically.

   Block size
       blocksize=int[,int][,int], bs=int[,int][,int]
	      The block	size in	bytes used for I/O  units.  Default:  4096.  A
	      single  value  applies  to reads,	writes,	and trims. Comma-sepa-
	      rated values may be specified for	reads, writes,	and  trims.  A
	      value not	terminated in a	comma applies to subsequent types. Ex-

		     bs=256k	    means 256k for reads, writes and trims.
		     bs=8k,32k	    means 8k for reads,	 32k  for  writes  and
		     bs=8k,32k,	     means  8k	for reads, 32k for writes, and
		     default for trims.
		     bs=,8k	    means default for reads, 8k	for writes and
		     bs=,8k,	     means  default  for reads,	8k for writes,
		     and default for trims.

	      A	 range	of  block sizes	in bytes for I/O units.	The issued I/O
	      unit will	always be a  multiple  of  the	minimum	 size,	unless
	      blocksize_unaligned is set.  Comma-separated ranges may be spec-
	      ified for	reads, writes, and trims as  described	in  blocksize.


	      Sometimes	you want even finer grained control of the block sizes
	      issued, not just an even split between them. This	option	allows
	      you  to  weight various block sizes, so that you are able	to de-
	      fine a specific amount of	block sizes  issued.  The  format  for
	      this option is:


	      for  as  many  block sizes as needed. So if you want to define a
	      workload that has	50% 64k	blocks,	10% 4k	blocks,	 and  40%  32k
	      blocks, you would	write:


	      Ordering	does  not matter. If the percentage is left blank, fio
	      will fill	in the remaining values	evenly.	So  a  bssplit	option
	      like this	one:


	      would  have  50% 4k ios, and 25% 1k and 32k ios. The percentages
	      always add up to 100, if bssplit is given	a range	that  adds  up
	      to more, it will error out.

	      Comma-separated  values  may be specified	for reads, writes, and
	      trims as described in blocksize.

	      If you want a workload that has 50% 2k reads and 50%  4k	reads,
	      while having 90% 4k writes and 10% 8k writes, you	would specify:


	      Fio  supports  defining up to 64 different weights for each data

       blocksize_unaligned, bs_unaligned
	      If set, fio will issue I/O units with  any  size	within	block-
	      size_range,  not	just multiples of the minimum size. This typi-
	      cally won't work with direct I/O,	as that	normally requires sec-
	      tor alignment.

	      If this option is	set, fio will use the normal read,write	block-
	      size settings as sequential,random blocksize  settings  instead.
	      Any  random read or write	will use the WRITE blocksize settings,
	      and any sequential read or write will  use  the  READ  blocksize

       blockalign=int[,int][,int], ba=int[,int][,int]
	      Boundary	to  which  fio	will  align random I/O units. Default:
	      blocksize. Minimum alignment is typically	512b for using	direct
	      I/O,  though it usually depends on the hardware block size. This
	      option is	mutually exclusive with	using a	random map for	files,
	      so  it  will turn	off that option. Comma-separated values	may be
	      specified	for reads, writes, and trims as	 described  in	block-

   Buffers and memory
	      Initialize  buffers  with	 all zeros. Default: fill buffers with
	      random data.

	      If this option is	given, fio will	refill the I/O buffers on  ev-
	      ery  submit. The default is to only fill it at init time and re-
	      use that data. Only makes	sense if zero_buffers isn't specified,
	      naturally.  If  data  verification is enabled, refill_buffers is
	      also automatically enabled.

	      If refill_buffers	is too costly and the  target  is  using  data
	      deduplication, then setting this option will slightly modify the
	      I/O buffer contents to defeat normal de-dupe attempts.  This  is
	      not enough to defeat more	clever block compression attempts, but
	      it will stop naive dedupe	of blocks. Default: true.

	      If this is set, then fio will attempt to provide I/O buffer con-
	      tent  (on	 WRITEs)  that	compresses to the specified level. Fio
	      does this	by providing a mix of random data  followed  by	 fixed
	      pattern  data. The fixed pattern is either zeros,	or the pattern
	      specified	by buffer_pattern. If  the  buffer_pattern  option  is
	      used, it might skew the compression ratio	slightly. Setting buf-
	      fer_compress_percentage to a value other than 100	will also  en-
	      able refill_buffers in order to reduce the likelihood that adja-
	      cent blocks are so similar that they over	compress when seen to-
	      gether.  See  buffer_compress_chunk  for	how  to	set a finer or
	      coarser granularity of the random/fixed data  regions.  Defaults
	      to  unset	 i.e.,	buffer data will not adhere to any compression

	      This setting allows fio to manage	how big	the random/fixed  data
	      region  is  when	using  buffer_compress_percentage.  When  buf-
	      fer_compress_chunk is set	to some	non-zero  value	 smaller  than
	      the  block size, fio can repeat the random/fixed region through-
	      out the I/O buffer at the	specified interval (which particularly
	      useful  when bigger block	sizes are used for a job). When	set to
	      0, fio will use a	chunk size that	matches	the block size result-
	      ing  in  a single	random/fixed region within the I/O buffer. De-
	      faults to	512. When the unit is omitted,	the  value  is	inter-
	      preted in	bytes.

	      If  set, fio will	fill the I/O buffers with this pattern or with
	      the contents of a	file. If not set, the contents of I/O  buffers
	      are defined by the other options related to buffer contents. The
	      setting can be any pattern of bytes, and can be prefixed with 0x
	      for  hex	values.	It may also be a string, where the string must
	      then be wrapped with "". Or it may also be a filename, where the
	      filename	must  be  wrapped  with	 ''  in	which case the file is
	      opened and read. Note that not all the  file  contents  will  be
	      read  if that would cause	the buffers to overflow. So, for exam-


	      Also you can combine everything together in any order:


	      If set, fio will generate	this percentage	of  identical  buffers
	      when  writing.  These  buffers  will be naturally	dedupable. The
	      contents of the buffers depend on	what other buffer  compression
	      settings	have  been  set.  It's possible	to have	the individual
	      buffers either fully compressible, or not	at all -- this	option
	      only  controls  the distribution of unique buffers. Setting this
	      option will also enable refill_buffers to	prevent	 every	buffer
	      being identical.

	      Invalidate  the  buffer/page cache parts of the files to be used
	      prior to starting	I/O if the platform and	file type support  it.
	      Defaults	to  true.   This  will	be ignored if pre_read is also
	      specified	for the	same job.

	      Whether, and what	type, of synchronous I/O to  use  for  writes.
	      The allowed values are:

		     none   Do not use synchronous IO, the default.

		     0	    Same as none.

		     sync   Use	 synchronous  file IO. For the majority	of I/O
			    engines, this means	using O_SYNC.

		     1	    Same as sync.

		     dsync  Use	synchronous data IO. For the majority  of  I/O
			    engines, this means	using O_DSYNC.

       iomem=str, mem=str
	      Fio  can use various types of memory as the I/O unit buffer. The
	      allowed values are:

		     malloc Use	memory from malloc(3) as the buffers.  Default
			    memory type.

		     shm    Use	  shared  memory  as  the  buffers.  Allocated
			    through shmget(2).

			    Same as shm, but use huge pages as backing.

		     mmap   Use	mmap(2)	to allocate  buffers.  May  either  be
			    anonymous memory, or can be	file backed if a file-
			    name is given after	 the  option.  The  format  is

			    Use	 a memory mapped huge file as the buffer back-
			    ing. Append	filename after mmaphuge, ala `mem=mma-

			    Same as mmap, but use a MMAP_SHARED	mapping.

			    Use	 GPU  memory as	the buffers for	GPUDirect RDMA
			    benchmark.	The ioengine must be rdma.

	      The area allocated is a function of the maximum allowed bs  size
	      for  the	job,  multiplied by the	I/O depth given. Note that for
	      shmhuge and mmaphuge to work, the	system	must  have  free  huge
	      pages  allocated.	 This can normally be checked and set by read-
	      ing/writing `/proc/sys/vm/nr_hugepages' on a Linux  system.  Fio
	      assumes  a huge page is 4MiB in size. So to calculate the	number
	      of huge pages you	need for a given job  file,  add  up  the  I/O
	      depth of all jobs	(normally one unless iodepth is	used) and mul-
	      tiply by the maximum bs set. Then	divide that number by the huge
	      page size. You can see the size of the huge pages	in `/proc/mem-
	      info'. If	no huge	pages are allocated by having a	non-zero  num-
	      ber in `nr_hugepages', using mmaphuge or shmhuge will fail. Also
	      see hugepage-size.

	      mmaphuge also needs to have hugetlbfs mounted and	the file loca-
	      tion  should  point  there.  So  if it's mounted in `/huge', you
	      would use	`mem=mmaphuge:/huge/somefile'.

       iomem_align=int,	mem_align=int
	      This indicates the memory	alignment of the I/O  memory  buffers.
	      Note  that  the given alignment is applied to the	first I/O unit
	      buffer, if using iodepth the alignment of	the following  buffers
	      are  given by the	bs used. In other words, if using a bs that is
	      a	multiple of the	page sized in the system, all buffers will  be
	      aligned  to  this	value. If using	a bs that is not page aligned,
	      the alignment of subsequent I/O memory buffers is	the sum	of the
	      iomem_align and bs used.

	      Defines  the  size of a huge page. Must at least be equal	to the
	      system setting, see `/proc/meminfo'. Defaults  to	 4MiB.	Should
	      probably	 always	  be   a   multiple  of	 megabytes,  so	 using
	      `hugepage-size=Xm' is the	preferred way to  set  this  to	 avoid
	      setting a	non-pow-2 bad value.

	      Pin the specified	amount of memory with mlock(2).	Can be used to
	      simulate a smaller amount	of memory. The amount specified	is per

   I/O size
	      The total	size of	file I/O for each thread of this job. Fio will
	      run until	this many bytes	has been transferred,  unless  runtime
	      is  limited  by other options (such as runtime, for instance, or
	      increased/decreased by io_size).	Fio will divide	this size  be-
	      tween the	available files	determined by options such as nrfiles,
	      filename,	unless filesize	is specified by	the job. If the	result
	      of  division  happens  to	 be 0, the size	is set to the physical
	      size of the given	files or devices if they exist.	 If  this  op-
	      tion  is	not specified, fio will	use the	full size of the given
	      files or devices.	If the files do	not exist, size	must be	given.
	      It  is  also possible to give size as a percentage between 1 and
	      100. If `size=20%' is given, fio will use	20% of the  full  size
	      of  the  given files or devices.	Can be combined	with offset to
	      constrain	the start and end range	that I/O will be done within.

       io_size=int, io_limit=int
	      Normally fio operates within the region set by size, which means
	      that  the	size option sets both the region and size of I/O to be
	      performed. Sometimes that	is not what you	want.  With  this  op-
	      tion,  it	 is possible to	define just the	amount of I/O that fio
	      should do. For instance, if size is set to 20GiB and io_size  is
	      set  to  5GiB,  fio  will	perform	I/O within the first 20GiB but
	      exit when	5GiB have been done. The opposite is also possible  --
	      if  size	is set to 20GiB, and io_size is	set to 40GiB, then fio
	      will do 40GiB of I/O within the 0..20GiB region.	Value  can  be
	      set  as percentage: io_size=N%.  In this case io_size multiplies
	      size= value.

	      Individual file sizes. May be a range, in	which  case  fio  will
	      select sizes for files at	random within the given	range and lim-
	      ited to size in total (if	that is	given).	 If  not  given,  each
	      created  file  is	 the same size.	 This option overrides size in
	      terms of file size, which	means this value is used  as  a	 fixed
	      size or possible range of	each file.

	      Perform I/O after	the end	of the file. Normally fio will operate
	      within the size of a file. If this option	is set,	then fio  will
	      append  to the file instead. This	has identical behavior to set-
	      ting offset to the size of a file. This  option  is  ignored  on
	      non-regular files.

       fill_device=bool, fill_fs=bool
	      Sets  size  to  something	 really	large and waits	for ENOSPC (no
	      space left on device) as the terminating condition.  Only	 makes
	      sense  with  sequential  write.  For  a read workload, the mount
	      point will be filled first then I/O started on the result.  This
	      option  doesn't  make  sense  if operating on a raw device node,
	      since the	size of	that is	already	known by the file system.  Ad-
	      ditionally,  writing beyond end-of-device	will not return	ENOSPC

   I/O engine
	      Defines how the job issues I/O to	the file. The following	 types
	      are defined:

		     sync   Basic read(2) or write(2) I/O. lseek(2) is used to
			    position the I/O location.	See fsync  and	fdata-
			    sync for syncing write I/Os.

		     psync  Basic  pread(2)  or	 pwrite(2) I/O.	Default	on all
			    supported operating	systems	except for Windows.

		     vsync  Basic readv(2)  or	writev(2)  I/O.	 Will  emulate
			    queuing  by	coalescing adjacent I/Os into a	single

		     pvsync Basic preadv(2) or pwritev(2) I/O.

			    Basic preadv2(2) or	pwritev2(2) I/O.

		     libaio Linux native asynchronous I/O. Note	that Linux may
			    only support queued	behavior with non-buffered I/O
			    (set `direct=1' or `buffered=0').  This engine de-
			    fines engine specific options.

			    POSIX   asynchronous  I/O  using  aio_read(3)  and

			    Solaris native asynchronous	I/O.

			    Windows native asynchronous	I/O. Default  on  Win-

		     mmap   File is memory mapped with mmap(2) and data	copied
			    to/from using memcpy(3).

		     splice splice(2) is used to transfer  the	data  and  vm-
			    splice(2)  to transfer data	from user space	to the

		     sg	    SCSI generic sg v3 I/O. May	either be  synchronous
			    using  the	SG_IO ioctl, or	if the target is an sg
			    character device we	use read(2) and	 write(2)  for
			    asynchronous  I/O.	Requires  filename  option  to
			    specify either block or  character	devices.  This
			    engine supports trim operations. The sg engine in-
			    cludes engine specific options.

		     libzbc Synchronous	I/O engine for	SMR  hard-disks	 using
			    the	libzbc library.	The target can be either an sg
			    character device or	a block	device file. This  en-
			    gine supports the zonemode=zbd zone	operations.

		     null   Doesn't  transfer any data,	just pretends to. This
			    is mainly used to exercise fio itself and for  de-
			    bugging/testing purposes.

		     net    Transfer  over  the	 network to given `host:port'.
			    Depending on  the  protocol	 used,	the  hostname,
			    port,  listen  and	filename  options  are used to
			    specify what sort of connection to make, while the
			    protocol  option determines	which protocol will be
			    used. This engine defines engine specific options.

			    Like net, but uses splice(2)  and  vmsplice(2)  to
			    map	 data  and  send/receive.  This	engine defines
			    engine specific options.

		     cpuio  Doesn't transfer any data, but  burns  CPU	cycles
			    according  to  the	cpuload	and cpuchunks options.
			    Setting cpuload=85 will cause that job to do noth-
			    ing	 but  burn  85%	of the CPU. In case of SMP ma-
			    chines, use	`numjobs=<nr_of_cpu>' to  get  desired
			    CPU	 usage,	as the cpuload only loads a single CPU
			    at the desired rate. A job never  finishes	unless
			    there is at	least one non-cpuio job.

		     rdma   The	 RDMA I/O engine supports both RDMA memory se-
			    mantics (RDMA_WRITE/RDMA_READ) and channel	seman-
			    tics  (Send/Recv)  for  the	 InfiniBand,  RoCE and
			    iWARP protocols. This engine defines  engine  spe-
			    cific options.

		     falloc I/O	engine that does regular fallocate to simulate
			    data transfer as fio ioengine.

			    DDIR_READ	     does   fallocate(,mode   =	  FAL-
			    DIR_WRITE	   does	fallocate(,mode	= 0).
			    DDIR_TRIM	     does   fallocate(,mode   =	  FAL-

			    I/O	engine that sends ftruncate(2)	operations  in
			    response to	write (DDIR_WRITE) events. Each	ftrun-
			    cate issued	sets the file's	size  to  the  current
			    block offset. blocksize is ignored.

			    I/O	 engine	 that  does  regular EXT4_IOC_MOVE_EXT
			    ioctls to simulate defragment activity in  request
			    to DDIR_WRITE event.

		     rados  I/O	 engine	supporting direct access to Ceph Reli-
			    able Autonomic Distributed	Object	Store  (RADOS)
			    via	 librados.  This  ioengine defines engine spe-
			    cific options.

		     rbd    I/O	engine supporting direct access	to Ceph	 Rados
			    Block Devices (RBD)	via librbd without the need to
			    use	the kernel rbd driver. This  ioengine  defines
			    engine specific options.

		     http   I/O	  engine   supporting  GET/PUT	requests  over
			    HTTP(S) with libcurl to a WebDAV or	 S3  endpoint.
			    This ioengine defines engine specific options.

			    This  engine only supports direct IO of iodepth=1;
			    you	need to	scale this via numjobs.	blocksize  de-
			    fines the size of the objects to be	created.

			    TRIM is translated to object deletion.

		     gfapi  Using  GlusterFS libgfapi sync interface to	direct
			    access to GlusterFS	volumes	without	having	to  go
			    through  FUSE.  This  ioengine defines engine spe-
			    cific options.

			    Using GlusterFS libgfapi async interface to	direct
			    access  to	GlusterFS volumes without having to go
			    through FUSE. This ioengine	 defines  engine  spe-
			    cific options.

			    Read and write through Hadoop (HDFS). The filename
			    option is used to specify host,port	 of  the  hdfs
			    name-node  to connect. This	engine interprets off-
			    sets a little differently.	In  HDFS,  files  once
			    created  cannot  be	 modified so random writes are
			    not	possible. To imitate this the  libhdfs	engine
			    expects  a bunch of	small files to be created over
			    HDFS and will randomly pick	a file from them based
			    on	the  offset  generated by fio backend (see the
			    example  job  file	to  create  such  files,   use
			    `rw=write'	option). Please	note, it may be	neces-
			    sary to set	environment  variables	to  work  with
			    HDFS/libhdfs  properly. Each job uses its own con-
			    nection to HDFS.

		     mtd    Read, write	and  erase  an	MTD  character	device
			    (e.g.,   `/dev/mtd0').  Discards  are  treated  as
			    erases. Depending on the underlying	 device	 type,
			    the	I/O may	have to	go in a	certain	pattern, e.g.,
			    on NAND, writing sequentially to erase blocks  and
			    discarding	before overwriting. The	trimwrite mode
			    works well for this	constraint.

			    Read and write using filesystem DAX	to a file on a
			    filesystem mounted with DAX	on a persistent	memory
			    device through the PMDK libpmemblk library.

			    Read and write using device	DAX  to	 a  persistent
			    memory device (e.g., /dev/dax0.0) through the PMDK
			    libpmem library.

			    Prefix to specify loading an external  I/O	engine
			    object file. Append	the engine filename, e.g. `io-
			    engine=external:/tmp/foo.o'	  to   load   ioengine
			    `foo.o' in `/tmp'. The path	can be either absolute
			    or relative. See `engines/skeleton_external.c'  in
			    the	 fio source for	details	of writing an external
			    I/O	engine.

			    Simply create the files and	do  no	I/O  to	 them.
			    You	still need to set filesize so that all the ac-
			    counting still occurs, but no actual I/O  will  be
			    done other than creating the file.

			    Simply  do	stat()	and do no I/O to the file. You
			    need to set	 'filesize'  and  'nrfiles',  so  that
			    files  will	be created.  This engine is to measure
			    file lookup	and meta data access.

			    Read and write using mmap  I/O  to	a  file	 on  a
			    filesystem mounted with DAX	on a persistent	memory
			    device through the PMDK libpmem library.

			    Synchronous	read and write	using  DDN's  Infinite
			    Memory Engine (IME). This engine is	very basic and
			    issues calls to IME	whenever an IO is queued.

			    Synchronous	read and write	using  DDN's  Infinite
			    Memory  Engine  (IME). This	engine uses iovecs and
			    will try to	stack as much IOs as possible (if  the
			    IOs	 are  "contiguous" and the IO depth is not ex-
			    ceeded) before issuing a call to IME.

			    Asynchronous read and write	using  DDN's  Infinite
			    Memory Engine (IME). This engine will try to stack
			    as much IOs	as possible by creating	 requests  for
			    IME.   FIO	will  then decide when to commit these

			    Read and write iscsi lun with libiscsi.

		     nbd    Synchronous	read and write a Network Block	Device

   I/O engine specific parameters
       In addition, there are some parameters which are	only valid when	a spe-
       cific ioengine is in use. These are used	identically to normal  parame-
       ters,  with  the	 caveat	 that when used	on the command line, they must
       come after the ioengine that defines them is selected.

	      Set the percentage of I/O	that will be issued with higher	prior-
	      ity  by  setting	the priority bit. Non-read I/O is likely unaf-
	      fected by	``cmdprio_percentage``.	 This option  cannot  be  used
	      with  the	 `prio`	or `prioclass` options.	For this option	to set
	      the priority bit properly, NCQ priority must  be	supported  and
	      enabled and `direct=1' option must be used. fio must also	be run
	      as the root user.

	      If fio is	asked to do direct IO, then Linux will map  pages  for
	      each  IO	call, and release them when IO is done.	If this	option
	      is set, the pages	are pre-mapped	before	IO  is	started.  This
	      eliminates  the  need  to	 map and release for each IO.  This is
	      more efficient, and reduces the IO latency as well.

	      If this option is	set, fio will attempt to use polled IO comple-
	      tions.  Normal  IO completions generate interrupts to signal the
	      completion of IO,	polled completions do not. Hence they are  re-
	      quire  active reaping by the application.	 The benefits are more
	      efficient	IO for high IOPS scenarios, and	 lower	latencies  for
	      low queue	depth IO.

	      With this	option,	fio registers the set of files being used with
	      the kernel.  This	avoids the overhead of managing	file counts in
	      the  kernel,  making  the	 submission  and  completion part more
	      lightweight. Required for	the below sqthread_poll	option.

	      Normally fio will	submit IO by issuing a system call  to	notify
	      the  kernel of available items in	the SQ ring. If	this option is
	      set, the act of submitting IO will be done by a  polling	thread
	      in  the kernel. This frees up cycles for fio, at the cost	of us-
	      ing more CPU in the system.

	      When `sqthread_poll` is set, this	option provides	a way  to  de-
	      fine which CPU should be used for	the polling thread.

	      Normally,	 with  the  libaio  engine  in	use,  fio will use the
	      io_getevents(3) system call to reap newly	returned events.  With
	      this  flag  turned  on,  the AIO ring will be read directly from
	      user-space to reap events. The reaping mode is only enabled when
	      polling for a minimum of 0 events	(e.g. when `iodepth_batch_com-

	      Set RWF_HIPRI on I/O, indicating to  the	kernel	that  it's  of
	      higher priority than normal.

	      When  hipri  is set this determines the probability of a pvsync2
	      I/O being	high priority. The default is 100%.

	      By default if a request cannot be	executed immediately (e.g. re-
	      source starvation, waiting on locks) it is queued	and the	initi-
	      ating process will be blocked until the  required	 resource  be-
	      comes  free.   This  option  sets	the RWF_NOWAIT flag (supported
	      from the 4.14 Linux kernel) and the call will  return  instantly
	      with EAGAIN or a partial result rather than waiting.

	      It is useful to also use ignore_error=EAGAIN when	using this op-
	      tion.  Note: glibc 2.27, 2.28 have a  bug	 in  syscall  wrappers
	      preadv2, pwritev2.  They return EOPNOTSUP	instead	of EAGAIN.

	      For  cached I/O, using this option usually means a request oper-
	      ates only	with cached data. Currently the	RWF_NOWAIT  flag  does
	      not  supported  for cached write.	 For direct I/O, requests will
	      only succeed if cache invalidation isn't required,  file	blocks
	      are fully	allocated and the disk request could be	issued immedi-

	      Attempt to use the specified percentage of CPU cycles. This is a
	      mandatory	option when using cpuio	I/O engine.

	      Split the	load into cycles of the	given time. In microseconds.

	      Detect when I/O threads are done,	then exit.

	      The  hostname  or	 IP address of a HDFS cluster namenode to con-

	      The listening port of the	HFDS cluster namenode.

	      The TCP or UDP port to bind to or	connect	to. If	this  is  used
	      with  numjobs  to	spawn multiple instances of the	same job type,
	      then this	will be	the starting port number since fio will	use  a
	      range of ports.

	      The  port	 to  use for RDMA-CM communication. This should	be the
	      same value on the	client and the server side.

	      The hostname or IP address to use	for TCP, UDP or	RDMA-CM	 based
	      I/O.   If	 the job is a TCP listener or UDP reader, the hostname
	      is not used and must be omitted unless it	is a valid UDP	multi-
	      cast address.

	      The  IP address of the network interface used to send or receive
	      UDP multicast.

	      Time-to-live value for outgoing UDP multicast packets.  Default:

	      Set TCP_NODELAY on TCP connections.

       (netsplice,net)protocol=str, proto=str
	      The network protocol to use. Accepted values are:

		     tcp    Transmission control protocol.

		     tcpv6  Transmission control protocol V6.

		     udp    User datagram protocol.

		     udpv6  User datagram protocol V6.

		     unix   UNIX domain	socket.

	      When the protocol	is TCP or UDP, the port	must also be given, as
	      well as the hostname if the job is a TCP listener	or UDP reader.
	      For  unix	sockets, the normal filename option should be used and
	      the port is invalid.

	      For TCP network connections, tell	fio  to	 listen	 for  incoming
	      connections  rather  than	initiating an outgoing connection. The
	      hostname must be omitted if this option is used.

	      Normally a network writer	will just continue writing data, and a
	      network  reader  will  just consume packages. If `pingpong=1' is
	      set, a writer will send its normal payload to the	 reader,  then
	      wait  for	 the reader to send the	same payload back. This	allows
	      fio to measure network latencies.	The submission and  completion
	      latencies	 then  measure	local time spent sending or receiving,
	      and the completion latency measures how long  it	took  for  the
	      other  end  to  receive and send back. For UDP multicast traffic
	      `pingpong=1' should only be set for a single reader when	multi-
	      ple readers are listening	to the same address.

	      Set the desired socket buffer size for the connection.

	      Set the TCP maximum segment size (TCP_MAXSEG).

	      File will	be used	as a block donor (swap extents between files).

	      Configure	donor file blocks allocation strategy:

		     0	    Default. Preallocate donor's file on init.

		     1	    Allocate   space   immediately  inside  defragment
			    event, and free right after	event.

	      Specifies	the name of the	Ceph cluster.

	      Specifies	the name of the	RBD.

	      Specifies	the name of the	Ceph  pool  containing	RBD  or	 RADOS

	      Specifies	 the  username	(without the 'client.' prefix) used to
	      access the Ceph cluster. If the clustername  is  specified,  the
	      clientname  shall	be the full ** string. If no type. pre-
	      fix is given, fio	will add 'client.'  by default.

	      Poll store instead of waiting for	completion. Usually this  pro-
	      vides  better  throughput	at cost	of higher(up to	100%) CPU uti-

	      Hostname to connect to. For S3, this could be the	 bucket	 name.
	      Default is localhost

	      Username for HTTP	authentication.

	      Password for HTTP	authentication.

	      Whether  to  use	HTTPS instead of plain HTTP. on	enables	HTTPS;
	      insecure will enable HTTPS, but disable  SSL  peer  verification
	      (use with	caution!).  Default is off.

	      Which  HTTP access mode to use: webdav, swift, or	s3. Default is

	      The S3 region/zone to include in the  request.  Default  is  us-

	      The S3 secret key.

	      The S3 key/access	id.

	      The  Swift auth token. See the example configuration file	on how
	      to retrieve this.

	      Enable verbose requests from libcurl. Useful  for	 debugging.  1
	      turns  on	 verbose  logging from libcurl,	2 additionally enables
	      HTTP IO tracing.	Default	is 0

	      Skip operations against known bad	blocks.

	      libhdfs will create chunk	in this	HDFS directory.

	      The size of the chunk to use for each file.

	      The RDMA verb to use on this side	of the RDMA  ioengine  connec-
	      tion.  Valid values are write, read, send	and recv. These	corre-
	      spond to the equivalent RDMA  verbs  (e.g.  write	 =  rdma_write
	      etc.).  Note  that this only needs to be specified on the	client
	      side of the connection. See the examples folder.

	      The name to use to bind the local	RDMA-CM	connection to a	 local
	      RDMA  device.  This  could  be a hostname	or an IPv4 or IPv6 ad-
	      dress.  On  the  server  side  this  will	 be  passed  into  the
	      rdma_bind_addr() function	and on the client site it will be used
	      in the rdma_resolve_add()	function. This can be useful when mul-
	      tiple  paths  exist between the client and the server or in cer-
	      tain loopback configurations.

	      Specify stat system call type to measure lookup/getattr  perfor-
	      mance.  Default is stat for stat(2).

	      With  readfua option set to 1, read operations include the force
	      unit access (fua)	flag. Default: 0.

	      With writefua option set to  1,  write  operations  include  the
	      force unit access	(fua) flag. Default: 0.

	      Specify  the  type  of  write commands to	issue. This option can
	      take three values:

		     write (default)
			    Write opcodes are issued as	usual

		     verify Issue WRITE	AND VERIFY commands. The BYTCHK	bit is
			    set	 to  0.	This directs the device	to carry out a
			    medium verification	with no	data  comparison.  The
			    writefua option is ignored with this selection.

		     same   Issue WRITE	SAME commands. This transfers a	single
			    block to the device	and writes this	same block  of
			    data to a contiguous sequence of LBAs beginning at
			    the	specified offset. fio's	block  size  parameter
			    specifies  the  amount  of	data written with each
			    command. However,  the  amount  of	data  actually
			    transferred	to the device is equal to the device's
			    block (sector) size. For a device  with  512  byte
			    sectors,  blocksize=8k  will write 16 sectors with
			    each command. fio will still generate 8k  of  data
			    for	 each command butonly the first	512 bytes will
			    be used and	transferred to the device. The	write-
			    fua	option is ignored with this selection.

	      Specify  the  NBD	 URI  of  the server to	test.  The string is a
	      standard	 NBD   URI   (see
	      vice/nbd/tree/master/doc).  Example URIs:




   I/O depth
	      Number  of  I/O  units  to keep in flight	against	the file. Note
	      that increasing iodepth beyond 1 will not	affect synchronous io-
	      engines  (except for small degrees when verify_async is in use).
	      Even async engines may impose OS restrictions  causing  the  de-
	      sired  depth  not	 to be achieved. This may happen on Linux when
	      using libaio and not setting `direct=1', since buffered  I/O  is
	      not  async on that OS. Keep an eye on the	I/O depth distribution
	      in the fio output	to verify that the achieved depth  is  as  ex-
	      pected. Default: 1.

       iodepth_batch_submit=int, iodepth_batch=int
	      This  defines  how  many pieces of I/O to	submit at once.	It de-
	      faults to	1 which	means that we submit each I/O as soon as it is
	      available,  but can be raised to submit bigger batches of	I/O at
	      the time.	If it is set to	0 the iodepth value will be used.

       iodepth_batch_complete_min=int, iodepth_batch_complete=int
	      This defines how many pieces of I/O to retrieve at once. It  de-
	      faults to	1 which	means that we'll ask for a minimum of 1	I/O in
	      the retrieval process from the kernel. The I/O retrieval will go
	      on  until	 we hit	the limit set by iodepth_low. If this variable
	      is set to	0, then	fio will always	check for completed events be-
	      fore  queuing  more  I/O.	 This helps reduce I/O latency,	at the
	      cost of more retrieval system calls.

	      This defines maximum pieces of I/O to  retrieve  at  once.  This
	      variable	 should	  be   used   along   with  iodepth_batch_com-
	      plete_min=int variable, specifying the  range  of	 min  and  max
	      amount  of I/O which should be retrieved.	By default it is equal
	      to iodepth_batch_complete_min value. Example #1:


	      which means that we will retrieve	at least 1 I/O and up  to  the
	      whole  submitted	queue depth. If	none of	I/O has	been completed
	      yet, we will wait.  Example #2:


	      which means that we can retrieve up to the whole submitted queue
	      depth,  but  if  none of I/O has been completed yet, we will NOT
	      wait and immediately exit	the system call. In  this  example  we
	      simply do	polling.

	      The  low	water  mark indicating when to start filling the queue
	      again. Defaults to the same as iodepth, meaning  that  fio  will
	      attempt  to  keep	the queue full at all times. If	iodepth	is set
	      to e.g. 16 and iodepth_low is set	 to  4,	 then  after  fio  has
	      filled  the  queue  of  16 requests, it will let the depth drain
	      down to 4	before starting	to fill	it again.

	      Serialize	in-flight I/Os that might otherwise  cause  or	suffer
	      from data	races.	When two or more I/Os are submitted simultane-
	      ously, there is no guarantee that	the I/Os will be processed  or
	      completed	 in  the  submitted  order. Further, if	two or more of
	      those I/Os are writes, any overlapping region between  them  can
	      become  indeterminate/undefined on certain storage. These	issues
	      can cause	verification to	fail erratically when at least one  of
	      the  racing I/Os is changing data	and the	overlapping region has
	      a	non-zero size. Setting serialize_overlap tells	fio  to	 avoid
	      provoking	this behavior by explicitly serializing	in-flight I/Os
	      that have	a non-zero overlap. Note that setting this option  can
	      reduce both performance and the iodepth achieved.

	      This  option only	applies	to I/Os	issued for a single job	except
	      when it is enabled along with io_submit_mode=offload. In offload
	      mode,  fio  will	check  for overlap among all I/Os submitted by
	      offload jobs with	serialize_overlap enabled.

	      Default: false.

	      This option controls how fio submits the I/O to the I/O  engine.
	      The  default  is	`inline', which	means that the fio job threads
	      submit and reap I/O directly.  If	 set  to  `offload',  the  job
	      threads  will  offload I/O submission to a dedicated pool	of I/O
	      threads. This requires some coordination and thus	has a  bit  of
	      extra  overhead,	especially  for	lower queue depth I/O where it
	      can increase latencies. The benefit is that fio can manage  sub-
	      mission rates independently of the device	completion rates. This
	      avoids skewed latency reporting if I/O gets backed up on the de-
	      vice side	(the coordinated omission problem). Note that this op-
	      tion cannot reliably be used with	async IO engines.

   I/O rate
	      Stall the	job for	the specified period of	time after an I/O  has
	      completed	 before	issuing	the next. May be used to simulate pro-
	      cessing being done by an application.  When the unit is omitted,
	      the  value  is interpreted in microseconds. See thinktime_blocks
	      and thinktime_spin.

	      Only valid if thinktime is set - pretend to spend	CPU time doing
	      something	 with the data received, before	falling	back to	sleep-
	      ing for the rest of the period specified by thinktime. When  the
	      unit is omitted, the value is interpreted	in microseconds.

	      Only  valid if thinktime is set -	control	how many blocks	to is-
	      sue, before waiting thinktime usecs. If not set, defaults	 to  1
	      which will make fio wait thinktime usecs after every block. This
	      effectively makes	any queue depth	setting	 redundant,  since  no
	      more than	1 I/O will be queued before we have to complete	it and
	      do our thinktime.	In other words,	this setting effectively  caps
	      the queue	depth if the latter is larger.

	      Cap  the bandwidth used by this job. The number is in bytes/sec,
	      the normal suffix	rules apply.  Comma-separated  values  may  be
	      specified	 for  reads,  writes, and trims	as described in	block-

	      For example, using `rate=1m,500k'	would limit reads to  1MiB/sec
	      and  writes  to  500KiB/sec. Capping only	reads or writes	can be
	      done with	`rate=,500k' or	`rate=500k,'  where  the  former  will
	      only limit writes	(to 500KiB/sec)	and the	latter will only limit

	      Tell fio to do whatever it can to	maintain at least  this	 band-
	      width.  Failing  to  meet	this requirement will cause the	job to
	      exit. Comma-separated values may be specified for	reads, writes,
	      and trims	as described in	blocksize.

	      Cap  the bandwidth to this number	of IOPS. Basically the same as
	      rate, just specified independently of bandwidth. If the  job  is
	      given  a block size range	instead	of a fixed value, the smallest
	      block size is used as the	metric.	Comma-separated	values may  be
	      specified	 for  reads,  writes, and trims	as described in	block-

	      If fio doesn't meet this rate of I/O, it will cause the  job  to
	      exit.   Comma-separated  values  may  be	specified  for	reads,
	      writes, and trims	as described in	blocksize.

	      This option controls how fio manages rated I/O submissions.  The
	      default  is `linear', which submits I/O in a linear fashion with
	      fixed delays between I/Os	that gets adjusted based on  I/O  com-
	      pletion  rates. If this is set to	`poisson', fio will submit I/O
	      based on a more real world random	request	 flow,	known  as  the
	      Poisson	    process	  (
	      son_point_process). The lambda will be 10^6 / IOPS for the given

	      By  default,  fio	will attempt to	catch up to the	specified rate
	      setting, if any kind of thinktime	setting	was used. If this  op-
	      tion is set, then	fio will ignore	the thinktime and continue do-
	      ing IO at	the specified rate, instead  of	 entering  a  catch-up
	      mode after thinktime is done.

   I/O latency
	      If  set, fio will	attempt	to find	the max	performance point that
	      the given	workload will run at while maintaining a latency below
	      this  target. When the unit is omitted, the value	is interpreted
	      in microseconds. See latency_window and latency_percentile.

	      Used with	latency_target to specify the sample window  that  the
	      job is run at varying queue depths to test the performance. When
	      the unit is omitted, the value is	interpreted in microseconds.

	      The percentage of	I/Os that must fall within the criteria	speci-
	      fied  by latency_target and latency_window. If not set, this de-
	      faults to	100.0, meaning that all	I/Os must be equal or below to
	      the value	set by latency_target.

	      Used  with latency_target. If false (default), fio will find the
	      highest queue depth that meets latency_target and	exit. If true,
	      fio  will	continue running and try to meet latency_target	by ad-
	      justing queue depth.

	      If set, fio will exit the	job with an ETIMEDOUT error if it  ex-
	      ceeds  this maximum latency. When	the unit is omitted, the value
	      is interpreted in	microseconds.

	      Average bandwidth	for rate and rate_min over this	number of mil-
	      liseconds. Defaults to 1000.

   I/O replay
	      Write  the  issued  I/O  patterns	 to  the  specified  file. See
	      read_iolog. Specify a separate file for each job,	otherwise  the
	      iologs will be interspersed and the file may be corrupt.

	      Open  an	iolog  with  the specified filename and	replay the I/O
	      patterns it contains. This can be	used to	store a	 workload  and
	      replay it	sometime later.	The iolog given	may also be a blktrace
	      binary file, which allows	fio to replay a	workload  captured  by
	      blktrace.	 See blktrace(8) for how to capture such logging data.
	      For blktrace replay, the file needs to be	turned into a blkparse
	      binary  data  file  first	 (`blkparse  <device>  -o /dev/null -d
	      file_for_fio.bin').  You can specify a number of files by	 sepa-
	      rating  the names	with a ':' character.  See the filename	option
	      for information on how to	escape ':' characters within the  file
	      names.  These  files will	be sequentially	assigned to job	clones
	      created by numjobs. '-' is a reserved name,  meaning  read  from
	      stdin,  notably  if  filename is set to '-' which	means stdin as
	      well, then this flag can't be set	to '-'.

	      Determines  how  iolog  is  read.	 If  false  (default)	entire
	      read_iolog  will	be  read at once. If selected true, input from
	      iolog will be read gradually.  Useful when iolog is very	large,
	      or it is generated.

	      When  specified,	rather	than  replaying	 the  logs  passed  to
	      read_iolog, the logs go through a	merge phase  which  aggregates
	      them  into a single blktrace.  The resulting file	is then	passed
	      on as the	read_iolog parameter. The intention here  is  to  make
	      the order	of events consistent. This limits the influence	of the
	      scheduler	compared to replaying multiple blktraces  via  concur-
	      rent jobs.

	      This  is a percentage based option that is index paired with the
	      list of files passed to read_iolog. When merging	is  performed,
	      scale  the  time	of each	event by the corresponding amount. For
	      example,	`--merge_blktrace_scalars="50:100"'  runs  the	 first
	      trace in halftime	and the	second trace in	realtime. This knob is
	      separately tunable from replay_time_scale	which scales the trace
	      during  runtime  and will	not change the output of the merge un-
	      like this	option.

	      This is a	whole number option that is index paired with the list
	      of  files	 passed	 to read_iolog.	When merging is	performed, run
	      each trace for the specified number of iterations. For  example,
	      `--merge_blktrace_iters="2:1"'  runs the first trace for two it-
	      erations and the second trace for	one iteration.

	      When replaying I/O with read_iolog the default  behavior	is  to
	      attempt to respect the timestamps	within the log and replay them
	      with the appropriate delay between IOPS. By setting  this	 vari-
	      able  fio	 will not respect the timestamps and attempt to	replay
	      them as fast as possible while still  respecting	ordering.  The
	      result  is the same I/O pattern to a given device, but different

	      When replaying I/O with read_iolog, fio will honor the  original
	      timing  in  the  trace. With this	option,	it's possible to scale
	      the time.	It's a percentage option, if set to 50 it means	run at
	      50%  the	original  IO  rate in the trace. If set	to 200,	run at
	      twice the	original IO rate. Defaults to 100.

	      While replaying I/O patterns using read_iolog the	default	behav-
	      ior  is to replay	the IOPS onto the major/minor device that each
	      IOP was recorded from. This is sometimes undesirable because  on
	      a	 different machine those major/minor numbers can map to	a dif-
	      ferent device. Changing hardware on the same system can also re-
	      sult in a	different major/minor mapping.	replay_redirect	causes
	      all I/Os to be replayed onto the single specified	device regard-
	      less  of	the  device  it	 was recorded from. i.e. `replay_redi-
	      rect=/dev/sdc' would cause all I/O in the	blktrace or  iolog  to
	      be replayed onto `/dev/sdc'. This	means multiple devices will be
	      replayed onto a single device, if	the  trace  contains  multiple
	      devices.	If  you	 want  multiple	devices	to be replayed concur-
	      rently to	multiple redirected devices  you  must	blkparse  your
	      trace  into separate traces and replay them with independent fio
	      invocations.  Unfortunately this also breaks the strict time or-
	      dering between multiple device accesses.

	      Force  alignment	of  the	byte offsets in	a trace	to this	value.
	      The value	must be	a power	of 2.

	      Scale bye	offsets	down by	this  factor  when  replaying  traces.
	      Should most likely use replay_align as well.

   Threads, processes and job synchronization
	      Sometimes	 it's  useful  to  skip	 certain  IO types in a	replay
	      trace. This could	be, for	instance, eliminating  the  writes  in
	      the trace. Or not	replaying the trims/discards, if you are redi-
	      recting to a device that	doesn't	 support  them.	  This	option
	      takes a comma separated list of read, write, trim, sync.

       thread Fio defaults to creating jobs by using fork, however if this op-
	      tion is given, fio will create  jobs  by	using  POSIX  Threads'
	      function pthread_create(3) to create threads instead.

	      If  set,	the  current job won't be started until	all workers of
	      the specified waitee job are done.  wait_for operates on the job
	      name  basis,  so	there are a few	limitations. First, the	waitee
	      must be defined prior to the waiter job (meaning no forward ref-
	      erences).	 Second,  if a job is being referenced as a waitee, it
	      must have	a unique name (no duplicate waitees).

	      Run the job with the given nice value. See man nice(2).  On Win-
	      dows,  values  less than -15 set the process class to "High"; -1
	      through -15 set "Above Normal"; 1	through	15 "Below Normal"; and
	      above 15 "Idle" priority class.

	      Set  the	I/O  priority  value of	this job. Linux	limits us to a
	      positive value between 0 and 7, with 0 being  the	 highest.  See
	      man ionice(1). Refer to an appropriate manpage for other operat-
	      ing systems since	meaning	of priority may	differ.	 For  per-com-
	      mand priority setting, see I/O engine specific `cmdprio_percent-
	      age` and `hipri_percentage` options.

	      Set the I/O priority class. See man ionice(1).  For  per-command
	      priority	setting,  see I/O engine specific `cmdprio_percentage`
	      and `hipri_percent` options.

	      Controls the same	options	as  cpumask,  but  accepts  a  textual
	      specification of the permitted CPUs instead and CPUs are indexed
	      from 0. So to use	CPUs 0	and  5	you  would  specify  `cpus_al-
	      lowed=0,5'. This option also allows a range of CPUs to be	speci-
	      fied -- say you wanted a binding to CPUs 0, 5, and 8 to 15,  you
	      would set	`cpus_allowed=0,5,8-15'.

	      On  Windows,  when  `cpus_allowed' is unset only CPUs from fio's
	      current processor	group will be used and affinity	 settings  are
	      inherited	 from  the  system.  An	fio build configured to	target
	      Windows 7	makes options that set CPUs processor group aware  and
	      values  will  set	both the processor group and a CPU from	within
	      that group. For example, on a system where processor group 0 has
	      40 CPUs and processor group 1 has	32 CPUs, `cpus_allowed'	values
	      between 0	and 39 will bind  CPUs	from  processor	 group	0  and
	      `cpus_allowed' values between 40 and 71 will bind	CPUs from pro-
	      cessor group 1. When using `cpus_allowed_policy=shared' all CPUs
	      specified	 by  a	single	`cpus_allowed' option must be from the
	      same processor group. For	Windows	fio builds not built for  Win-
	      dows  7,	CPUs  will  only be selected from (and be relative to)
	      whatever processor group fio happens to be running in  and  CPUs
	      from other processor groups cannot be used.

	      Set  the	policy	of  how	 fio distributes the CPUs specified by
	      cpus_allowed or cpumask. Two policies are	supported:

		     shared All	jobs will share	the CPU	set specified.

		     split  Each job will get a	unique CPU from	the CPU	set.

	      shared is	the default behavior, if the option  isn't  specified.
	      If  split	is specified, then fio will assign one cpu per job. If
	      not enough CPUs are given	for the	jobs  listed,  then  fio  will
	      roundrobin the CPUs in the set.

	      Set  the	CPU affinity of	this job. The parameter	given is a bit
	      mask of allowed CPUs the job may run on. So if you want the  al-
	      lowed CPUs to be 1 and 5,	you would pass the decimal value of (1
	      << 1 | 1 << 5), or 34. See man  sched_setaffinity(2).  This  may
	      not  work	on all supported operating systems or kernel versions.
	      This option doesn't work well for	a higher CPU count  than  what
	      you  can	store  in an integer mask, so it can only control cpus
	      1-32. For	boxes with larger CPU counts, use cpus_allowed.

	      Set this job running on specified	NUMA nodes'  CPUs.  The	 argu-
	      ments  allow comma delimited list	of cpu numbers,	A-B ranges, or
	      `all'. Note, to enable NUMA options support, fio must  be	 built
	      on a system with libnuma-dev(el) installed.

	      Set  this	job's memory policy and	corresponding NUMA nodes. For-
	      mat of the arguments:


	      `mode' is	one of the following memory policies: `default', `pre-
	      fer', `bind', `interleave' or `local'. For `default' and `local'
	      memory policies, no node needs to	be  specified.	For  `prefer',
	      only one node is allowed.	For `bind' and `interleave' the	`node-
	      list' may	be as follows: a comma delimited list of numbers,  A-B
	      ranges, or `all'.

	      Add  job	to this	control	group. If it doesn't exist, it will be
	      created. The system must have a mounted cgroup blkio mount point
	      for  this	 to  work. If your system doesn't have it mounted, you
	      can do so	with:

		     # mount -t	cgroup -o blkio	none /cgroup

	      Set the weight of	the cgroup to this value. See  the  documenta-
	      tion that	comes with the kernel, allowed values are in the range
	      of 100..1000.

	      Normally fio will	delete the cgroups it has  created  after  the
	      job  completion.	To override this behavior and to leave cgroups
	      around after the job completion, set  `cgroup_nodelete=1'.  This
	      can be useful if one wants to inspect various cgroup files after
	      job completion. Default: false.

	      The ID of	the flow. If not specified, it	defaults  to  being  a
	      global flow. See flow.

	      Weight  in token-based flow control. If this value is used, then
	      fio regulates the	activity between two or	more jobs sharing  the
	      same  flow_id.   Fio  attempts to	keep each job activity propor-
	      tional to	other jobs' activities in the same flow_id group, with
	      respect  to  requested  weight per job.  That is,	if one job has
	      `flow=3',	another	job has	`flow=2' and  another  with  `flow=1`,
	      then there will be a roughly 3:2:1 ratio in how much one runs vs
	      the others.

	      The period of time, in microseconds,  to	wait  after  the  flow
	      counter has exceeded its proportion before retrying operations.

       stonewall, wait_for_previous
	      Wait for preceding jobs in the job file to exit, before starting
	      this one.	Can be used to insert serialization points in the  job
	      file.  A stone wall also implies starting	a new reporting	group,
	      see group_reporting. Optionally you  can	use  `stonewall=0`  to
	      disable or `stonewall=1` to enable it.

	      By  default,  fio	 will continue running all other jobs when one
	      job finishes.  Sometimes this is not the desired action. Setting
	      exitall  will  instead  make  fio	terminate all jobs in the same
	      group, as	soon as	one job	of that	group finishes.

	      By default, fio will continue running all	other  jobs  when  one
	      job finishes.  Sometimes this is not the desired action. Setting
	      exitall will instead make	fio terminate all  jobs	 in  the  same
	      group. The option	exit_what allows you to	control	which jobs get
	      terminated when exitall is enabled.  The default value is	group.
	      The allowed values are:

		     all    terminates all jobs.

		     group  is	the  default and does not change the behaviour
			    of exitall.

			    terminates all currently running jobs  across  all
			    groups  and	 continues  execution  with  the  next
			    stonewalled	group.

	      Before running this job, issue  the  command  specified  through
	      system(3).  Output  is redirected	in a file called `jobname.pre-

	      After the	job completes, issue the command specified though sys-
	      tem(3).	Output	 is   redirected   in	a  file	 called	 `job-

	      Instead of running as the	invoking user, set the user ID to this
	      value before the thread/process does any work.

	      Set group	ID, see	uid.

	      Do  not  perform	specified  workload,  only  verify  data still
	      matches previous invocation of this workload. This option	allows
	      one  to  check data multiple times at a later date without over-
	      writing it. This option makes  sense  only  for  workloads  that
	      write  data,  and	does not support workloads with	the time_based
	      option set.

	      Run the verify phase after a write phase.	Only valid  if	verify
	      is set. Default: true.

	      If  writing  to  a  file,	fio can	verify the file	contents after
	      each iteration of	the job. Each verification method also implies
	      verification  of	special	header,	which is written to the	begin-
	      ning of each block. This header also includes meta  information,
	      like offset of the block,	block number, timestamp	when block was
	      written, etc. verify can be combined with	verify_pattern option.
	      The allowed values are:

		     md5    Use	 an  md5  sum of the data area and store it in
			    the	header of each block.

		     crc64  Use	an experimental	crc64 sum of the data area and
			    store it in	the header of each block.

		     crc32c Use	 a crc32c sum of the data area and store it in
			    the	header of each block. This will	 automatically
			    use	 hardware  acceleration	(e.g. SSE4.2 on	an x86
			    or CRC crypto extensions on	ARM64) but  will  fall
			    back  to  software crc32c if none is found.	Gener-
			    ally the fastest checksum fio supports when	 hard-
			    ware accelerated.

			    Synonym for	crc32c.

		     crc32  Use	 a  crc32 sum of the data area and store it in
			    the	header of each block.

		     crc16  Use	a crc16	sum of the data	area and store	it  in
			    the	header of each block.

		     crc7   Use	 a  crc7  sum of the data area and store it in
			    the	header of each block.

		     xxhash Use	xxhash as the checksum function. Generally the
			    fastest software checksum that fio supports.

		     sha512 Use	sha512 as the checksum function.

		     sha256 Use	sha256 as the checksum function.

		     sha1   Use	optimized sha1 as the checksum function.

			    Use	optimized sha3-224 as the checksum function.

			    Use	optimized sha3-256 as the checksum function.

			    Use	optimized sha3-384 as the checksum function.

			    Use	optimized sha3-512 as the checksum function.

		     meta   This option	is deprecated, since now meta informa-
			    tion is included in	 generic  verification	header
			    and	 meta verification happens by default. For de-
			    tailed information see the description of the ver-
			    ify	 setting.  This	option is kept because of com-
			    patibility's sake with old configurations. Do  not
			    use	it.

			    Verify  a  strict pattern. Normally	fio includes a
			    header with	some basic information	and  checksum-
			    ming, but if this option is	set, only the specific
			    pattern set	with verify_pattern is verified.

		     null   Only pretend to verify. Useful for testing	inter-
			    nals with `ioengine=null', not for much else.

	      This  option  can	be used	for repeated burn-in tests of a	system
	      to make sure that	the written data is also correctly read	 back.
	      If  the  data direction given is a read or random	read, fio will
	      assume that it should verify a previously	written	file.  If  the
	      data direction includes any form of write, the verify will be of
	      the newly	written	data.

	      To avoid false verification errors, do not use  the  norandommap
	      option when verifying data with async I/O	engines	and I/O	depths
	      >	1.  Or use the norandommap and the lfsr	random	generator  to-
	      gether  to  avoid	 writing  to the same offset with muliple out-
	      standing I/Os.

	      Swap the verification header with	data  somewhere	 else  in  the
	      block before writing. It is swapped back before verifying.

	      Write  the  verification	header at a finer granularity than the
	      blocksize. It will be written for	chunks the size	of  verify_in-
	      terval. blocksize	should divide this evenly.

	      If set, fio will fill the	I/O buffers with this pattern. Fio de-
	      faults to	filling	with totally random bytes, but sometimes  it's
	      interesting  to  fill  with a known pattern for I/O verification
	      purposes.	Depending on the width of the pattern, fio  will  fill
	      1/2/3/4 bytes of the buffer at the time (it can be either	a dec-
	      imal or a	hex number).  The  verify_pattern  if  larger  than  a
	      32-bit  quantity	has to be a hex	number that starts with	either
	      "0x" or "0X". Use	with verify. Also, verify_pattern supports  %o
	      format,  which  means that for each block	offset will be written
	      and then verified	back, e.g.:


	      Or use combination of everything:


	      Normally fio will	keep checking the entire contents before quit-
	      ting on a	block verification failure. If this option is set, fio
	      will exit	the job	on the first observed failure. Default:	false.

	      If set, dump the contents	of both	the original  data  block  and
	      the  data	 block	we  read  off disk to files. This allows later
	      analysis to inspect just what kind of data corruption  occurred.
	      Off by default.

	      Fio  will	normally verify	I/O inline from	the submitting thread.
	      This option takes	an integer describing how many	async  offload
	      threads  to  create for I/O verification instead,	causing	fio to
	      offload the duty of verifying I/O	contents to one	or more	 sepa-
	      rate  threads.  If  using	this offload option, even sync I/O en-
	      gines can	benefit	from using an iodepth setting higher  than  1,
	      as  it allows them to have I/O in	flight while verifies are run-
	      ning.  Defaults to 0 async threads,  i.e.	 verification  is  not

	      Tell  fio	to set the given CPU affinity on the async I/O verifi-
	      cation threads. See cpus_allowed for the format used.

	      Fio will normally	verify the written contents of a job that uti-
	      lizes verify once	that job has completed.	In other words,	every-
	      thing is written then everything is read back and	verified.  You
	      may want to verify continually instead for a variety of reasons.
	      Fio stores the meta data associated with an I/O block in memory,
	      so  for  large  verify workloads,	quite a	bit of memory would be
	      used up holding this meta	data. If this option is	 enabled,  fio
	      will write only N	blocks before verifying	these blocks.

	      Control  how  many  blocks  fio will verify if verify_backlog is
	      set. If not set, will default to	the  value  of	verify_backlog
	      (meaning	the  entire  queue is read back	and verified). If ver-
	      ify_backlog_batch	is  less  than	verify_backlog	then  not  all
	      blocks  will be verified,	if verify_backlog_batch	is larger than
	      verify_backlog, some blocks will be verified more	than once.

	      When a job exits during the write	phase of  a  verify  workload,
	      save  its	current	state. This allows fio to replay up until that
	      point, if	the verify state is loaded for the verify read	phase.
	      The format of the	filename is, roughly:


	      <type>  is  "local"  for a local run, "sock" for a client/server
	      socket connection, and "ip" (,	for  instance)	for  a
	      networked	client/server connection. Defaults to true.

	      If a verify termination trigger was used,	fio stores the current
	      write state of each thread. This can  be	used  at  verification
	      time  so	that  fio knows	how far	it should verify. Without this
	      information, fio will run	a full verification pass, according to
	      the settings in the job file used. Default false.

	      Number of	verify blocks to discard/trim.

	      Verify that trim/discarded blocks	are returned as	zeros.

	      Verify that trim/discarded blocks	are returned as	zeros.

	      Trim this	number of I/O blocks.

	      Enable experimental verification.

   Steady state
       steadystate=str:float, ss=str:float
	      Define  the  criterion and limit for assessing steady state per-
	      formance.	The first parameter designates the  criterion  whereas
	      the  second  parameter  sets  the	 threshold. When the criterion
	      falls below the threshold	for the	specified  duration,  the  job
	      will  stop.  For	example,  `iops_slope:0.1%' will direct	fio to
	      terminate	the job	when the least squares regression slope	 falls
	      below  0.1% of the mean IOPS. If group_reporting is enabled this
	      will apply to all	jobs in	the group. Below is the	list of	avail-
	      able  steady state assessment criteria. All assessments are car-
	      ried out using only data from  the  rolling  collection  window.
	      Threshold	 limits	can be expressed as a fixed value or as	a per-
	      centage of the mean in the collection window.

	      When using this feature, most jobs should	include	the time_based
	      and  runtime  options  or	 the loops option so that fio does not
	      stop running after it has	covered	the full size of the specified
	      file(s) or device(s).

			    iops   Collect  IOPS data. Stop the	job if all in-
				   dividual IOPS measurements are  within  the
				   specified  limit  of	 the  mean IOPS	(e.g.,
				   `iops:2' means  that	 all  individual  IOPS
				   values  must	 be  within  2	of  the	 mean,
				   whereas `iops:0.2%' means that all individ-
				   ual	IOPS values must be within 0.2%	of the
				   mean	IOPS to	terminate the job).

				   Collect IOPS	data and calculate  the	 least
				   squares  regression	slope. Stop the	job if
				   the slope falls below the specified limit.

			    bw	   Collect bandwidth data. Stop	the job	if all
				   individual	bandwidth   measurements   are
				   within the  specified  limit	 of  the  mean

				   Collect  bandwidth  data  and calculate the
				   least squares regression  slope.  Stop  the
				   job	if the slope falls below the specified

	      steadystate_duration=time, ss_dur=time
		     A rolling window of this duration will be used  to	 judge
		     whether  steady state has been reached. Data will be col-
		     lected once per second. The default is 0  which  disables
		     steady  state  detection.	When  the unit is omitted, the
		     value is interpreted in seconds.

	      steadystate_ramp_time=time, ss_ramp=time
		     Allow the job to run for the  specified  duration	before
		     beginning	data  collection for checking the steady state
		     job termination criterion.	The default  is	 0.  When  the
		     unit is omitted, the value	is interpreted in seconds.

   Measurements	and reporting
	      If  set,	this  generates	bw/clat/iops log with per file private
	      filenames. If not	set, jobs with identical names will share  the
	      log filename. Default: true.

	      It may sometimes be interesting to display statistics for	groups
	      of jobs as a whole instead of for	each individual	job.  This  is
	      especially  true	if  numjobs  is	 used;	looking	 at individual
	      thread/process output quickly becomes unwieldy. To see the final
	      report  per-group	 instead of per-job, use group_reporting. Jobs
	      in a file	will be	part of	the same reporting  group,  unless  if
	      separated	by a stonewall,	or by using new_group.

	      Start a new reporting group. See:	group_reporting. If not	given,
	      all jobs in a file will be part of the same reporting group, un-
	      less separated by	a stonewall.

	      By  default, fio collects	and shows final	output results for all
	      jobs that	run. If	this option is set to 0, then fio will	ignore
	      it in the	final stat output.

	      If  given,  write	 a  bandwidth log for this job.	Can be used to
	      store data of the	bandwidth of the jobs in their lifetime.

	      If no str	argument is  given,  the  default  filename  of	 `job-
	      name_type.x.log'	is  used. Even when the	argument is given, fio
	      will still append	the type of log. So if one specifies:


	      The actual log name will be `foo_bw.x.log' where `x' is the  in-
	      dex  of  the  job	 (1..N,	 where	N  is  the number of jobs). If
	      per_job_logs is false, then the filename will  not  include  the
	      `.x` job index.

	      The  included  fio_generate_plots	 script	 uses  gnuplot to turn
	      these text files into nice graphs. See the LOG FILE FORMATS sec-
	      tion for how data	is structured within the file.

	      Same  as write_bw_log, except this option	creates	I/O submission
	      (e.g., `name_slat.x.log'), completion (e.g., `name_clat.x.log'),
	      and  total  (e.g.,  `name_lat.x.log') latency files instead. See
	      write_bw_log for details about the filename format and  the  LOG
	      FILE  FORMATS  section  for  how	data  is structured within the

	      Same as write_bw_log but writes an I/O completion	 latency  his-
	      togram  file  (e.g.,  `name_hist.x.log') instead.	Note that this
	      file will	be empty unless	log_hist_msec has also been set.   See
	      write_bw_log  for	 details about the filename format and the LOG
	      FILE FORMATS section for how data	is structured within the file.

	      Same  as	write_bw_log,  but   writes   an   IOPS	  file	 (e.g.
	      `name_iops.x.log`)  instead.  Because fio	defaults to individual
	      I/O logging, the value entry in the IOPS log will	 be  1	unless
	      windowed	logging	 (see  log_avg_msec)  has  been	 enabled.  See
	      write_bw_log for details about the filename format and LOG  FILE
	      FORMATS for how data is structured within	the file.

	      By  default,  fio	 will log an entry in the iops,	latency, or bw
	      log for every I/O	that completes.	When writing to	the disk  log,
	      that  can	quickly	grow to	a very large size. Setting this	option
	      makes fio	average	the each log entry over	the  specified	period
	      of  time,	 reducing the resolution of the	log. See log_max_value
	      as well. Defaults	to 0, logging all entries.  Also see LOG  FILE
	      FORMATS section.

	      Same  as	log_avg_msec,  but logs	entries	for completion latency
	      histograms. Computing latency percentiles	from averages  of  in-
	      tervals  using  log_avg_msec  is inaccurate. Setting this	option
	      makes fio	log histogram entries over  the	 specified  period  of
	      time,  reducing  log sizes for high IOPS devices while retaining
	      percentile accuracy. See log_hist_coarseness and	write_hist_log
	      as well.	Defaults to 0, meaning histogram logging is disabled.

	      Integer ranging from 0 to	6, defining the	coarseness of the res-
	      olution of the histogram logs enabled  with  log_hist_msec.  For
	      each increment in	coarseness, fio	outputs	half as	many bins. De-
	      faults to	0, for which histogram logs contain 1216 latency bins.
	      See LOG FILE FORMATS section.

	      If  log_avg_msec	is set,	fio logs the average over that window.
	      If you instead want to log the maximum value, set	this option to
	      1. Defaults to 0,	meaning	that averaged values are logged.

	      If  this	is set,	the iolog options will include the byte	offset
	      for the I/O entry	as well	as the other data values. Defaults  to
	      0	 meaning  that	offsets	 are not present in logs. Also see LOG
	      FILE FORMATS section.

	      If this is set, fio will compress	the I/O	logs as	 it  goes,  to
	      keep  the	 memory	footprint lower. When a	log reaches the	speci-
	      fied size, that chunk is removed and  compressed	in  the	 back-
	      ground. Given that I/O logs are fairly highly compressible, this
	      yields a nice memory savings for longer runs.  The  downside  is
	      that the compression will	consume	some background	CPU cycles, so
	      it may impact the	run. This, however, is also true if  the  log-
	      ging  ends  up consuming most of the system memory. So pick your
	      poison. The I/O logs are saved normally at the end of a run,  by
	      decompressing  the  chunks and storing them in the specified log
	      file. This feature depends on the	availability of	zlib.

	      Define the set of	CPUs that are allowed  to  handle  online  log
	      compression  for the I/O jobs. This can provide better isolation
	      between performance sensitive jobs, and  background  compression
	      work. See	cpus_allowed for the format used.

	      If  set,	fio  will  store the log files in a compressed format.
	      They can be decompressed with fio, using the --inflate-log  com-
	      mand  line parameter. The	files will be stored with a `.fz' suf-

	      If set, fio will log Unix	timestamps to the log  files  produced
	      by enabling write_type_log for each log type, instead of the de-
	      fault zero-based timestamps.

	      If set, record errors in trim block-sized	units from writes  and
	      trims and	output a histogram of how many trims it	took to	get to
	      errors, and what kind of error was encountered.

	      Average the calculated bandwidth over the	given time.  Value  is
	      specified	 in  milliseconds. If the job also does	bandwidth log-
	      ging through write_bw_log, then the minimum of this  option  and
	      log_avg_msec will	be used. Default: 500ms.

	      Average the calculated IOPS over the given time. Value is	speci-
	      fied in milliseconds. If the job also does IOPS logging  through
	      write_iops_log, then the minimum of this option and log_avg_msec
	      will be used. Default: 500ms.

	      Generate disk utilization	statistics, if the  platform  supports
	      it.  Default: true.

	      Disable  measurements  of	total latency numbers. Useful only for
	      cutting back the number of calls	to  gettimeofday(2),  as  that
	      does  impact performance at really high IOPS rates. Note that to
	      really get rid of	a large	amount of  these  calls,  this	option
	      must  be	used  with  disable_slat and disable_bw_measurement as

	      Disable measurements of completion  latency  numbers.  See  dis-

	      Disable  measurements  of	 submission  latency numbers. See dis-

       disable_bw_measurement=bool, disable_bw=bool
	      Disable measurements of throughput/bandwidth numbers.  See  dis-

	      Report submission	latency	percentiles. Submission	latency	is not
	      recorded for synchronous ioengines.

	      Report completion	latency	percentiles.

	      Report total latency percentiles.	Total latency is  the  sum  of
	      submission latency and completion	latency.

	      Overwrite	 the default list of percentiles for latencies and the
	      block error histogram. Each number is a floating point number in
	      the range	(0,100], and the maximum length	of the list is 20. Use
	      ':'   to	 separate   the	  numbers.   For   example,    `--per-
	      centile_list=99.5:99.9' will cause fio to	report the latency du-
	      rations below which 99.5%	and 99.9% of  the  observed  latencies
	      fell, respectively.

	      If  using	 --output-format of `normal', set the significant fig-
	      ures to this value. Higher values	will yield more	 precise  IOPS
	      and  throughput units, while lower values	will round. Requires a
	      minimum value of 1 and a maximum value of	10. Defaults to	4.

   Error handling
	      When one job finishes in error, terminate	the rest. The  default
	      is to wait for each job to finish.

	      Normally fio will	exit the job on	the first observed failure. If
	      this option is set, fio will continue the	job when  there	 is  a
	      'non-fatal  error' (EIO or EILSEQ) until the runtime is exceeded
	      or the I/O size specified	is completed. If this option is	 used,
	      there  are  two  more  stats  that are appended, the total error
	      count and	the first error. The error field given in the stats is
	      the first	error that was hit during the run.  The	allowed	values

		     none   Exit on any	I/O or verify errors.

		     read   Continue on	read errors, exit on all others.

		     write  Continue on	write errors, exit on all others.

		     io	    Continue on	any I/O	error, exit on all others.

		     verify Continue on	verify errors, exit on all others.

		     all    Continue on	all errors.

		     0	    Backward-compatible	alias for 'none'.

		     1	    Backward-compatible	alias for 'all'.

	      Sometimes	you want to ignore some	errors	during	test  in  that
	      case  you	can specify error list for each	error type, instead of
	      only being able to ignore	the default  'non-fatal	 error'	 using
	      continue_on_error.				   `ignore_er-
	      given  error  type  is  separated	 with ':'. Error may be	symbol
	      ('ENOSPC', 'ENOMEM') or integer. Example:


	      This option  will	 ignore	 EAGAIN	 from  READ,  and  ENOSPC  and
	      122(EDQUOT)  from	 WRITE.	 This  option works by overriding con-
	      tinue_on_error with the list of errors for each  error  type  if

	      If  set  dump  every  error even if it is	non fatal, true	by de-
	      fault. If	disabled only fatal error will be dumped.

   Running predefined workloads
       Fio includes predefined profiles	that mimic the I/O workloads generated
       by other	tools.

	      The predefined workload to run. Current profiles are:

			    Threaded  I/O  bench (tiotest/tiobench) like work-

		     act    Aerospike Certification Tool (ACT) like workload.

       To view a profile's additional options use --cmdhelp  after  specifying
       the profile. For	example:

	      $	fio --profile=act --cmdhelp

   Act profile options
	      Devices to use.

	      ACT load multiplier. Default: 1.

	      How long the entire test takes to	run. When the unit is omitted,
	      the value	is given in seconds. Default: 24h.

	      Number of	read I/O threads per device. Default: 8.

	      Number of	512B blocks to read at the time. Default: 3.

	      Size of large block ops in KiB (writes). Default:	131072.

       prep   Set to run ACT prep phase.

   Tiobench profile options
	      Size in MiB.

	      Block size in bytes. Default: 4096.

	      Number of	runs.

	      Test directory.

	      Number of	threads.

       Fio spits out a lot of output. While running, fio will display the sta-
       tus of the jobs created.	An example of that would be:

		 Jobs: 1 (f=1):	[_(1),M(1)][24.8%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta	01m:31s]

       The  characters inside the first	set of square brackets denote the cur-
       rent status of each thread. The first character is the  first  job  de-
       fined  in  the  job file, and so	forth. The possible values (in typical
       life cycle order) are:

	      P	     Thread setup, but not started.
	      C	     Thread created.
	      I	     Thread initialized, waiting or generating necessary data.
	      p	     Thread running pre-reading	file(s).
	      /	     Thread is in ramp period.
	      R	     Running, doing sequential reads.
	      r	     Running, doing random reads.
	      W	     Running, doing sequential writes.
	      w	     Running, doing random writes.
	      M	     Running, doing mixed sequential reads/writes.
	      m	     Running, doing mixed random reads/writes.
	      D	     Running, doing sequential trims.
	      d	     Running, doing random trims.
	      F	     Running, currently	waiting	for fsync(2).
	      V	     Running, doing verification of written data.
	      f	     Thread finishing.
	      E	     Thread exited, not	reaped by main thread yet.
	      -	     Thread reaped.
	      X	     Thread reaped, exited with	an error.
	      K	     Thread reaped, exited due to signal.

       Fio will	condense the thread string as not to take up more space	on the
       command	line  than needed. For instance, if you	have 10	readers	and 10
       writers running,	the output would look like this:

		 Jobs: 20 (f=20): [R(10),W(10)][4.0%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 57m:36s]

       Note that the status string is displayed	in order, so it's possible  to
       tell  which  of the jobs	are currently doing what. In the example above
       this means that jobs 1--10 are readers and 11--20 are writers.

       The other values	are fairly self	explanatory -- number of threads  cur-
       rently  running and doing I/O, the number of currently open files (f=),
       the estimated completion	percentage, the	rate of	I/O since  last	 check
       (read  speed  listed first, then	write speed and	optionally trim	speed)
       in terms	of bandwidth and IOPS, and time	to completion for the  current
       running	group.	It's  impossible  to estimate runtime of the following
       groups (if any).

       When fio	is done	(or interrupted	by Ctrl-C), it will show the data  for
       each  thread, group of threads, and disks in that order.	For each over-
       all thread (or group) the output	looks like:

		 Client1: (groupid=0, jobs=1): err= 0: pid=16109: Sat Jun 24 12:07:54 2017
		   write: IOPS=88, BW=623KiB/s (638kB/s)(30.4MiB/50032msec)
		     slat (nsec): min=500, max=145500, avg=8318.00, stdev=4781.50
		     clat (usec): min=170, max=78367, avg=4019.02, stdev=8293.31
		      lat (usec): min=174, max=78375, avg=4027.34, stdev=8291.79
		     clat percentiles (usec):
		      |	 1.00th=[  302],  5.00th=[  326], 10.00th=[  343], 20.00th=[  363],
		      |	30.00th=[  392], 40.00th=[  404], 50.00th=[  416], 60.00th=[  445],
		      |	70.00th=[  816], 80.00th=[ 6718], 90.00th=[12911], 95.00th=[21627],
		      |	99.00th=[43779], 99.50th=[51643], 99.90th=[68682], 99.95th=[72877],
		      |	99.99th=[78119]
		    bw (  KiB/s): min=	532, max=  686,	per=0.10%, avg=622.87, stdev=24.82, samples=  100
		    iops	: min=	 76, max=   98,	avg=88.98, stdev= 3.54,	samples=  100
		   lat (usec)	: 250=0.04%, 500=64.11%, 750=4.81%, 1000=2.79%
		   lat (msec)	: 2=4.16%, 4=1.84%, 10=4.90%, 20=11.33%, 50=5.37%
		   lat (msec)	: 100=0.65%
		   cpu		: usr=0.27%, sys=0.18%,	ctx=12072, majf=0, minf=21
		   IO depths	: 1=85.0%, 2=13.1%, 4=1.8%, 8=0.1%, 16=0.0%, 32=0.0%, >=64=0.0%
		      submit	: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
		      complete	: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
		      issued rwt: total=0,4450,0, short=0,0,0, dropped=0,0,0
		      latency	: target=0, window=0, percentile=100.00%, depth=8

       The job name (or	 first	job's  name  when  using  group_reporting)  is
       printed,	 along with the	group id, count	of jobs	being aggregated, last
       error id	seen (which is 0 when there are	no errors),  pid/tid  of  that
       thread  and the time the	job/group completed. Below are the I/O statis-
       tics for	each data direction performed (showing writes in  the  example
       above). In the order listed, they denote:

		     The  string  before the colon shows the I/O direction the
		     statistics	are for. IOPS is the  average  I/Os  performed
		     per  second.  BW  is the average bandwidth	rate shown as:
		     value in power of 2 format	(value in power	of 10 format).
		     The  last	two values show: (total	I/O performed in power
		     of	2 format / runtime of that thread).

	      slat   Submission	latency	(min being the minimum,	max being  the
		     maximum,  avg being the average, stdev being the standard
		     deviation). This is the time it took to submit  the  I/O.
		     For sync I/O this row is not displayed as the slat	is re-
		     ally the completion latency (since	queue/complete is  one
		     operation	there).	 This value can	be in nanoseconds, mi-
		     croseconds	or milliseconds	--- fio	will choose  the  most
		     appropriate  base	and  print  that (in the example above
		     nanoseconds was the best scale). Note: in --minimal  mode
		     latencies are always expressed in microseconds.

	      clat   Completion	 latency. Same names as	slat, this denotes the
		     time from submission to completion	of the I/O pieces. For
		     sync  I/O,	 clat will usually be equal (or	very close) to
		     0,	as the time from submit	to complete is basically  just
		     CPU  time	(I/O  has already been done, see slat explana-

	      lat    Total latency. Same names as slat and clat, this  denotes
		     the time from when	fio created the	I/O unit to completion
		     of	the I/O	operation.

	      bw     Bandwidth statistics based	on samples. Same names as  the
		     xlat stats, but also includes the number of samples taken
		     (samples) and an approximate percentage of	 total	aggre-
		     gate  bandwidth  this thread received in its group	(per).
		     This last value is	only really useful if the  threads  in
		     this group	are on the same	disk, since they are then com-
		     peting for	disk access.

	      iops   IOPS statistics based on samples. Same names as bw.

	      lat (nsec/usec/msec)
		     The distribution of I/O completion	latencies. This	is the
		     time from when I/O	leaves fio and when it gets completed.
		     Unlike the	separate read/write/trim sections  above,  the
		     data here and in the remaining sections apply to all I/Os
		     for the reporting group. 250=0.04%	means  that  0.04%  of
		     the  I/Os completed in under 250us. 500=64.11% means that
		     64.11% of the I/Os	required 250 to	499us for completion.

	      cpu    CPU usage.	User and system	time, along with the number of
		     context  switches this thread went	through, usage of sys-
		     tem and user time,	and finally the	number	of  major  and
		     minor  page faults. The CPU utilization numbers are aver-
		     ages for the jobs in that reporting group,	while the con-
		     text and fault counters are summed.

	      IO depths
		     The distribution of I/O depths over the job lifetime. The
		     numbers are divided into powers of	2 and each entry  cov-
		     ers  depths  from	that  value up to those	that are lower
		     than the next entry -- e.g., 16= covers depths from 16 to
		     31.  Note	that the range covered by a depth distribution
		     entry can be different to the range covered by the	equiv-
		     alent submit/complete distribution	entry.

	      IO submit
		     How many pieces of	I/O were submitting in a single	submit
		     call. Each	entry denotes that amount and below, until the
		     previous  entry  -- e.g., 16=100% means that we submitted
		     anywhere between 9	to 16 I/Os per submit call. Note  that
		     the  range	 covered by a submit distribution entry	can be
		     different to the range covered by	the  equivalent	 depth
		     distribution entry.

	      IO complete
		     Like  the	above  submit  number, but for completions in-

	      IO issued	rwt
		     The number	of read/write/trim requests  issued,  and  how
		     many of them were short or	dropped.

	      IO latency
		     These  values are for latency_target and related options.
		     When these	options	are engaged,  this  section  describes
		     the I/O depth required to meet the	specified latency tar-

       After each client has been listed, the group  statistics	 are  printed.
       They will look like this:

		 Run status group 0 (all jobs):
		    READ: bw=20.9MiB/s (21.9MB/s), 10.4MiB/s-10.8MiB/s (10.9MB/s-11.3MB/s), io=64.0MiB (67.1MB), run=2973-3069msec
		   WRITE: bw=1231KiB/s (1261kB/s), 616KiB/s-621KiB/s (630kB/s-636kB/s),	io=64.0MiB (67.1MB), run=52747-53223msec

       For each	data direction it prints:

	      bw     Aggregate	bandwidth of threads in	this group followed by
		     the minimum and maximum bandwidth of all the  threads  in
		     this  group.   Values  outside of brackets	are power-of-2
		     format and	those within are the  equivalent  value	 in  a
		     power-of-10 format.

	      io     Aggregate I/O performed of	all threads in this group. The
		     format is the same	as bw.

	      run    The smallest and longest runtimes of the threads in  this

       And  finally,  the disk statistics are printed. This is Linux specific.
       They will look like this:

		   Disk	stats (read/write):
		     sda: ios=16398/16511, merge=30/162, ticks=6853/819634, in_queue=826487, util=100.00%

       Each value is printed for both reads and	writes,	with reads first.  The
       numbers denote:

	      ios    Number of I/Os performed by all groups.

	      merge  Number of merges performed	by the I/O scheduler.

	      ticks  Number of ticks we	kept the disk busy.

		     Total time	spent in the disk queue.

	      util   The  disk	utilization. A value of	100% means we kept the
		     disk busy constantly, 50% would be	a disk idling half  of
		     the time.

       It  is  also possible to	get fio	to dump	the current output while it is
       running,	without	terminating the	job. To	do that,  send	fio  the  USR1
       signal.	You  can  also	get  regularly timed dumps by using the	--sta-
       tus-interval  parameter,	 or  by	 creating  a  file  in	`/tmp'	 named
       `fio-dump-status'.  If  fio  sees this file, it will unlink it and dump
       the current output status.

       For scripted usage where	you  typically	want  to  generate  tables  or
       graphs  of the results, fio can output the results in a semicolon sepa-
       rated format. The format	is one long line of values, such as:

		 A description of this job goes	here.

       The job description (if provided) follows on a second  line  for	 terse
       v2.  It appears on the same line	for other terse	versions.

       To  enable  terse  output, use the --minimal or `--output-format=terse'
       command line options. The first value is	the version of the terse  out-
       put  format. If the output has to be changed for	some reason, this num-
       ber will	be incremented by 1 to signify that change.

       Split up, the format is as follows (comments in brackets	denote when  a
       field was introduced or whether it's specific to	some terse version):

		      terse version, fio version [v3], jobname,	groupid, error

	      READ status:

		      Total IO (KiB), bandwidth	(KiB/sec), IOPS, runtime (msec)
		      Submission latency: min, max, mean, stdev	(usec)
		      Completion latency: min, max, mean, stdev	(usec)
		      Completion latency percentiles: 20 fields	(see below)
		      Total latency: min, max, mean, stdev (usec)
		      Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev,	number of samples [v5]
		      IOPS [v5]: min, max, mean, stdev,	number of samples

	      WRITE status:

		      Total IO (KiB), bandwidth	(KiB/sec), IOPS, runtime (msec)
		      Submission latency: min, max, mean, stdev	(usec)
		      Completion latency: min, max, mean, stdev	(usec)
		      Completion latency percentiles: 20 fields	(see below)
		      Total latency: min, max, mean, stdev (usec)
		      Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev,	number of samples [v5]
		      IOPS [v5]: min, max, mean, stdev,	number of samples

	      TRIM status [all but version 3]:

		      Fields are similar to READ/WRITE status.

	      CPU usage:

		      user, system, context switches, major faults, minor faults

	      I/O depths:

		      <=1, 2, 4, 8, 16,	32, >=64

	      I/O latencies microseconds:

		      <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000

	      I/O latencies milliseconds:

		      <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000

	      Disk utilization [v3]:

		      disk name, read ios, write ios, read merges, write merges, read ticks, write ticks, time spent in	queue, disk utilization	percentage

	      Additional Info (dependent on continue_on_error, default off):

		      total # errors, first error code

	      Additional Info (dependent on description	being set):

		      Text description

       Completion  latency  percentiles	can be a grouping of up	to 20 sets, so
       for the terse output fio	writes all of them. Each field will look  like


       which is	the Xth	percentile, and	the `usec' latency associated with it.

       For Disk	utilization, all disks used by fio are shown. So for each disk
       there will be a disk utilization	section.

       Below is	a single line containing short names for each of the fields in
       the minimal output v3, separated	by semicolons:


       In  client/server mode terse output differs from	what appears when jobs
       are run locally.	Disk utilization data is  omitted  from	 the  standard
       terse  output  and for v3 and later appears on its own separate line at
       the end of each terse reporting cycle.

       The json	output format is intended to be	both human readable and	conve-
       nient  for  automated  parsing.	For  the most part its sections	mirror
       those of	the normal output. The runtime value is	reported in  msec  and
       the bw value is reported	in 1024	bytes per second units.

       The  json+  output format is identical to the json output format	except
       that it adds a full dump	of the completion latency bins.	Each bins  ob-
       ject  contains a	set of (key, value) pairs where	keys are latency dura-
       tions and values	count how many I/Os had	completion  latencies  of  the
       corresponding duration. For example, consider:

	      "bins"  :	 { "87552" : 1,	"89600"	: 1, "94720" : 1, "96768" : 1,
	      "97792" :	1, "99840" : 1,	"100864" : 2, "103936" : 6, "104960" :
	      534, "105984" : 5995, "107008" : 7529, ... }

       This  data  indicates  that  one	I/O required 87,552ns to complete, two
       I/Os required 100,864ns to complete, and	7529 I/Os  required  107,008ns
       to complete.

       Also  included  with  fio is a Python script fio_jsonplus_clat2csv that
       takes json+ output and generates	CSV-formatted  latency	data  suitable
       for plotting.

       The  latency  durations actually	represent the midpoints	of latency in-
       tervals.	 For details refer to `stat.h' in the fio source.

       There are two trace file	format that you	can encounter. The older  (v1)
       format  is  unsupported	since  version	1.20-rc3 (March	2008). It will
       still be	described below	in case	that you get an	old trace and want  to
       understand it.

       In  any	case  the trace	is a simple text file with a single action per

       Trace file format v1
	      Each line	represents a single I/O	action in the  following  for-

		     rw, offset, length

	      where `rw=0/1' for read/write, and the `offset' and `length' en-
	      tries being in bytes.

	      This format is not supported in fio versions >= 1.20-rc3.

       Trace file format v2
	      The second version of the	trace file format  was	added  in  fio
	      version  1.17.  It allows	to access more then one	file per trace
	      and has a	bigger set of possible file actions.

	      The first	line of	the trace file has to be:

		     "fio version 2 iolog"

	      Following	this can be lines in two different formats, which  are
	      described	below.

	      The file management format:
		     filename action

		     The `filename' is given as	an absolute path. The `action'
		     can be one	of these:

			    add	   Add the given `filename' to the trace.

			    open   Open	the file with  the  given  `filename'.
				   The	`filename' has to have been added with
				   the add action before.

			    close  Close the file with the  given  `filename'.
				   The file has	to have	been opened before.

	      The file I/O action format:
		     filename action offset length

		     The  `filename'  is given as an absolute path, and	has to
		     have been added and opened	before it  can	be  used  with
		     this  format.  The	 `offset'  and	`length'  are given in
		     bytes. The	`action' can be	one of these:

			    wait   Wait	for `offset' microseconds.  Everything
				   below  100 is discarded.  The time is rela-
				   tive	to the previous	`wait' statement.

			    read   Read	`length' bytes	beginning  from	 `off-

			    write  Write  `length'  bytes beginning from `off-

			    sync   fsync(2) the	file.

				   fdatasync(2)	the file.

			    trim   Trim	the given file from the	given `offset'
				   for `length'	bytes.

       Colocation  is a	common practice	used to	get the	most out of a machine.
       Knowing which workloads play nicely with	 each  other  and  which  ones
       don't  is  a  much  harder task.	While fio can replay workloads concur-
       rently via multiple jobs, it leaves some	variability up to  the	sched-
       uler  making  results harder to reproduce. Merging is a way to make the
       order of	events consistent.

       Merging is integrated into  I/O	replay	and  done  when	 a  merge_blk-
       trace_file  is  specified.  The	list  of files passed to read_iolog go
       through the merge process and output a single file stored to the	speci-
       fied  file.  The	 output	 file is passed	on as if it were the only file
       passed to read_iolog. An	example	would look like:

	      $	     fio      --read_iolog="<file1>:<file2>"	  --merge_blk-

       Creating	 only  the merged file can be done by passing the command line
       argument	merge-blktrace-only.

       Scaling traces can be done to see the relative impact of	any particular
       trace  being  slowed down or sped up. merge_blktrace_scalars takes in a
       colon separated list of percentage scalars. It is index paired with the
       files passed to read_iolog.

       With  scaling,  it  may	be  desirable to match the running time	of all
       traces.	This can be done with merge_blktrace_iters. It is index	paired
       with read_iolog just like merge_blktrace_scalars.

       In  an example, given two traces, A and B, each 60s long. If we want to
       see the impact of trace A issuing IOs twice as fast and repeat trace  A
       over the	runtime	of trace B, the	following can be done:

	      $	    fio	   --read_iolog="<trace_a>:"<trace_b>"	  --merge_blk-
	      trace_file"<output_file>"	     --merge_blktrace_scalars="50:100"

       This runs trace A at 2x the speed twice for approximately the same run-
       time as a single	run of trace B.

       In some cases, we want to understand CPU	overhead in a test. For	 exam-
       ple,  we	 test patches for the specific goodness	of whether they	reduce
       CPU usage.  Fio implements a balloon approach to	create	a  thread  per
       CPU  that  runs at idle priority, meaning that it only runs when	nobody
       else needs the cpu.  By measuring the amount of work completed  by  the
       thread, idleness	of each	CPU can	be derived accordingly.

       An unit work is defined as touching a full page of unsigned characters.
       Mean and	standard deviation of time to complete an  unit	 work  is  re-
       ported in "unit work" section. Options can be chosen to report detailed
       percpu idleness or overall system idleness by aggregating percpu	stats.

       Fio is usually run in one of two	ways, when data	verification is	 done.
       The  first is a normal write job	of some	sort with verify enabled. When
       the write phase has completed, fio switches to reads and	 verifies  ev-
       erything	 it  wrote.  The second	model is running just the write	phase,
       and then	later on running the same  job	(but  with  reads  instead  of
       writes)	to  repeat the same I/O	patterns and verify the	contents. Both
       of these	methods	depend on the write phase being	completed, as fio oth-
       erwise has no idea how much data	was written.

       With  verification  triggers,  fio  supports  dumping the current write
       state to	local files. Then a subsequent read verify workload  can  load
       this  state  and	know exactly where to stop. This is useful for testing
       cases where power is cut	to a server in	a  managed  fashion,  for  in-

       A verification trigger consists of two things:

	      1) Storing the write state of each job.

	      2) Executing a trigger command.

       The  write state	is relatively small, on	the order of hundreds of bytes
       to single kilobytes. It contains	information on the number  of  comple-
       tions done, the last X completions, etc.

       A  trigger  is invoked either through creation ('touch')	of a specified
       file in the system, or through a	timeout	setting. If fio	 is  run  with
       `--trigger-file=/tmp/trigger-file',  then it will continually check for
       the existence of	`/tmp/trigger-file'. When it sees this file,  it  will
       fire off	the trigger (thus saving state,	and executing the trigger com-

       For client/server runs, there's both a local and	remote trigger.	If fio
       is running as a server backend, it will send the	job states back	to the
       client for safe storage,	then execute the remote	trigger, if specified.
       If  a  local  trigger is	specified, the server will still send back the
       write state, but	the client will	then execute the trigger.

       Verification trigger example
	      Let's say	we want	to run a powercut test on the remote Linux ma-
	      chine  'server'.	 Our write workload is in `write-test.fio'. We
	      want to cut power	to 'server' at some point during the run,  and
	      we'll  run  this test from the safety or our local machine, 'lo-
	      calbox'. On the server, we'll start the fio backend normally:

		     server# fio --server

	      and on the client, we'll fire off	the workload:

		     localbox$	     fio	--client=server	       --trig-
		     ger-file=/tmp/my-trigger  --trigger-remote="bash -c "echo
		     b > /proc/sysrq-triger""

	      We set `/tmp/my-trigger' as the trigger file, and	we tell	fio to

		     echo b > /proc/sysrq-trigger

	      on  the  server once it has received the trigger and sent	us the
	      write state. This	will work, but it's not	really	cutting	 power
	      to  the  server, it's merely abruptly rebooting it. If we	have a
	      remote way of cutting power to the server	through	IPMI or	 simi-
	      lar,  we	could do that through a	local trigger command instead.
	      Let's assume we have a script that does IPMI reboot of  a	 given
	      hostname,	 ipmi-reboot.  On localbox, we could then have run fio
	      with a local trigger instead:

		     localbox$	     fio	--client=server	       --trig-
		     ger-file=/tmp/my-trigger --trigger="ipmi-reboot server"

	      For  this	 case,	fio  would  wait for the server	to send	us the
	      write state, then	execute	`ipmi-reboot server'  when  that  hap-

       Loading verify state
	      To  load	stored	write state, a read verification job file must
	      contain the verify_state_load option. If that is set,  fio  will
	      load  the	 previously  stored state. For a local fio run this is
	      done by loading the files	directly, and on a client/server  run,
	      the  server  backend  will ask the client	to send	the files over
	      and load them from there.

       Fio supports a variety of log  file  formats,  for  logging  latencies,
       bandwidth,  and	IOPS. The logs share a common format, which looks like

	      time (msec), value, data direction, block	size  (bytes),	offset
	      (bytes), command priority

       `Time'  for the log entry is always in milliseconds. The	`value'	logged
       depends on the type of log, it will be one of the following:

	      Latency log
		     Value is latency in nsecs

	      Bandwidth	log
		     Value is in KiB/sec

	      IOPS log
		     Value is IOPS

       `Data direction'	is one of the following:

	      0	     I/O is a READ

	      1	     I/O is a WRITE

	      2	     I/O is a TRIM

       The entry's `block size'	is always in bytes. The	`offset' is the	 posi-
       tion  in	 bytes from the	start of the file for that particular I/O. The
       logging of the offset can be toggled with log_offset.

       `Command	priority` is 0 for normal priority and 1  for  high  priority.
       This is controlled by the ioengine specific cmdprio_percentage.

       Fio  defaults to	logging	every individual I/O but when windowed logging
       is set through log_avg_msec, either the average	(by  default)  or  the
       maximum	(log_max_value	is set)	`value'	seen over the specified	period
       of time is recorded. Each `data direction' seen within the  window  pe-
       riod  will  aggregate its values	in a separate row. Further, when using
       windowed	logging	the `block size' and `offset' entries will always con-
       tain 0.

       Normally	 fio  is  invoked  as a	stand-alone application	on the machine
       where the I/O workload should be	generated. However,  the  backend  and
       frontend	of fio can be run separately i.e., the fio server can generate
       an I/O workload on the "Device Under Test" while	being controlled by  a
       client on another machine.

       Start the server	on the machine which has access	to the storage DUT:

	      $	fio --server=args

       where `args' defines what fio listens to. The arguments are of the form
       `type,hostname' or `IP,port'. `type' is either `ip' (or ip4) for	TCP/IP
       v4,  `ip6'  for	TCP/IP	v6,  or	`sock' for a local unix	domain socket.
       `hostname' is either a hostname or IP address, and `port' is  the  port
       to  listen  to  (only valid for TCP/IP, not a local socket). Some exam-

	      1) fio --server
		     Start a fio server, listening on all  interfaces  on  the
		     default port (8765).

	      2) fio --server=ip:hostname,4444
		     Start a fio server, listening on IP belonging to hostname
		     and on port 4444.

	      3) fio --server=ip6:::1,4444
		     Start a fio server, listening on IPv6 localhost  ::1  and
		     on	port 4444.

	      4) fio --server=,4444
		     Start  a  fio server, listening on	all interfaces on port

	      5) fio --server=
		     Start a fio server, listening on IP  on  the  de-
		     fault port.

	      6) fio --server=sock:/tmp/fio.sock
		     Start  a  fio  server,  listening	on  the	 local	socket

       Once a server is	running, a "client" can	 connect  to  the  fio	server

	      $	fio <local-args> --client=<server> <remote-args> <job file(s)>

       where  `local-args'  are	 arguments for the client where	it is running,
       `server'	is the connect string, and `remote-args' and `job file(s)' are
       sent  to	 the server. The `server' string follows the same format as it
       does on the server side,	to allow IP/hostname/socket and	port strings.

       Fio can connect to multiple servers this	way:

	      $	fio --client=<server1> <job file(s)>  --client=<server2>  <job

       If  the	job  file  is located on the fio server, then you can tell the
       server to load a	local file as  well.  This  is	done  by  using	 --re-

	      $	fio --client=server --remote-config /path/to/file.fio

       Then fio	will open this local (to the server) job file instead of being
       passed one from the client.

       If you have many	servers	(example: 100 VMs/containers), you can input a
       pathname	of a file containing host IPs/names as the parameter value for
       the --client option. For	example, here is an example  `host.list'  file
       containing 2 hostnames:


       The fio command would then be:

	      $	fio --client=host.list <job file(s)>

       In  this	mode, you cannot input server-specific parameters or job files
       -- all servers receive the same job file.

       In order	to let `fio --client' runs use a shared	filesystem from	multi-
       ple  hosts, `fio	--client' now prepends the IP address of the server to
       the filename. For example, if fio is using the directory	`/mnt/nfs/fio'
       and  is	writing	filename `fileio.tmp', with a --client `hostfile' con-
       taining two hostnames `h1' and `h2' with	 IP  addresses
       and, then	fio will create	two files:


       Terse  output  in  client/server	mode will differ slightly from what is
       produced	when fio is run	in stand-alone mode. See the terse output sec-
       tion for	details.

       fio was written by Jens Axboe <>.
       This  man  page	was  written by	Aaron Carroll <>
       based on	documentation by Jens Axboe.
       This man	page was rewritten  by	Tomohiro  Kusumi  <>
       based on	documentation by Jens Axboe.

       Report bugs to the fio mailing list <>.


       For further documentation see HOWTO and README.
       Sample jobfiles are available in	the `examples/'	directory.
       These are typically located under `/usr/share/doc/fio'.


User Manual			  August 2017				fio(1)


Want to link to this manual page? Use this URL:

home | help