Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
ZSTD(1)				 User Commands			       ZSTD(1)

NAME
       zstd  -	zstd,  zstdmt,	unzstd,	 zstdcat - Compress or decompress .zst
       files

SYNOPSIS
       zstd [OPTIONS] [-|INPUT-FILE] [-o OUTPUT-FILE]

       zstdmt is equivalent to zstd -T0

       unzstd is equivalent to zstd -d

       zstdcat is equivalent to	zstd -dcf

DESCRIPTION
       zstd is a fast lossless	compression  algorithm	and  data  compression
       tool,  with  command  line syntax similar to gzip (1) and xz (1). It is
       based on	the LZ77 family, with further FSE & huff0 entropy stages. zstd
       offers  highly configurable compression speed, with fast	modes at > 200
       MB/s per	code, and strong modes nearing	lzma  compression  ratios.  It
       also features a very fast decoder, with speeds >	500 MB/s per core.

       zstd command line syntax	is generally similar to	gzip, but features the
       following differences :

       o   Source files	are preserved by default. It's possible	to remove them
	   automatically by using the --rm command.

       o   When	 compressing  a	 single	file, zstd displays progress notifica-
	   tions and result summary by default.	Use -q to turn them off.

       o   zstd	does not accept	input from console, but	 it  properly  accepts
	   stdin when it's not the console.

       o   zstd	 displays a short help page when command line is an error. Use
	   -q to turn it off.

       zstd compresses or decompresses each file  according  to	 the  selected
       operation  mode.	 If  no	 files are given or file is -, zstd reads from
       standard	input and writes the processed data to standard	 output.  zstd
       will refuse to write compressed data to standard	output if it is	a ter-
       minal : it will display an error	message	and skip the file.  Similarly,
       zstd will refuse	to read	compressed data	from standard input if it is a
       terminal.

       Unless --stdout or -o is	specified, files are written  to  a  new  file
       whose name is derived from the source file name:

       o   When	 compressing,  the suffix .zst is appended to the source file-
	   name	to get the target filename.

       o   When	decompressing, the .zst	suffix	is  removed  from  the	source
	   filename to get the target filename

   Concatenation with .zst files
       It  is  possible	 to concatenate	.zst files as is. zstd will decompress
       such files as if	they were a single .zst	file.

OPTIONS
   Integer suffixes and	special	values
       In most places where an integer argument	is expected, an	optional  suf-
       fix  is	supported  to easily indicate large integers. There must be no
       space between the integer and the suffix.

       KiB    Multiply the integer by 1,024 (2^10). Ki,	K, and KB are accepted
	      as synonyms for KiB.

       MiB    Multiply	the  integer  by  1,048,576  (2^20). Mi, M, and	MB are
	      accepted as synonyms for MiB.

   Operation mode
       If multiple operation mode  options  are	 given,	 the  last  one	 takes
       effect.

       -z, --compress
	      Compress.	 This  is the default operation	mode when no operation
	      mode option is specified and no other operation mode is  implied
	      from  the	 command  name	(for  example, unzstd implies --decom-
	      press).

       -d, --decompress, --uncompress
	      Decompress.

       -t, --test
	      Test the integrity of compressed files. This option  is  equiva-
	      lent  to --decompress --stdout except that the decompressed data
	      is discarded instead of being written  to	 standard  output.  No
	      files are	created	or removed.

       -b#    Benchmark	file(s)	using compression level	#

       --train FILEs
	      Use FILEs	as a training set to create a dictionary. The training
	      set should contain a lot of small	files (> 100).

       -l, --list
	      Display information related to a zstd compressed file,  such  as
	      size,  ratio,  and  checksum.  Some  of  these fields may	not be
	      available. This command can be augmented with the	-v modifier.

   Operation modifiers
       -#     #	compression level [1-19] (default: 3)

       --ultra
	      unlocks high compression levels 20+ (maximum 22),	 using	a  lot
	      more memory. Note	that decompression will	also require more mem-
	      ory when using these levels.

       --long[=#]
	      enables long distance matching with # windowLog, if not #	is not
	      present  it defaults to 27. This increases the window size (win-
	      dowLog) and memory usage for both	the compressor and  decompres-
	      sor.  This  setting is designed to improve the compression ratio
	      for files	with long matches at a large distance.

	      Note: If windowLog is set	to larger than 27, --long=windowLog or
	      --memory=windowSize needs	to be passed to	the decompressor.

       -T#, --threads=#
	      Compress	using  #  threads  (default: 1). If # is 0, attempt to
	      detect and use the number	of physical CPU	cores. In  all	cases,
	      the  nb  of threads is capped to ZSTDMT_NBTHREADS_MAX==256. This
	      modifier does nothing if zstd is	compiled  without  multithread
	      support.

       -D file
	      use file as Dictionary to	compress or decompress FILE(s)

       --nodictID
	      do  not store dictionary ID within frame header (dictionary com-
	      pression). The decoder will have to rely on  implicit  knowledge
	      about which dictionary to	use, it	won't be able to check if it's
	      correct.

       -o file
	      save result into file (only possible with	a single INPUT-FILE)

       -f, --force
	      overwrite	output without prompting,  and	(de)compress  symbolic
	      links

       -c, --stdout
	      force write to standard output, even if it is the	console

       --[no-]sparse
	      enable  /	 disable  sparse  FS  support, to make files with many
	      zeroes smaller on	disk. Creating	sparse	files  may  save  disk
	      space  and speed up decompression	by reducing the	amount of disk
	      I/O. default: enabled when output	is into	a file,	 and  disabled
	      when  output  is	stdout.	This setting overrides default and can
	      force sparse mode	over stdout.

       --rm   remove source file(s) after successful compression or decompres-
	      sion

       -k, --keep
	      keep  source  file(s) after successful compression or decompres-
	      sion. This is the	default	behavior.

       -r     operate recursively on dictionaries

       --format=FORMAT
	      compress and decompress in other formats.	If compiled with  sup-
	      port,  zstd can compress to or decompress	from other compression
	      algorithm	formats. Possibly  available  options  are  gzip,  xz,
	      lzma, and	lz4.

       -h/-H, --help
	      display help/long	help and exit

       -V, --version
	      display  version	number	and exit. Advanced : -vV also displays
	      supported	formats. -vvV also displays POSIX support.

       -v     verbose mode

       -q, --quiet
	      suppress warnings,  interactivity,  and  notifications.  specify
	      twice to suppress	errors too.

       -C, --[no-]check
	      add  integrity  check  computed from uncompressed	data (default:
	      enabled)

       --     All arguments after -- are treated as files

DICTIONARY BUILDER
       zstd offers dictionary compression, which greatly  improves  efficiency
       on  small files and messages. It's possible to train zstd with a	set of
       samples,	the result of which is saved into a file called	a  dictionary.
       Then  during  compression and decompression, reference the same dictio-
       nary, using command -D dictionaryFileName. Compression of  small	 files
       similar to the sample set will be greatly improved.

       --train FILEs
	      Use  FILEs  as training set to create a dictionary. The training
	      set should contain a lot of small	files (> 100), and weight typ-
	      ically 100x the target dictionary	size (for example, 10 MB for a
	      100 KB dictionary).

	      Supports multithreading if zstd is compiled with threading  sup-
	      port. Additional parameters can be specified with	--train-cover.
	      The   legacy   dictionary	  builder   can	  be   accessed	  with
	      --train-legacy. Equivalent to --train-cover=d=8,steps=4.

       -o file
	      Dictionary saved into file (default name:	dictionary).

       --maxdict=#
	      Limit dictionary to specified size (default: 112640).

       -#     Use  # compression level during training (optional). Will	gener-
	      ate  statistics  more  tuned  for	 selected  compression	level,
	      resulting	 in  a	small  compression  ratio improvement for this
	      level.

       -B#    Split input files	in blocks of size # (default: no split)

       --dictID=#
	      A	dictionary ID is a locally unique ID that a decoder can	use to
	      verify  it  is using the right dictionary. By default, zstd will
	      create a 4-bytes random number ID. It's possible to give a  pre-
	      cise  number  instead. Short numbers have	an advantage : an ID <
	      256 will only need 1 byte	in the compressed frame	header,	and an
	      ID  < 65536 will only need 2 bytes. This compares	favorably to 4
	      bytes default. However, it's up to the dictionary	manager	to not
	      assign twice the same ID to 2 different dictionaries.

       --train-cover[=k#,d=#,steps=#]
	      Select  parameters  for the default dictionary builder algorithm
	      named cover. If d	is not specified, then it tries	d = 6 and d  =
	      8.  If  k	 is  not  specified, then it tries steps values	in the
	      range [50, 2000].	If steps is not	specified,  then  the  default
	      value of 40 is used. Requires that d <= k.

	      Selects segments of size k with highest score to put in the dic-
	      tionary. The score of a segment is computed by the  sum  of  the
	      frequencies of all the subsegments of size d. Generally d	should
	      be in the	range [6, 8], occasionally up to 16, but the algorithm
	      will run faster with d <=	8. Good	values for k vary widely based
	      on the input data, but a safe range is [2	* d,  2000].  Supports
	      multithreading if	zstd is	compiled with threading	support.

	      Examples:

	      zstd --train-cover FILEs

	      zstd --train-cover=k=50,d=8 FILEs

	      zstd --train-cover=d=8,steps=500 FILEs

	      zstd --train-cover=k=50 FILEs

       --train-legacy[=selectivity=#]
	      Use  legacy  dictionary builder algorithm	with the given dictio-
	      nary selectivity	(default:  9).	The  smaller  the  selectivity
	      value,  the  denser the dictionary, improving its	efficiency but
	      reducing its possible maximum size. --train-legacy=s=#  is  also
	      accepted.

	      Examples:

	      zstd --train-legacy FILEs

	      zstd --train-legacy=selectivity=8	FILEs

BENCHMARK
       -b#    benchmark	file(s)	using compression level	#

       -e#    benchmark	file(s)	using multiple compression levels, from	-b# to
	      -e# (inclusive)

       -i#    minimum evaluation time, in  seconds  (default:  3s),  benchmark
	      mode only

       -B#, --block-size=#
	      cut  file(s)  into  independent  blocks  of  size	# (default: no
	      block)

       --priority=rt
	      set process priority to real-time

       Output Format: CompressionLevel#Filename	 :  IntputSize	->  OutputSize
       (CompressionRatio), CompressionSpeed, DecompressionSpeed

       Methodology:  For  both compression and decompression speed, the	entire
       input is	compressed/decompressed	in-memory  to  measure	speed.	A  run
       lasts  at  least	 1  sec,  so  when  files  are	small,	they  are com-
       pressed/decompressed several times per run, in order  to	 improve  mea-
       surement	accuracy.

ADVANCED COMPRESSION OPTIONS
   --zstd[=options]:
       zstd provides 22	predefined compression levels. The selected or default
       predefined compression level can	be changed with	 advanced  compression
       options.	 The  options  are provided as a comma-separated list. You may
       specify only the	options	you want to change and the rest	will be	 taken
       from  the  selected or default compression level. The list of available
       options:

       strategy=strat, strat=strat
	      Specify a	strategy used by a match finder.

	      There are	8 strategies numbered from 1  to  8,  from  faster  to
	      stronger:	1=ZSTD_fast, 2=ZSTD_dfast, 3=ZSTD_greedy, 4=ZSTD_lazy,
	      5=ZSTD_lazy2, 6=ZSTD_btlazy2, 7=ZSTD_btopt, 8=ZSTD_btultra.

       windowLog=wlog, wlog=wlog
	      Specify the maximum number of bits for a match distance.

	      The higher number	of increases the chance	to find	a match	 which
	      usually  improves	 compression  ratio.  It also increases	memory
	      requirements for the compressor and  decompressor.  The  minimum
	      wlog is 10 (1 KiB) and the maximum is 30 (1 GiB) on 32-bit plat-
	      forms and	31 (2 GiB) on 64-bit platforms.

	      Note: If windowLog is set	to larger than 27, --long=windowLog or
	      --memory=windowSize needs	to be passed to	the decompressor.

       hashLog=hlog, hlog=hlog
	      Specify the maximum number of bits for a hash table.

	      Bigger  hash  tables  cause  less	collisions which usually makes
	      compression faster, but requires more memory during compression.

	      The minimum hlog is 6 (64	B) and the maximum is 26 (128 MiB).

       chainLog=clog, clog=clog
	      Specify the maximum number of bits for a hash chain or a	binary
	      tree.

	      Higher  numbers  of  bits	 increases  the	chance to find a match
	      which usually improves compression ratio.	 It  also  slows  down
	      compression speed	and increases memory requirements for compres-
	      sion. This option	is ignored for the ZSTD_fast strategy.

	      The minimum clog is 6 (64	B) and the maximum is 28 (256 MiB).

       searchLog=slog, slog=slog
	      Specify the maximum number of searches in	 a  hash  chain	 or  a
	      binary tree using	logarithmic scale.

	      More searches increases the chance to find a match which usually
	      increases	compression ratio but decreases	compression speed.

	      The minimum slog is 1 and	the maximum is 26.

       searchLength=slen, slen=slen
	      Specify the minimum searched length of a match in	a hash table.

	      Larger search lengths usually  decrease  compression  ratio  but
	      improve decompression speed.

	      The minimum slen is 3 and	the maximum is 7.

       targetLen=tlen, tlen=tlen
	      Specify  the  minimum match length that causes a match finder to
	      stop searching for better	matches.

	      A	larger minimum match length usually improves compression ratio
	      but  decreases  compression speed. This option is	only used with
	      strategies ZSTD_btopt and	ZSTD_btultra.

	      The minimum tlen is 4 and	the maximum is 999.

       overlapLog=ovlog, ovlog=ovlog
	      Determine	overlapSize, amount of	data  reloaded	from  previous
	      job.  This  parameter  is	 only available	when multithreading is
	      enabled. Reloading more data  improves  compression  ratio,  but
	      decreases	speed.

	      The  minimum ovlog is 0, and the maximum is 9. 0 means "no over-
	      lap", hence completely independent jobs. 9 means "full overlap",
	      meaning up to windowSize is reloaded from	previous job. Reducing
	      ovlog by 1 reduces the amount of reload by a factor  2.  Default
	      ovlog is 6, which	means "reload windowSize / 8". Exception : the
	      maximum compression level	(22) has a default ovlog of 9.

       ldmHashLog=ldmhlog, ldmhlog=ldmhlog
	      Specify the maximum size for a hash table	used for long distance
	      matching.

	      This option is ignored unless long distance matching is enabled.

	      Bigger  hash  tables  usually  improve  compression ratio	at the
	      expense of more memory during compression	and a decrease in com-
	      pression speed.

	      The minimum ldmhlog is 6 and the maximum is 26 (default: 20).

       ldmSearchLength=ldmslen,	ldmslen=ldmslen
	      Specify the minimum searched length of a match for long distance
	      matching.

	      This option is ignored unless long distance matching is enabled.

	      Larger/very small	values usually decrease	compression ratio.

	      The minumum ldmslen is 4 and the maximum is 4096 (default: 64).

       ldmBucketSizeLog=ldmblog, ldmblog=ldmblog
	      Specify the size of each bucket for the hash table used for long
	      distance matching.

	      This option is ignored unless long distance matching is enabled.

	      Larger  bucket  sizes  improve collision resolution but decrease
	      compression speed.

	      The minimum ldmblog is 0 and the maximum is 8 (default: 3).

       ldmHashEveryLog=ldmhevery, ldmhevery=ldmhevery
	      Specify the frequency of inserting entries into  the  long  dis-
	      tance matching hash table.

	      This option is ignored unless long distance matching is enabled.

	      Larger values will improve compression speed. Deviating far from
	      the default value	will likely result in a	decrease  in  compres-
	      sion ratio.

	      The default value	is wlog	- ldmhlog.

   -B#:
       Select  the  size  of each compression job. This	parameter is available
       only when multi-threading is enabled. Default value is 4	*  windowSize,
       which means it varies depending on compression level. -B# makes it pos-
       sible to	select a custom	value. Note that job size must respect a mini-
       mum value which is enforced transparently. This minimum is either 1 MB,
       or overlapSize, whichever is largest.

   Example
       The following parameters	sets advanced compression options to those  of
       predefined level	19 for files bigger than 256 KB:

       --zstd=windowLog=23,chainLog=23,hashLog=22,searchLog=6,search-
       Length=3,targetLength=48,strategy=6

BUGS
       Report bugs at: https://github.com/facebook/zstd/issues

AUTHOR
       Yann Collet

zstd 1.3.4			  2018-01-27			       ZSTD(1)

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | DICTIONARY BUILDER | BENCHMARK | ADVANCED COMPRESSION OPTIONS | BUGS | AUTHOR

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=zstd&manpath=FreeBSD+12.0-RELEASE+and+Ports>

home | help