Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
ZSTD(1)				 User Commands			       ZSTD(1)

NAME
       zstd  -	zstd,  zstdmt,	unzstd,	 zstdcat - Compress or decompress .zst
       files

SYNOPSIS
       zstd [OPTIONS] [-|INPUT-FILE] [-o OUTPUT-FILE]

       zstdmt is equivalent to zstd -T0

       unzstd is equivalent to zstd -d

       zstdcat is equivalent to	zstd -dcf

DESCRIPTION
       zstd is a fast lossless	compression  algorithm	and  data  compression
       tool,  with  command  line syntax similar to gzip (1) and xz (1). It is
       based on	the LZ77 family, with further FSE & huff0 entropy stages. zstd
       offers  highly configurable compression speed, with fast	modes at > 200
       MB/s per	code, and strong modes nearing	lzma  compression  ratios.  It
       also features a very fast decoder, with speeds >	500 MB/s per core.

       zstd command line syntax	is generally similar to	gzip, but features the
       following differences :

       o   Source files	are preserved by default. It's possible	to remove them
	   automatically by using the --rm command.

       o   When	 compressing  a	 single	file, zstd displays progress notifica-
	   tions and result summary by default.	Use -q to turn them off.

       o   zstd	does not accept	input from console, but	 it  properly  accepts
	   stdin when it's not the console.

       o   zstd	 displays a short help page when command line is an error. Use
	   -q to turn it off.

       zstd compresses or decompresses each file according to the selected op-
       eration mode. If	no files are given or file is -, zstd reads from stan-
       dard input and writes the processed data	to standard output. zstd  will
       refuse  to write	compressed data	to standard output if it is a terminal
       : it will display an error message and skip the file.  Similarly,  zstd
       will refuse to read compressed data from	standard input if it is	a ter-
       minal.

       Unless --stdout or -o is	specified, files are written  to  a  new  file
       whose name is derived from the source file name:

       o   When	 compressing,  the suffix .zst is appended to the source file-
	   name	to get the target filename.

       o   When	decompressing, the .zst	suffix	is  removed  from  the	source
	   filename to get the target filename

   Concatenation with .zst files
       It  is  possible	 to concatenate	.zst files as is. zstd will decompress
       such files as if	they were a single .zst	file.

OPTIONS
   Integer suffixes and	special	values
       In most places where an integer argument	is expected, an	optional  suf-
       fix  is	supported  to easily indicate large integers. There must be no
       space between the integer and the suffix.

       KiB    Multiply the integer by 1,024 (2^10). Ki,	K, and KB are accepted
	      as synonyms for KiB.

       MiB    Multiply	the integer by 1,048,576 (2^20). Mi, M,	and MB are ac-
	      cepted as	synonyms for MiB.

   Operation mode
       If multiple operation mode options are given, the last  one  takes  ef-
       fect.

       -z, --compress
	      Compress.	 This  is the default operation	mode when no operation
	      mode option is specified and no other operation mode is  implied
	      from  the	 command  name	(for  example, unzstd implies --decom-
	      press).

       -d, --decompress, --uncompress
	      Decompress.

       -t, --test
	      Test the integrity of compressed files. This option  is  equiva-
	      lent  to --decompress --stdout except that the decompressed data
	      is discarded instead of being written  to	 standard  output.  No
	      files are	created	or removed.

       -b#    Benchmark	file(s)	using compression level	#

       --train FILEs
	      Use FILEs	as a training set to create a dictionary. The training
	      set should contain a lot of small	files (> 100).

       -l, --list
	      Display information related to a zstd compressed file,  such  as
	      size,  ratio,  and  checksum.  Some  of  these fields may	not be
	      available. This command can be augmented with the	-v modifier.

   Operation modifiers
       -#     #	compression level [1-19] (default: 3)

       --ultra
	      unlocks high compression levels 20+ (maximum 22),	 using	a  lot
	      more memory. Note	that decompression will	also require more mem-
	      ory when using these levels.

       --long[=#]
	      enables long distance matching with # windowLog, if not #	is not
	      present  it defaults to 27. This increases the window size (win-
	      dowLog) and memory usage for both	the compressor and  decompres-
	      sor.  This  setting is designed to improve the compression ratio
	      for files	with long matches at a large distance.

	      Note: If windowLog is set	to larger than 27, --long=windowLog or
	      --memory=windowSize needs	to be passed to	the decompressor.

       -T#, --threads=#
	      Compress using # threads (default: 1). If	# is 0,	attempt	to de-
	      tect and use the number of physical CPU cores. In	all cases, the
	      nb of threads is capped to ZSTDMT_NBTHREADS_MAX==256. This modi-
	      fier does	nothing	if zstd	is compiled without  multithread  sup-
	      port.

       -D file
	      use file as Dictionary to	compress or decompress FILE(s)

       --nodictID
	      do  not store dictionary ID within frame header (dictionary com-
	      pression). The decoder will have to rely on  implicit  knowledge
	      about which dictionary to	use, it	won't be able to check if it's
	      correct.

       -o file
	      save result into file (only possible with	a single INPUT-FILE)

       -f, --force
	      overwrite	output without prompting,  and	(de)compress  symbolic
	      links

       -c, --stdout
	      force write to standard output, even if it is the	console

       --[no-]sparse
	      enable  /	disable	sparse FS support, to make files with many ze-
	      roes smaller on disk. Creating sparse files may save disk	 space
	      and  speed  up decompression by reducing the amount of disk I/O.
	      default: enabled when output is into a file, and	disabled  when
	      output  is  stdout. This setting overrides default and can force
	      sparse mode over stdout.

       --rm   remove source file(s) after successful compression or decompres-
	      sion

       -k, --keep
	      keep  source  file(s) after successful compression or decompres-
	      sion. This is the	default	behavior.

       -r     operate recursively on dictionaries

       --format=FORMAT
	      compress and decompress in other formats.	If compiled with  sup-
	      port,  zstd can compress to or decompress	from other compression
	      algorithm	formats. Possibly  available  options  are  gzip,  xz,
	      lzma, and	lz4.

       -h/-H, --help
	      display help/long	help and exit

       -V, --version
	      display  version	number	and exit. Advanced : -vV also displays
	      supported	formats. -vvV also displays POSIX support.

       -v     verbose mode

       -q, --quiet
	      suppress warnings,  interactivity,  and  notifications.  specify
	      twice to suppress	errors too.

       -C, --[no-]check
	      add  integrity  check  computed from uncompressed	data (default:
	      enabled)

       --     All arguments after -- are treated as files

DICTIONARY BUILDER
       zstd offers dictionary compression, which greatly  improves  efficiency
       on  small files and messages. It's possible to train zstd with a	set of
       samples,	the result of which is saved into a file called	a  dictionary.
       Then  during  compression and decompression, reference the same dictio-
       nary, using command -D dictionaryFileName. Compression of  small	 files
       similar to the sample set will be greatly improved.

       --train FILEs
	      Use  FILEs  as training set to create a dictionary. The training
	      set should contain a lot of small	files (> 100), and weight typ-
	      ically 100x the target dictionary	size (for example, 10 MB for a
	      100 KB dictionary).

	      Supports multithreading if zstd is compiled with threading  sup-
	      port. Additional parameters can be specified with	--train-cover.
	      The   legacy   dictionary	  builder   can	  be   accessed	  with
	      --train-legacy. Equivalent to --train-cover=d=8,steps=4.

       -o file
	      Dictionary saved into file (default name:	dictionary).

       --maxdict=#
	      Limit dictionary to specified size (default: 112640).

       -#     Use  # compression level during training (optional). Will	gener-
	      ate statistics more tuned	for selected  compression  level,  re-
	      sulting in a small compression ratio improvement for this	level.

       -B#    Split input files	in blocks of size # (default: no split)

       --dictID=#
	      A	dictionary ID is a locally unique ID that a decoder can	use to
	      verify it	is using the right dictionary. By default,  zstd  will
	      create  a	4-bytes	random number ID. It's possible	to give	a pre-
	      cise number instead. Short numbers have an advantage : an	 ID  <
	      256 will only need 1 byte	in the compressed frame	header,	and an
	      ID < 65536 will only need	2 bytes. This compares favorably to  4
	      bytes default. However, it's up to the dictionary	manager	to not
	      assign twice the same ID to 2 different dictionaries.

       --train-cover[=k#,d=#,steps=#]
	      Select parameters	for the	default	dictionary  builder  algorithm
	      named  cover. If d is not	specified, then	it tries d = 6 and d =
	      8. If k is not specified,	then it	 tries	steps  values  in  the
	      range  [50,  2000].  If steps is not specified, then the default
	      value of 40 is used. Requires that d <= k.

	      Selects segments of size k with highest score to put in the dic-
	      tionary.	The  score  of a segment is computed by	the sum	of the
	      frequencies of all the subsegments of size d. Generally d	should
	      be in the	range [6, 8], occasionally up to 16, but the algorithm
	      will run faster with d <=	8. Good	values for k vary widely based
	      on  the  input data, but a safe range is [2 * d, 2000]. Supports
	      multithreading if	zstd is	compiled with threading	support.

	      Examples:

	      zstd --train-cover FILEs

	      zstd --train-cover=k=50,d=8 FILEs

	      zstd --train-cover=d=8,steps=500 FILEs

	      zstd --train-cover=k=50 FILEs

       --train-legacy[=selectivity=#]
	      Use legacy dictionary builder algorithm with the	given  dictio-
	      nary  selectivity	 (default:  9).	 The  smaller  the selectivity
	      value, the denser	the dictionary,	improving its  efficiency  but
	      reducing	its  possible maximum size. --train-legacy=s=# is also
	      accepted.

	      Examples:

	      zstd --train-legacy FILEs

	      zstd --train-legacy=selectivity=8	FILEs

BENCHMARK
       -b#    benchmark	file(s)	using compression level	#

       -e#    benchmark	file(s)	using multiple compression levels, from	-b# to
	      -e# (inclusive)

       -i#    minimum  evaluation  time,  in  seconds (default:	3s), benchmark
	      mode only

       -B#, --block-size=#
	      cut file(s) into independent  blocks  of	size  #	 (default:  no
	      block)

       --priority=rt
	      set process priority to real-time

       Output  Format:	CompressionLevel#Filename  :  IntputSize -> OutputSize
       (CompressionRatio), CompressionSpeed, DecompressionSpeed

       Methodology: For	both compression and decompression speed,  the	entire
       input  is  compressed/decompressed  in-memory  to  measure speed. A run
       lasts at	least 1	sec, so	when files are small, they are	compressed/de-
       compressed several times	per run, in order to improve measurement accu-
       racy.

ADVANCED COMPRESSION OPTIONS
   --zstd[=options]:
       zstd provides 22	predefined compression levels. The selected or default
       predefined  compression	level can be changed with advanced compression
       options.	The options are	provided as a comma-separated  list.  You  may
       specify	only the options you want to change and	the rest will be taken
       from the	selected or default compression	level. The list	 of  available
       options:

       strategy=strat, strat=strat
	      Specify a	strategy used by a match finder.

	      There  are  8  strategies	 numbered  from	1 to 8,	from faster to
	      stronger:	1=ZSTD_fast, 2=ZSTD_dfast, 3=ZSTD_greedy, 4=ZSTD_lazy,
	      5=ZSTD_lazy2, 6=ZSTD_btlazy2, 7=ZSTD_btopt, 8=ZSTD_btultra.

       windowLog=wlog, wlog=wlog
	      Specify the maximum number of bits for a match distance.

	      The  higher number of increases the chance to find a match which
	      usually improves compression ratio. It also increases memory re-
	      quirements for the compressor and	decompressor. The minimum wlog
	      is 10 (1 KiB) and	the maximum is 30 (1 GiB) on 32-bit  platforms
	      and 31 (2	GiB) on	64-bit platforms.

	      Note: If windowLog is set	to larger than 27, --long=windowLog or
	      --memory=windowSize needs	to be passed to	the decompressor.

       hashLog=hlog, hlog=hlog
	      Specify the maximum number of bits for a hash table.

	      Bigger hash tables cause less  collisions	 which	usually	 makes
	      compression faster, but requires more memory during compression.

	      The minimum hlog is 6 (64	B) and the maximum is 26 (128 MiB).

       chainLog=clog, clog=clog
	      Specify  the maximum number of bits for a	hash chain or a	binary
	      tree.

	      Higher numbers of	bits increases the  chance  to	find  a	 match
	      which  usually  improves	compression  ratio. It also slows down
	      compression speed	and increases memory requirements for compres-
	      sion. This option	is ignored for the ZSTD_fast strategy.

	      The minimum clog is 6 (64	B) and the maximum is 28 (256 MiB).

       searchLog=slog, slog=slog
	      Specify  the maximum number of searches in a hash	chain or a bi-
	      nary tree	using logarithmic scale.

	      More searches increases the chance to find a match which usually
	      increases	compression ratio but decreases	compression speed.

	      The minimum slog is 1 and	the maximum is 26.

       searchLength=slen, slen=slen
	      Specify the minimum searched length of a match in	a hash table.

	      Larger search lengths usually decrease compression ratio but im-
	      prove decompression speed.

	      The minimum slen is 3 and	the maximum is 7.

       targetLen=tlen, tlen=tlen
	      Specify the minimum match	length that causes a match  finder  to
	      stop searching for better	matches.

	      A	larger minimum match length usually improves compression ratio
	      but decreases compression	speed. This option is only  used  with
	      strategies ZSTD_btopt and	ZSTD_btultra.

	      The minimum tlen is 4 and	the maximum is 999.

       overlapLog=ovlog, ovlog=ovlog
	      Determine	 overlapSize,  amount  of  data	reloaded from previous
	      job. This	parameter is only available when multithreading	is en-
	      abled.  Reloading	 more data improves compression	ratio, but de-
	      creases speed.

	      The minimum ovlog	is 0, and the maximum is 9. 0 means "no	 over-
	      lap", hence completely independent jobs. 9 means "full overlap",
	      meaning up to windowSize is reloaded from	previous job. Reducing
	      ovlog  by	 1 reduces the amount of reload	by a factor 2. Default
	      ovlog is 6, which	means "reload windowSize / 8". Exception : the
	      maximum compression level	(22) has a default ovlog of 9.

       ldmHashLog=ldmhlog, ldmhlog=ldmhlog
	      Specify the maximum size for a hash table	used for long distance
	      matching.

	      This option is ignored unless long distance matching is enabled.

	      Bigger hash tables usually improve compression ratio at the  ex-
	      pense  of	 more memory during compression	and a decrease in com-
	      pression speed.

	      The minimum ldmhlog is 6 and the maximum is 26 (default: 20).

       ldmSearchLength=ldmslen,	ldmslen=ldmslen
	      Specify the minimum searched length of a match for long distance
	      matching.

	      This option is ignored unless long distance matching is enabled.

	      Larger/very small	values usually decrease	compression ratio.

	      The minumum ldmslen is 4 and the maximum is 4096 (default: 64).

       ldmBucketSizeLog=ldmblog, ldmblog=ldmblog
	      Specify the size of each bucket for the hash table used for long
	      distance matching.

	      This option is ignored unless long distance matching is enabled.

	      Larger bucket sizes improve collision  resolution	 but  decrease
	      compression speed.

	      The minimum ldmblog is 0 and the maximum is 8 (default: 3).

       ldmHashEveryLog=ldmhevery, ldmhevery=ldmhevery
	      Specify  the  frequency  of inserting entries into the long dis-
	      tance matching hash table.

	      This option is ignored unless long distance matching is enabled.

	      Larger values will improve compression speed. Deviating far from
	      the  default  value will likely result in	a decrease in compres-
	      sion ratio.

	      The default value	is wlog	- ldmhlog.

   -B#:
       Select the size of each compression job.	This  parameter	 is  available
       only  when multi-threading is enabled. Default value is 4 * windowSize,
       which means it varies depending on compression level. -B# makes it pos-
       sible to	select a custom	value. Note that job size must respect a mini-
       mum value which is enforced transparently. This minimum is either 1 MB,
       or overlapSize, whichever is largest.

   Example
       The  following parameters sets advanced compression options to those of
       predefined level	19 for files bigger than 256 KB:

       --zstd=windowLog=23,chainLog=23,hashLog=22,searchLog=6,search-
       Length=3,targetLength=48,strategy=6

BUGS
       Report bugs at: https://github.com/facebook/zstd/issues

AUTHOR
       Yann Collet

zstd 1.3.4			  2018-01-27			       ZSTD(1)

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | DICTIONARY BUILDER | BENCHMARK | ADVANCED COMPRESSION OPTIONS | BUGS | AUTHOR

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=zstd&manpath=FreeBSD+12.0-RELEASE+and+Ports>

home | help