Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
JIGDO-FILE(1)							 JIGDO-FILE(1)

NAME
       jigdo-file  -  Prepare  files for Jigsaw	Download (distribution of huge
       files, e.g. CD images).

SYNOPSIS
       jigdo-file   COMMAND
	[ --image=cdrom.iso ] [	--jigdo=cdrom.jigdo ] [	 --template=cdrom.tem-
       plate ] [ --force ] [ MORE OPTIONS ] [ FILES ...	| --files-from=f ]
	Common COMMANDs: make-template,	make-image, verify

DESCRIPTION
       Jigsaw  Download,  or  short  jigdo, is a scheme	developed primarily to
       make it easy to distribute huge filesystem images (e.g. CD (ISO9660) or
       DVD  (UDF)  images)  over  the  internet, but it	could also be used for
       other data which	is awkward to handle due to its	size, like audio/video
       files or	large software packages.

       jigdo tries to ensure that the large file (always called	image from now
       on) is downloaded in small parts	 which	can  be	 stored	 on  different
       servers.	 People	 who  want  to download	the image do so	by telling the
       jigdo(1)	(NOT IMPLEMENTED YET) download tool to	process	 one  `.jigdo'
       file;  using  it,  jigdo	downloads the parts and	reassembles the	image.
       jigdo-file is used to prepare the files for download.

       What makes jigdo	special	is that	the parts that are used	to reconstruct
       the  image  can	have  any size and content - they only need to be con-
       tained in a contiguous region anywhere in the image.

       For example, if you wish	to distribute an ISO9660 image which  contains
       a snapshot of an	FTP server, you	can instruct jigdo-file	to prepare the
       download	data in	such a way that	when people use	jigdo to download  the
       image,  jigdo actually fetches the individual files from	the FTP	server
       and assembles them into an exact	copy of	your image - during the	 down-
       load!  (If  the image is	not a filesystem dump, you can use split(1) to
       create the small	parts that the image will be reassembled from.)

       You are completely free to choose where the individual parts of the im-
       age  are	 stored: They may be in	entirely different directories on dif-
       ferent servers (e.g. because  of	 storage/bandwidth  constraints),  but
       this is invisible to the	people downloading your	image. The information
       about available servers only needs to be	added to the `.jigdo' file  by
       you before distributing it.

       The  `DETAILS'  section	below  contains	technical details on how jigdo
       works. The `EXAMPLES' section lists a number of	common	scenarios  and
       may help	you to get an idea of what jigdo is useful for.

OPTIONS
       Many  options  are specific to a	particular COMMAND; the	ones below are
       general or used by several commands. Further options are	 listed	 below
       with  the individual commands. All options are silently ignored if they
       are not applicable to the current command. For any BYTES	parameters  to
       options,	 you  can  append  one	of  the	letters	`k', `M' or `G'	to the
       amount you specify, to indicate kilobytes, megabytes or gigabytes.

       -h --help
	      Output short summary of commands and options.

       -H --help-all
	      Output complete summary of commands and options.

       -v --version
	      Output program version.

       -i --image=cdrom.iso
	      Specify location of the file containing the image. The image  is
	      the large	file that you want to distribute.

       -j --jigdo=cdrom.jigdo
	      Specify  location	 of  the Jigsaw	Download description file. The
	      jigdo file is a human-readable file generated by jigdo-file,  to
	      which you	add information	about all the servers you are going to
	      upload the files to.  jigdo will download	this file as the first
	      step of retrieving the image.

       -t --template=cdrom.template
	      Specify location of the image `template' file. The template file
	      is a binary file generated by jigdo-file,	it  contains  informa-
	      tion  on	how  to	 reassemble  the image and also	(in compressed
	      form) all	the data from the image	which was not found in any  of
	      the parts.

	      Depending	 on  the  command,  each  of these three files is used
	      sometimes	for input, sometimes for output. If the	file is	to  be
	      used for output for a particular command and the output file al-
	      ready exists, jigdo-file exits with an error, unless --force  is
	      present.

	      In  most	cases,	you will only need to specify one out of -i -j
	      -t, because any missing filenames	will be	deduced	from  the  one
	      you  specify. This is done by first stripping any	extension from
	      the supplied name	and then appending nothing (if deducing	 --im-
	      age), `.jigdo' or	`.template'.

       -r --report=default|noprogress|quiet|grep
	      Control  how  verbose the	program	is, and	what format the	output
	      has: noprogress is the same as default except that no `x%	 done'
	      progress	messages  are  printed.	 quiet restricts the output to
	      what is absolutely necessary, mostly error  messages.   grep  is
	      only  different  from  default for the make-template command: It
	      enables output in	a simple `_offset_  _file_'  format  which  is
	      useful when searching for	binary files in	other binary files.

       -f --force
	      Overwrite	existent output	files without complaining.

       --no-force
	      This is the default. Refuse to overwrite existent	output files.

       -c --cache=jigdo-cache.db
	      jigdo-file  usually needs	to read	the entire contents of all the
	      FILES you	specify. If you	use it repeatedly  (e.g.  because  you
	      make  a new CD image available daily), caching the file informa-
	      tion will	increase the program's speed significantly. The	 cache
	      file  is automatically created if	it is not yet present. Data is
	      usually both read	from and written to it.

       --no-cache
	      This is the default. Do not use a	cache.

       --cache-expiry=SECONDS
	      Set maximum age of cache entries.	Any entries  older  than  this
	      will  be removed from the	cache. The default is 30 days. You can
	      append one of the	letters	`h', `d',  `w',	 `m',  `y'  to	denote
	      hours,  days,  weeks,  months or years, respectively. A value of
	      `0' or `off' disables expiry, so that all	entries	will  stay  in
	      the  cache forever. See the section `CACHE FILES'	below for more
	      information.

       --readbuffer=BYTES
	      Set size of internal buffers. The	default	is 128k	- if you  have
	      a	 fast  disc, increasing	this value may make jigdo-file faster,
	      but in general, changing it is not necessary.

       --md5-block-size=BYTES
	      Uninteresting internal parameter.	 Set size of blocks into which
	      files are	subdivided. The	default	is 128k. If you	change it, any
	      cache file will have to be regenerated.  Internally,  jigdo-file
	      may choose to use	a slightly larger or smaller value.

       -T --files-from=file
	      Read  file and directory names from the specified	file.  If file
	      is `-', read names from standard input. Each line	in the file is
	      taken  as	 a name, so the	names may contain spaces, but not new-
	      line characters. An empty	line causes jigdo-file to stop reading
	      from the file.

	      find(1)  is  a powerful tool for generating file lists, but make
	      sure to use `find	-type f' if possible - otherwise, if  you  in-
	      struct  find  to	output	both  a	filename and a symlink to that
	      filename,	jigdo-file will	read the file contents twice.

       --hex  Output checksums in hexadecimal instead of  Base64-like  format.
	      This  should not be used with the	make-template command, because
	      the resulting `.jigdo' file violates the `.jigdo'	 file  format.
	      Its  intended  use is to make jigdo-file more interoperable with
	      other Unix shell utilities like md5sum(1).

       --no-hex
	      This is the default. Use jigdo's	own  Base64-like  encoding  of
	      checksums.

       --debug[=help|=all|=UNIT,~UNIT... ]
	      Switch  on or off	debugging output. Just `--debug' is equivalent
	      to `--debug=all'.	The argument is	a comma-separated list of unit
	      names  for  which	debugging output is to be enabled, or disabled
	      if the name is preceded by `~'. The special name `all' means all
	      units.  By  default, debugging output is switched	off except for
	      the units	`assert' and `general'.	The exact  list	 of  available
	      units  for which debugging can be	switched on depends on whether
	      jigdo was	compiled with debugging	support	 -  the	 list  can  be
	      printed with `--debug=help'.

       FILES  Names  of	 files	or  directories	to use as input. These are the
	      parts that are contained in the image. In	case one of the	 names
	      is  a directory, the program recursively scans the directory and
	      adds all files contained in it. While  doing  this,  it  follows
	      symbolic links, but avoids symlink loops.

	      If  one of the filenames starts with the character `-', you must
	      precede the list of files	with `--'. A value of `-' has no  spe-
	      cial  meaning in this list, it stands for	a file whose name is a
	      single hyphen.

COMMANDS
       The command name	is the first non-option	argument passed	to jigdo-file.
       Most commands have short	abbreviations as well as long names. The short
       command names should not	be used	in scripts - there may be incompatible
       changes to them in the future!

   MAKE-TEMPLATE, MT
       Reads  image  and  FILES, creates `.jigdo' and `.template'. This	is the
       main functionality of jigdo-file.

       It is possible to specify both --image=-	and  --files-from=-.  In  this
       case,  first  the  list	of  files is read from standard	input until an
       empty line is encountered. Everything following it is assumed to	be the
       image  data.  This  can be useful if you	use mkisofs(1) or similar pro-
       grams that can output the complete image	on their standard output,  be-
       cause there is no need to store the image on disc temporarily.

       If  a FILES argument contains the characters `//' (Unix)	or `\.\' (Win-
       dows), this has special meaning.	In the final  jigdo  file  that	 users
       will download, each of the parts	is referenced in the `[Parts]' section
       with a URI of the form `Label:some/filename'. (See  `FORMAT  OF	.JIGDO
       FILES' below for	a detailed description.) The `[Servers]' section gives
       a mapping of labels to servers on the internet, with  lines  like  `La-
       bel=http://myserver.org/jigdofiles/'.  Using  this  information,	 jigdo
       will  create  the  final	 download  URI	for  the   part,   `http://my-
       server.org/jigdofiles/some/filename'.   Specifying `//' (or `\.\') in a
       file or directory name serves to	`cut off' the names at the  right  di-
       rectory	level.	For  example, if the Unix path of one of your FILES is
       `/path/some/filename', you can tell jigdo-file to  cut  off  after  the
       `/path' by passing it the argument `/path//some/filename', or `/path//'
       if you want the whole directory scanned.	The path names need not	be ab-
       solute; `somedirectory//' is also possible.

       --label Label=/path
	      Specify a	name to	use as the label name for a path on disc. (In-
	      fluences the output jigdo	file.) If you used `//'	in  the	 FILES
	      arguments	 as  described	above, jigdo-file will by default pick
	      label names automatically	(`A', `B' etc.). With this option, you
	      can  give	labels more meaningful names. Note that	the label name
	      will only	be used	if one or more FILES begin with	`/path//'.

	      Try to use label names that start	with uppercase characters,  to
	      disambiguate  them  clearly  from	 protocol  names  like `http',
	      `ftp'.

       --uri Label=http://some.server.org/
	      By default, using	--label	as described above will	cause lines of
	      the  form	 `Label=file:/path/'  to be written to the `[Servers]'
	      section of the output jigdo file.	If you want  to	 override  the
	      `file:'	  URI	  so	 that	  the	 line	 reads	  `La-
	      bel=http://some.server.org/', you	can do so by specifying	 --uri
	      along with --label. Giving just --uri Label=... without the cor-
	      responding --label Label=... has no  effect,  and	 even  if  you
	      specify  both, an	entry is only added to the `[Servers]' section
	      if the label is referenced by at least one `[Parts]' entry.

	      The supplied value is not	quoted by the program; if it  contains
	      characters  such as space	or any of the characters #"'\ then you
	      must quote it.  (Under Unix, you may need	 to  quote  the	 value
	      twice  to	 also  protect it from the shell, e.g. \\\\ or '\\' to
	      get a single backslash in	the URI.)

	      The mapping specified with an --uri option is ignored if	it  is
	      already present in the output jigdo file.

	      Users  of	 the Windows version may notice	that the `\' directory
	      separators are converted into `/'	in the `file:' URIs  that  are
	      generated	 by  default.  This is done to increase	cross-platform
	      compatibility of `file:' - the print-missing command of the Win-
	      dows  version  will automatically	re-convert the characters when
	      it prints	the URIs. In case you supply your own `file:' URIs un-
	      der Windows using	--uri, you must	also exchange `/' and `\'.

       -0 to -9
	      Set  amount  of compression in the output	template file, from -0
	      (no compression) to -9 (maximum compression). The	default	is -9,
	      which  can  make the template generation quite slow. By default,
	      the compression algorithm	used is	the same as for	gzip(1).

       --gzip and --bzip2
	      Choose between the gzip and bzip2	 compression  algorithms.  The
	      default is gzip. Bzip2 usually gives a better compression	ratio,
	      but compression is significantly slower than with	gzip.

       --min-length=BYTES
	      Set minimum length of a part for jigdo-file to look  for	it  in
	      the  image.   The	 default  is  1k. Parts	smaller	than this will
	      never be found in	the image, so their data will be  included  in
	      the  template  file.  The	 search	algorithm used requires	such a
	      minimum length, otherwise	template generation could  become  ex-
	      tremely  slow.  If  you  know  for  sure that all	your FILES are
	      larger than a certain  amount,  you  can	increase  jigdo-file's
	      speed slightly by	specifying the amount with this	option.	 There
	      is a hard-wired absolute minimum of 256 bytes -  anything	 lower
	      will silently be set to 256.

       --merge=FILE
	      Include  the  contents  of FILE in the output `.jigdo' file. The
	      file can contain data which you want added to  the  output  (for
	      example,	a  `[Servers]'	section	with a list of your servers as
	      entries),	or it can be the jigdo file output by an  earlier  run
	      of jigdo-file.

	      It  is  possible to specify the same file	for input with --merge
	      and for output with --jigdo. However, you	will also need to  use
	      --force  to  make	 the  program overwrite	the old	version	of the
	      jigdo file with the new one.  FILE can be	`-' for	 standard  in-
	      put.

	      When  adding  new	 information  to the supplied file, jigdo-file
	      will not insert new lines	into the `[Parts]' section if an entry
	      for  the	same  MD5  checksum (but not necessarily with the same
	      URI!) already exists, and	it will	not insert new lines into  the
	      `[Servers]'  section if a	completely identical entry already ex-
	      ists.

	      When reading in the existing FILE,  the  behaviour  is  slightly
	      different:  The  program preserves entries in the	`[Parts]' sec-
	      tion with	identical checksum, but	different URIs.	For completely
	      identical	 entries  (same	 checksum  and URI), only one entry is
	      preserved	and the	duplicates are removed.	The  `[Servers]'  sec-
	      tion is left untouched.

       --image-section
	      This  is the default. Causes jigdo-file to add an	`[Image]' sec-
	      tion to the `.jigdo' file.

	      As an exception, a new `[Image]' section is not added if you use
	      --merge and the file to merge contains an	`[Image]' section with
	      a	line which reads `Template-MD5Sum=' (end  of  line  after  the
	      `=').  In	 this case, the	generated template data's MD5 checksum
	      value is just added after	the `='	of the first line of this form
	      in  the  file - no whole new `[Image]' section is	appended. This
	      behaviour	is useful because it allows you	to pass	via --merge an
	      `[Image]'	 section  with arbitrary content and then have the MD5
	      checksum automatically added by jigdo-file. The section  `FORMAT
	      OF `[Image]' section contents in greater detail.

       --no-image-section
	      Do  not  include	an `[Image]' section in	the `.jigdo' file. You
	      need to add one yourself if you use this option. However,	 doing
	      that  is not easy	(you also need to add a	`Template-MD5Sum' line
	      with the correct checksum, or jigdo will complain),  so  use  of
	      this option is discouraged.

       --servers-section
	      This is the default. Causes jigdo-file to	add a `[Servers]' sec-
	      tion to the `.jigdo' file. This  default	section	 uses  `file:'
	      URIs,  which  allows  for	immediate reassembly of	the image from
	      the local	filesystem, and	is also	useful if you want to edit the
	      file manually and	replace	the `file:' URIs with other URIs.

       --no-servers-section
	      Do  not  add  a  `[Servers]'  section at the end of the `.jigdo'
	      file. Useful e.g.	if you are going to append the section with  a
	      script.

       --match-exec=SHELLCOMMAND
	      Whenever a file is found in the image, execute the supplied com-
	      mand string by passing it	to a shell.  jigdo-file	sets up	a num-
	      ber  of  environment  variables  with information	about the file
	      match. For example, if the file `/path//a/b/file'	was  found  in
	      the  image  and  `Label:a/b/file'	 is going to be	written	to the
	      `.jigdo' file:

	      o	LABEL="Label" -	Name of	the label for the  file.  The  example
		assumes	 that  `--label	Label=/path' was specified by you.  In
		the absence of such an option, LABEL will be set but empty.

	      o	LABELPATH="/path/" - The path corresponding to the  label,  or
		in  other  words,  the	prefix of the matched file's path that
		will not appear	in the output `.jigdo' file. Is	set even with-
		out any	`--label' option present.  Ends	with a slash.

	      o	MATCHPATH="a/b/"  - The	rest of	the path, without the leafname
		of the matched file. Is	either empty or	ends with a slash.

	      o	LEAF="file" - The leafname of the matched file.

	      o	MD5SUM="lNVdUSqbo2yqm33webrhnw"	- The md5sum  of  the  matched
		file, in Base64-like format.

	      o	FILE="/path//a/b/file" - For convenience, the complete path of
		the file. The variable	is  always  set	 to  $LABELPATH$MATCH-
		PATH$LEAF.

       Please  be careful to correctly quote the string	passed to this option,
       otherwise your supplied command will not	work with filenames that  con-
       tain  spaces.  As  an  example, to create a backup of hard links	to the
       matched files, use the following	option:	--match-exec='mkdir -p	"${LA-
       BEL:-.}/$MATCHPATH" && ln -f "$FILE" "${LABEL:-.}/$MATCHPATH$LEAF"'

       By  default,  no	 command  is executed. Use --match-exec="" to remove a
       command string which was	set with an earlier use	of this	option.

       --greedy-matching
	      This is the default. Imagine that	your  image  contains  a  .tar
	      file which in turn contains another file x, and that you provide
	      both the .tar and	the files inside it on the command line.  When
	      jigdo-file  scans	 the image, it encounters the beginning	of the
	      .tar file, and then the file x.

	      At this point, a decision	must be	made: Should the smaller  file
	      x	 be  recorded as matched, or should it be ignored in favour of
	      the larger (and thus better) match of the	 .tar  file?  Unfortu-
	      nately,  at  this	point it is not	clear whether there will actu-
	      ally be a	full match of the .tar,	so  by	default,  the  program
	      prefers the small	match.

       --no-greedy-matching
	      In the case where	a large	partial	match is present and a shorter
	      match has	been confirmed,	ignore the small match.	(See  the  op-
	      tion above.)

   MAKE-IMAGE, MI
       Reads  `.template'  and FILES, creates image (or	`imagename.tmp'). Pro-
       vides a rudimentary way of reassembling images -	jigdo is usually  bet-
       ter  suited  for	 this task. However, in	contrast to jigdo, no `.jigdo'
       file is required.

       If the image is to be written to	a file (and not	to  standard  output),
       it is possible to create	the image in several steps, with several invo-
       cations of `jigdo-file make-image', as follows: You first invoke	jigdo-
       file,  specifying as many files as are available	at this	time. The pro-
       gram scans the files, and those that are	contained  in  the  image  are
       copied to a temporary file, whose name is formed	by appending `.tmp' to
       the image filename.

       For all further files which could be parts of  the  image,  you	repeat
       this process. As	soon as	all parts are present, the temporary file will
       be truncated slightly (to delete	some administrative data  that	jigdo-
       file  appends at	the end) and renamed to	the final image	name. The pos-
       sibility	of reassembling	the image in several steps is especially  use-
       ful for gathering files from removable media, e.g. several older	CDs.

       Scripts	using make-image can detect whether image creation is complete
       by checking the exit status: 0 signals successful creation,  whereas  1
       means  that  more  files	need to	be supplied. Other errors result in an
       exit status of 2	(`recoverable',	e.g. file not found) or	3  (non-recov-
       erable, e.g.  write error).

       --check-files
	      This  is	the default. Whenever any part is copied to the	image,
	      re-check its checksum against the	checksum stored	 in  the  tem-
	      plate.  It  is recommended that you leave	this switched on, even
	      if it slows down image creation a	bit.

       --no-check-files
	      Do not check files' checksums when copying them  to  the	image.
	      This  can	be safely used when no cache file is used (which means
	      that files will be written to the	image immediately after	 being
	      scanned)	or  the	 whole	image is checked later with the	verify
	      command.

   PRINT-MISSING, PM
       Reads `.jigdo', `.template' and (if present) `imagename.tmp', outputs a
       list of URIs still needed to completely reassemble the image.

       Together	 with  the make-image command, this provides most of the func-
       tionality of jigdo on the command line.

       For each	part that is not yet present in	the temporary image file,  the
       file  checksum is looked	up in the `[Parts]' section of the jigdo file.
       Any label in the	corresponding entry is then expanded according to  the
       label  definitions  in  the `[Servers]' section and printed on standard
       output. jigdo allows you	to specify several alternative	locations  for
       each  label  in	this  section,	but print-missing will only output the
       first one for each missing part.

       If the checksum cannot be found in the `[Parts]'	section	 (this	Should
       Not  Happen  unless you deleted that section), a	lookup is instead made
       for `MD5Sum:_checksum_',	just like with jigdo. (Thus, if	 you  want  to
       get rid of the `[Parts]'	section, you can do so if you rename each part
       to its own checksum.)

       --uri Label=http://some.server.org/
	      Override the entries in the `.jigdo' file	for any	label  with  a
	      URI of your choice. With the example above, a `[Parts]' entry of
	      `Label:some/filename'	 will	    cause	the	  line
	      `http://some.server.org/some/filename' to	be printed.

	      The  supplied value is not quoted	by the program;	if it contains
	      characters such as space or any of the characters	#"'\ then  you
	      must  quote  it.	 (Under	 Unix, you may need to quote the value
	      twice to also protect it from the	shell, e.g. \\\\  or  '\\'  to
	      get a single backslash in	the URI.)

   PRINT-MISSING-ALL, PMA
       Just  like  print-missing,  this	 command  outputs a list of URIs still
       needed to completely reassemble the  image.  However,  all  alternative
       download	 locations are printed instead of just one. In the output, the
       URIs for	a file are separated from other	files' URIs with blank	lines.
       The --uri option	has the	same effect as for print-missing.

   VERIFY, VER
       Reads  image  (presumably  generated  with make-image) and `.template',
       checks for correct checksum of image.

       The template data does not only contain	checksums  of  the  individual
       parts, but also of the image as a whole.	 make-image already performs a
       number of internal checks, but if you like, you can additionally	 check
       the image with this command.

   SCAN, SC
       Reads all the FILES and enters them into	the cache, unless they are al-
       ready cached. The --cache option	must be	present	for this command.

       --no-scan-whole-file
	      This is the default. This	only causes the	first --md5-block-size
	      bytes  of	 each  file  to	be read. If the	cache is used later by
	      jigdo-file make-image, the rest of the file will	be  read  once
	      these first bytes	are recognized in the input image.

       --scan-whole-file
	      Immediately  read	the entire file	contents and store them	in the
	      cache.

   MD5SUM, MD5
       Reads all the FILES and prints out MD5  checksums  of  their  contents.
       This command is quite similar to	md5sum(1), except that the checksum is
       output in the Base64-like encoding which	 is  also  used	 elsewhere  by
       jigdo-file.

       The  FILES  arguments  are  processed in	the same way as	with the other
       commands, which means that recursion automatically takes	place for  any
       arguments  that are directories,	and that symbolic links	are not	listed
       except when the file(s) they point to are not reachable directly.

       In the checksum list printed on standard	output,	only the part  of  the
       filename	 following  any	 `//'  (or  `\.\'  on Windows) is printed. Any
       --cache will be used for	querying files'	MD5 checksums  and/or  writing
       the checksums of	scanned	files.

   LIST-TEMPLATE, LS
       Reads  a	 `.template'  file and outputs low-level information about the
       image and all parts contained  in  it,  including  offset,  length  and
       checksum.

       You can also use	this command with temporary image files	(by specifying
       something like --template=imagename.tmp)	- in  that  case,  the	output
       also  distinguishes  between  parts that	have been written to the image
       and parts that haven't.

       The exact output	format may change incompatibly between different jigdo
       releases.  The following	different types	of lines can be	output.	`have-
       file' only occurs for `.tmp' files, indicating a	file that has  already
       been successfully written to the	temporary file:

       in-template offset-in-image length
       need-file offset-in-image length	file-md5sum filestart-rsyncsum
       have-file offset-in-image length	file-md5sum filestart-rsyncsum
       image-info image-length image-md5sum rsyncsum-size

DETAILS
       Jigsaw  Download	 was  created  with the	format of ISO9660 CD images in
       mind - however, the following also applies  to  many  other  filesystem
       formats,	 as well as to `tar' archives and uncompressed `zip' archives.
       A CD image contains both	 information  for  organizing  the  filesystem
       (header with disc name etc., ISO9660 directory data, data of extensions
       such as Joliet or RockRidge, zero padding) and the files	 contained  on
       the CD. An important property that jigdo	relies on is that each file is
       stored in one contiguous	section	of the image; it is not	split into two
       or more parts.

       When  jigdo-file	 is given a number of files that might be contained in
       an image, it detects whether any	of  the	 files	are  present  using  a
       `rolling	 checksum' inspired by the one used by rsync(1). The resulting
       data is written to the `.template' file:	If  a  section	of  the	 image
       could  not  be matched (e.g. it was directory information), the data is
       compressed and written directly to the template.	However, if a matching
       file  was found,	its data is omitted from the template. Instead,	only a
       reference (an MD5 checksum of the file) is inserted in the template.

       Note that the template data only	contains binary	data, it does not con-
       tain  any  filenames  or	URIs, since it cannot be easily	edited in case
       any of these values need	to be changed. All that	information is	stored
       in  the	`.jigdo'  file,	a text file to which you can add URLs for your
       server(s). The jigdo file provides a mapping for	each MD5  checksum  to
       one or more alternative download	locations for the corresponding	part.

       Apart  from  the	 mapping of MD5	sums to	URIs, the jigdo	file also con-
       tains an	URI pointing to	a download location  for  the  template	 file.
       This  way, the jigdo download tool only needs to	be given one URI (that
       of the `.jigdo' file) to	be able	to download and	 reassemble  the  com-
       plete image.

FORMAT OF .JIGDO FILES
       The  overall  format of `.jigdo'	files follows that of `.ini' files, as
       also used by the	Gnome and KDE projects for some	data. The file is  or-
       ganized	into  sections,	 each  of  which is preceded by	a line reading
       `[Sectionname]'.	 Within	 each  section,	 lines	have  the  form	  `La-
       bel=Value'.  Such  lines	 are also called `entries' below. All `.jigdo'
       files use UTF-8 as their	character encoding.

       Comments	are introduced with the	`#' character and extend to the	end of
       the line. Whitespace is ignored at line start and end as	well as	to the
       left and	right of section names and the `='  in	entries.  Furthermore,
       the jigdo utilities split up the	text of	the entry value	(i.e. the part
       after the `=') into whitespace-separated	 words,	 much  like  the  Unix
       shell.  Single '' and double "" quotes can be used to prevent that e.g.
       URIs containing whitespace are split apart. Similarly, characters  with
       special meaning (the characters '"#\ and	space/tab) must	be quoted with
       \ to appear in the value. As with the shell, there is a difference  be-
       tween  '	 ' and " ": Within ' ',	the characters "#\ and whitespace lose
       their special meaning and become	ordinary  characters,  whereas	within
       " ", only the characters	'# and whitespace lose their special meaning -
       in other	words, backslash escapes still work inside " ",	but not	' '.

       `.jigdo'	files can optionally be	compressed  with  gzip(1).  jigdo-file
       always  outputs	uncompressed  files,  which you	can compress yourself.
       jigdo-lite supports single uncompressed and compressed files.

       (Behaviour which	may change in the future and which should not  be  re-
       lied  upon:  jigdo  additionally	 supports  any	number of concatenated
       plaintext and gzipped parts in the files	- for example,	you  can  com-
       press  a	 `.jigdo'  file	and then add a couple of lines of uncompressed
       data to the end.)

       In all cases, the `.gz' extension should	be removed from	the filename -
       the tools will determine	automatically from the file contents whether a
       file is compressed or not.

       Below is	a description of the individual	section	names used by jigdo.

   JIGDO SECTION
       [Jigdo]
       Version=1.1
       Generator=jigdo-file/1.0.0

       Information about the version of	the jigdo file format  used,  and  the
       program	that  generated	 it.  There  should  be	 one  such section per
       `.jigdo'	file.

   IMAGE SECTION
       [Image]
       Filename="filename for saving on	user's disc"
       Template="URI where to fetch template file"
       Template-MD5Sum=OQ8riqT1BuyzsrT9964A7g
       ShortInfo=single-line description of the	image (200 characters max.)
       Info=long description (5000 characters max.)

       The value for the `Template' entry can be either	an  URL	 (absolute  or
       relative	 to  the  URL  of the jigdo file) or a string of the form `La-
       bel:pathname' (UNIMPLEMENTED), as described below.

       The `Template-MD5Sum' entry is added by jigdo-file  and	specifies  the
       MD5  checksum of	the generated `.template' file.	It is used by jigdo to
       detect cases where the downloaded template data is corrupted or belongs
       to a different image.

       Unlike other entry values, the values of	the `ShortInfo'	and `Info' en-
       tries are not split up into words, instead all quoting is preserved.

       The value of the	`Info' entry is	special	in that	jigdo(1)  can  option-
       ally parse XML markup it	contains. If the markup	has errors such	as un-
       balanced/unsupported tags, the string is	displayed  literally,  without
       XML  parsing.  Supported	 tags  are  <b></b>  (bold), <i></i> (italic),
       <tt></tt> (typewriter font), <u></u> (underline),  <big></big>  (larger
       font),  <small></small> (smaller	font) and <br/>	(linebreak). Supported
       entities	include	&lt; (`<'), &gt; (`>') and &amp; (`&').	Note that  the
       whole `Info' entry must be on one line in the jigdo file.

       This  section  may  occur  multiple times, but all except the first one
       will be ignored.	This is	useful e.g. when creating a `.jigdo' file  for
       a  DVD image when you already have `.jigdo' files for CDs with the same
       content:	You can	simply `[Include]' (see	below) the CDs'	jigdo files at
       the end of the DVD jigdo	file, after its	`[Image]' section.

   PARTS SECTION
       [Parts]
       xJNkjrq8NYMraeGavUpllw=LabelA:part0
       GoTResP2EC6Lb_2wTsqOoQ=LabelA:part1
       kyfebwu6clbYqqWUdFIyaw=LabelB:some/path/part2
       -J9UAimo0Bqg9c0oOXI1mQ=http://some.where.com/part3

       All lines in the	section, which provides	the mapping from MD5 checksums
       to URIs,	have the same format: On the left side of the `=' the checksum
       (encoded	 with  a  Base64-like  encoding)  is given, and	on the right a
       string corresponding to the part	with this checksum; either a  complete
       URI  or	a  string of the form `Label:pathname',	which is expanded into
       one or more URIs	by looking up the definition(s)	for the	Label  in  the
       `[Servers]' section.

       In case a particular MD5	checksum cannot	be found in any	`[Parts]' sec-
       tion by jigdo, the program will perform a  lookup  for  `MD5Sum:_check-
       sum_', e.g. for `MD5Sum:xJNkjrq8NYMraeGavUpllw' if you deleted the line
       for `part0' above.

       A checksum appearing multiple times in this section indicates  alterna-
       tive download locations for the part.

       There may be any	number of `[Parts]' sections in	the file; they are all
       considered when looking up MD5 checksums.

       jigdo-file always puts the `[Parts]' section at the end	of  the	 file,
       and it even rearranges any file specified with --merge to have only one
       such section at the end.	This is	done to	allow jigdo to display the in-
       formation  from	the  `[Image]'	section	while the rest of that file is
       still being downloaded.

   SERVERS SECTION
       [Servers]
       LabelA=http://myserver.org/
       LabelA=ftp://mirror.myserver.org/
       LabelB=LabelC:subdirectory/
       LabelC=http://some.where.com/jigdo/

       All lines in the	section, which provides	the mapping from server	labels
       to  server locations, have the same format: On the left side of the `='
       the label name is given,	and on the right the value to expand the label
       name to.

       A  label	name appearing multiple	times in this section indicates	alter-
       native download locations for the parts	that  use  the	label  in  the
       `[Parts]'  section.  This notation makes	it very	easy to	add mirrors to
       the jigdo file.

       As shown	by the example above, the label	values may  themselves	refer-
       ence  other labels. In this case, the entry `LabelB:some/path/part2' in
       the `[Parts]' section will expand to  `http://some.where.com/jigdo/sub-
       directory/some/path/part2'.   Loops  in the label definitions result in
       undefined behaviour and must be avoided.

       There may be any	number of `[Servers]' sections in the file;  they  are
       all   considered	 when  looking	up  labels.  Either  of	 `[Parts]'  or
       `[Servers]', but	not both, can be omitted from the jigdo	file.

   COMMENT SECTION
       [Comment]
       Any text, except	that lines must	not begin with `['.

       All text	following a `[Comment]'	or `[comment]' line is ignored,	up  to
       the next	line with a section label.

   INCLUDE DIRECTIVE
       [Include	http://some.url/file.jigdo]

       Lines  of this form cause the content of	the specified jigdo file to be
       downloaded and parsed just like the main	jigdo file. The	effect will be
       the  same  as  copying the included file's contents into	the file which
       contains	the include directive. (Exception: Any relative	URLs  are  al-
       ways  resolved  using  the  URL of the `.jigdo' file that contains that
       relative	URL.)

       The URL argument	can be an absolute or relative URL.  Relative URLs are
       assumed	to be relative to the URL of the jigdo file which contains the
       include directive. Includes can be nested, but it is an error to	create
       a  loop	of  include  directives. It is not possible to use URLs	of the
       form `Label:pathname'.

       The URL cannot be quoted	with "". Any `]' characters  in	 the  argument
       must be escaped as `%5D', and any spaces	as `%20'.

       Include	directives  are	 only  supported by jigdo, they	are ignored by
       jigdo-lite.

       An include directive terminates any previous section, but it  does  not
       start  a	 new one. In other words, a new	section	must always be started
       after the include line, jigdo does not allow normal entries  to	appear
       below the `[Include]'.

CACHE FILES
       Any file	specified with the --cache option is used to store information
       about the FILES presented to jigdo-file.	When  querying	the  cache,  a
       file  is	 considered  unchanged	(and  the cached data is used) only if
       filename, file size and last modification time (mtime)  match  exactly.
       For  the	filename match,	not the	entire file name is used, but only the
       part following any `//',	so that	any changes to	the  part  before  the
       `//' will not invalidate	the cache.

       Old cache entries are removed from the cache if they have not been read
       from or written to for the amount of time  specified  with  --cache-ex-
       piry.  Entries  are  not	immediately removed from the cache if the file
       they refer to no	longer exists -	this makes it possible to cache	infor-
       mation about files on removable media.

       Cache expiry only takes place after jigdo-file has done its main	work -
       if any old entries are accessed before expiry takes place, they will be
       kept.  For example, if the program is run using the default expiry time
       of 30 days, but accesses	a cache	file with entries generated  2	months
       ago,  then  entries  in	that  cache will be considered,	and only those
       cache entries that were not needed during the program run will  be  ex-
       pired.

       Due to a	peculiarity of the underlying database library (libdb3), cache
       files never shrink, they	only grow. If a	large number  of  entries  was
       expired	from your cache	file and you want it to	shrink,	you can	either
       just delete it (of course then everything will have to be  regenerated)
       or  use the utilities accompanying libdb3 to dump and restore the data-
       base, with a  command  like  `db3_dump  old-cache.db  |	db3_load  new-
       cache.db'.  For	Debian,	 these	programs  are  supplied	in the package
       `libdb3-util'.

       If a different --md5-block-size is specified, the entire	file needs  to
       be  re-read  to	update its cache entry.	If a different --min-length is
       specified, only the first `md5-block-size' bytes	of the file need to be
       re-read.

EXAMPLES
   PREPARING YOUR CD IMAGE FOR DISTRIBUTION
       You  have  created a CD image `image.iso' from some of the files	stored
       in the directory	`/home/ftp' on your harddisc, which is also  available
       online  as `ftp://mysite.org'.  As you don't want to waste space	by ef-
       fectively hosting the same data twice (once as files on the FTP server,
       once inside the image), and you are fed up with users' downloads	abort-
       ing after 200MB and their restarting the	download dozens	of times,  you
       decide to use jigdo. How	do you prepare the image for download?

       In fact,	only one command is necessary:

	      jigdo-file make-template --image=image.iso --jigdo=/home/ftp/im-
	      age.jigdo	--template=/home/ftp/image.template /home/ftp//	 --la-
	      bel Mysite=/home/ftp --uri Mysite=ftp://mysite.org/

       People  can  now	point jigdo at `ftp://mysite.org/image.jigdo' to down-
       load  your  image.  The	template  file	needs  to  be  accessible   as
       `ftp://mysite.org/image.template'.

       Note  that  nothing  prevents you from doing the	same for an FTP	server
       that isn't administrated	by you - in that case, you only	need  to  host
       the `.jigdo' and	`.template' files on your own server/homepage.

   PREPARING AN	ARBITRARY LARGE	FILE FOR DISTRIBUTION
       We  assume  that	 you  have a large file	that is	not a filesystem, e.g.
       `movie.mpeg'. Because of	space problems,	you  want  to  distribute  the
       data on two servers.

       In  this	case, the parts	of the image need to be	generated artificially
       with the	split command. For example, to create chunks of	4MB each,  use
       `split  -b  4m movie.mpeg part'.	Copy the resulting files `partXX' into
       two directories `1' and `2' that	you create, according to how you  want
       the  files  distributed between the servers. Next, create the jigdo and
       template	files with `jigdo-file	make-template  --image=movie.mpeg  1//
       2//'.  You  will	 need  to edit the `.jigdo' file and provide the right
       URIs for	the two	servers	that you are  going  to	 upload	 the  `partXX'
       files to.

   CUSTOMIZED VERSIONS OF IMAGES
       Because	it  is	possible to assign a different URI for each part of an
       image if	necessary, jigdo is very flexible. Only	 one  example  is  the
       possibility  of	customized versions of images: Suppose that someone is
       distributing a CD image,	and that you want to make a few	small  changes
       to  it  and  redistribute  your	own  version.  You download the	`offi-
       cial.iso' CD image with jigdo (passing it the URL of `official.jigdo'),
       write  it  to  CD-R, make your changes (say, adding files from the `my-
       files' directory	on  your  harddisc)  and  produce  your	 own  version,
       `myversion.iso'.	 Next, you instruct jigdo-file to create the jigdo and
       template	files for your modified	image, using the command

	      jigdo-file make-template --image=myversion.iso  /mnt/cdrom/  my-
	      files//  --label	My=myfiles/  --uri  My=http://my.homepage.net/
	      --merge=official.jigdo
       while `official.iso' is mounted under `/mnt/cdrom'. By  using  --merge,
       you  have told jigdo-file to take the contents of `official.jigdo', add
       to it a new `[Image]' section for `myversion.iso' and write the result-
       ing  jigdo  file	to `myversion.jigdo' - so now `myversion.jigdo'	offers
       two images for download,	the original version and  your	modified  ver-
       sion. (If you do	not want it to offer the official version, edit	it and
       remove the `[Image]' section that lists `official.iso'.)

       Now you can upload the `.jigdo' file, the `.template' file and also the
       files  in  `myfiles' to `http://my.homepage.net/'.  Thus, for people to
       download	your modified image, you do not	need to	 upload	 the  complete
       image contents to your web space, but only the changes you made!

       (In  case  you only made	very few changes, you could also omit the `my-
       files' parameter	in the command above, then all your changes end	up  in
       the new template	file.)

   COMBINING MANY JIGDO-MANAGED	IMAGES INTO ONE
       It  is  also  no	 problem to combine data from several sources that use
       jigdo. For example, if of five different	and unrelated servers each one
       distributes a different CD image	via jigdo, you can create a customized
       DVD image that contains the data	from all these CDs.  When  people  use
       jigdo  to  download  your  image,  the  individual files	on the DVD are
       fetched from the	same sources as	the original CDs.

       Consequently, even though you will be distributing  a  3.2GB  file  via
       your web	space, the actual amount of data that is stored	on your	server
       will only be in the order of several MBs.

BUGS
       For certain contents of one of the input	files, most notably a sequence
       of  zero	bytes longer than --min-length at the start of the file	and an
       area of zeros preceding the file	data in	the  image,  jigdo-file	 make-
       template	 may  fail  to find the	file in	the image. Unfortunately, this
       restriction cannot be avoided because the  program  could  become  very
       slow  otherwise.	If you use the --debug option, all instances of	jigdo-
       file discarding possible	matches	are indicated by lines containing  the
       word `DROPPED'.

       In  fact,  not  only  all-zeroes	files trigger this behaviour, but also
       files which contain at their start a long sequence of  short  identical
       strings.	 For  example,	both a file containing only `a'	characters and
       one containing `abcabcabcabc...'	are problematic.

SEE ALSO
       jigdo(1)	 (NOT  YET   IMPLEMENTED),   jigdo-lite(1),   jigdo-mirror(1),
       split(1)	 (or  `info  split'),  find(1)	(or  `info find'), mkisofs(1),
       md5sum(1)

AUTHOR
       Jigsaw Download <URL:http://atterer.net/jigdo/> was written by  Richard
       Atterer	<jigdo	atterer.net>, to make downloading of CD	ROM images for
       the Debian Linux	distribution more convenient.

				  19 May 2006			 JIGDO-FILE(1)

NAME | SYNOPSIS | DESCRIPTION | OPTIONS | COMMANDS | DETAILS | FORMAT OF .JIGDO FILES | CACHE FILES | EXAMPLES | BUGS | SEE ALSO | AUTHOR

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=jigdo-file&sektion=1&manpath=FreeBSD+12.1-RELEASE+and+Ports>

home | help