5.4. The Distribution Files

The second part of the Makefile describes the files that must be downloaded to build the port, and where they can be downloaded.

5.4.1. DISTVERSION/DISTNAME

DISTNAME is the name of the port as called by the authors of the software. DISTNAME defaults to ${PORTNAME}-${DISTVERSIONPREFIX}${DISTVERSION}${DISTVERSIONSUFFIX}, and DISTVERSION defaults to ${PORTVERSION} so override it only if necessary. DISTNAME is only used in two places. First, the distribution file list (DISTFILES) defaults to ${DISTNAME}${EXTRACT_SUFX}. Second, the distribution file is expected to extract into a subdirectory named WRKSRC, which defaults to work/${DISTNAME}.

Some vendor's distribution names which do not fit into the ${PORTNAME}-${PORTVERSION}-scheme can be handled automatically by setting DISTVERSION. PORTVERSION will be derived from it automatically.

Note:

Only one of PORTVERSION and DISTVERSION can be set at a time. If DISTVERSION does not derive a correct PORTVERSION, do not use DISTVERSION, set PORTVERSION to the right value and set DISTNAME with PORTNAME with either some computation of PORTVERSION or the verbatim upstream version.

Table 5.2. Examples of DISTVERSION and the Derived PORTVERSION
DISTVERSIONPORTVERSION
0.7.1d0.7.1.d
10Alpha310.a3
3Beta7-pre23.b7.p2
8:f_178f.17

Note:

PKGNAMEPREFIX and PKGNAMESUFFIX do not affect DISTNAME. Also note that if WRKSRC is equal to ${WRKDIR}/${DISTNAME} while the original source archive is named something other than ${PORTNAME}-${PORTVERSION}${EXTRACT_SUFX}, leave DISTNAME alone— defining only DISTFILES is easier than both DISTNAME and WRKSRC (and possibly EXTRACT_SUFX).

5.4.2. MASTER_SITES

Record the directory part of the FTP/HTTP-URL pointing at the original tarball in MASTER_SITES. Do not forget the trailing slash (/)!

The make macros will try to use this specification for grabbing the distribution file with FETCH if they cannot find it already on the system.

It is recommended that multiple sites are included on this list, preferably from different continents. This will safeguard against wide-area network problems. We are even planning to add support for automatically determining the closest master site and fetching from there; having multiple sites will go a long way towards helping this effort.

If the original tarball is part of one of the popular archives such as SourceForge, GNU, or Perl CPAN, it may be possible refer to those sites in an easy compact form using predefined macros (for example, SF, GNU or CPAN). Set MASTER_SITES to one of these values. Here is an example:

MASTER_SITES=	GNU/make

The older expanded format can still be used, although there really is no reason to do so:

MASTER_SITES=		${MASTER_SITE_GNU}
	MASTER_SITE_SUBDIR=	make

These values and variables are defined in /usr/ports/Mk/bsd.sites.mk. There are new entries added all the time, so make sure to check the latest version of this file before submitting a port.

Several magic macros exist for popular sites with a predictable directory structure. For these, just use the abbreviation and the system will try to guess the correct subdirectory automatically.

MASTER_SITES=	SF

If the guess is incorrect, it can be overridden as follows.

MASTER_SITES=	SF/stardict/WyabdcRealPeopleTTS/${PORTVERSION}

This can also be written as

MASTER_SITES=	SF
	MASTER_SITE_SUBDIR=	stardict/WyabdcRealPeopleTTS/${PORTVERSION}
Table 5.3. Popular Magic MASTER_SITES Macros
MacroAssumed subdirectory
APACHE_JAKARTA/dist/jakarta/${PORTNAME:S,-,,/,}/source
BERLIOS/${PORTNAME:L}
CHEESESHOP/packages/source/source/${DISTNAME:C/(.).*/\1/}/${DISTNAME:C/(.*)-[0-9].*/\1/}
DEBIAN/debian/pool/main/${PORTNAME:C/^((lib)?.).*$/\1/}/${PORTNAME}
GCC/pub/gcc/releases/${DISTNAME}
GH/${GH_ACCOUNT}/${GH_PROJECT}/legacy.tar.gz/${GH_TAGNAME}?dummy=/
GHC/downloads/${GH_ACCOUNT}/${GH_PROJECT}/
GNOME/pub/GNOME/sources/${PORTNAME}/${PORTVERSION:C/^([0-9]+\.[0-9]+).*/\1/}
GNU/gnu/${PORTNAME}
HORDE/pub/${PORTNAME}
LOGILAB/pub/${PORTNAME}
MATE/releases/${PORTVERSION:C/^([0-9]+\.[0-9]+).*/\1/}
MOZDEV/pub/mozdev/${PORTNAME:L}
CPAN/pub/CPAN/modules/by-module/${PORTNAME:C/-.*//}
PYTHON/ftp/python/${PYTHON_PORTVERSION:C/rc[0-9]//}
RUBYFORGE/${PORTNAME:L}
SAVANNAH/${PORTNAME:L}
SF/project/${PORTNAME:L}/${PORTNAME:L}/${PORTVERSION}

5.4.2.1. USE_GITHUB

If the distribution file comes from a specific commit or tag on GitHub for which there is no officially released file, there is an easy way to set the right DISTNAME and MASTER_SITES automatically. These variables are available:

Table 5.4. USE_GITHUB Description
VariableDescriptionDefaultMandatory
GH_ACCOUNTAccount name of the GitHub user hosting the projectnoneMandatory
GH_PROJECTName of the project on GitHub${PORTNAME} 
GH_TAGNAMEName of the tag to download (2.0.1, hash, ...) Using the name of a branch here is incorrect. It is possible to do GH_TAGNAME=${GH_COMMIT} to do a snapshot${DISTVERSION} 
GH_COMMITfirst 7 digits of the commit that generated GH_TAGNAME (see git-describe(1))noneMandatory

Example 5.1. Simple Use of USE_GITHUB

While trying to make a port for version 1.2.7 of pkg from the FreeBSD user on github, at https://github.com/freebsd/pkg, The Makefile would end up looking like this (slightly stripped for the example):

PORTNAME=	pkg
	    PORTVERSION=	1.2.7

	    USE_GITHUB=	yes
	    GH_ACCOUNT=	freebsd
	    GH_COMMIT=	f53e577

It will automatically have MASTER_SITES set to GH GHC and WRKSRC to ${WRKDIR}/freebsd-pkg-f53e577.


Example 5.2. More Complete Use of USE_GITHUB

While trying to make a port for the bleeding edge version of pkg from the FreeBSD user on github, at https://github.com/freebsd/pkg, The Makefile would end up looking like this (slightly stripped for the example):

PORTNAME=	pkg-devel
	    PORTVERSION=	1.3.0.a.20140411

	    USE_GITHUB=	yes
	    GH_ACCOUNT=	freebsd
	    GH_PROJECT=	pkg
	    GH_TAGNAME=	${GH_COMMIT}
	    GH_COMMIT=	6dbb17b

It will automatically have MASTER_SITES set to GH GHC and WRKSRC to ${WRKDIR}/freebsd-pkg-6dbb17b.


5.4.3. EXTRACT_SUFX

If there is one distribution file, and it uses an odd suffix to indicate the compression mechanism, set EXTRACT_SUFX.

For example, if the distribution file was named foo.tar.gzip instead of the more normal foo.tar.gz, write:

DISTNAME=	foo
	EXTRACT_SUFX=	.tar.gzip

The USES=tar[:xxx], USES=lha or USES=zip automatically set EXTRACT_SUFX to the most common archives extensions as necessary, see Chapter 15, Values of USES for more details. If neither of these are set then EXTRACT_SUFX defaults to .tar.gz.

Note:

As EXTRACT_SUFX is only used in DISTFILES, only set one of them..

5.4.4. DISTFILES

Sometimes the names of the files to be downloaded have no resemblance to the name of the port. For example, it might be called source.tar.gz or similar. In other cases the application's source code might be in several different archives, all of which must be downloaded.

If this is the case, set DISTFILES to be a space separated list of all the files that must be downloaded.

DISTFILES=	source1.tar.gz source2.tar.gz

If not explicitly set, DISTFILES defaults to ${DISTNAME}${EXTRACT_SUFX}.

5.4.5. EXTRACT_ONLY

If only some of the DISTFILES must be extracted—for example, one of them is the source code, while another is an uncompressed document—list the filenames that must be extracted in EXTRACT_ONLY.

DISTFILES=	source.tar.gz manual.html
	EXTRACT_ONLY=	source.tar.gz

When none of the DISTFILES need to be uncompressed, set EXTRACT_ONLY to the empty string.

EXTRACT_ONLY=

5.4.6. PATCHFILES

If the port requires some additional patches that are available by FTP or HTTP, set PATCHFILES to the names of the files and PATCH_SITES to the URL of the directory that contains them (the format is the same as MASTER_SITES).

If the patch is not relative to the top of the source tree (that is, WRKSRC) because it contains some extra pathnames, set PATCH_DIST_STRIP accordingly. For instance, if all the pathnames in the patch have an extra foozolix-1.0/ in front of the filenames, then set PATCH_DIST_STRIP=-p1.

Do not worry if the patches are compressed; they will be decompressed automatically if the filenames end with .Z, .gz, .bz2 or .xz.

If the patch is distributed with some other files, such as documentation, in a gzipped tarball, using PATCHFILES is not possible. If that is the case, add the name and the location of the patch tarball to DISTFILES and MASTER_SITES. Then, use EXTRA_PATCHES to point to those files and bsd.port.mk will automatically apply them. In particular, do not copy patch files into ${PATCHDIR}. That directory may not be writable.

Tip:

If there are multiple patches and they need mixed values for the strip parameter, it can be added alongside the patch name in PATCHFILES, e.g:

PATCHFILES=	patch1 patch2:-p1

This does not conflict with the master site grouping feature, adding a group also works:

PATCHFILES=	patch2:-p1:source2

Note:

The tarball will have been extracted alongside the regular source by then, so there is no need to explicitly extract it if it is a regular gzipped or compressed tarball. Take extra care not to overwrite something that already exists in that directory if extracting it manually. Also, do not forget to add a command to remove the copied patch in the pre-clean target.

5.4.7. Multiple Distribution Files or Patches from Different Sites and Subdirectories (MASTER_SITES:n)

(Consider this to be a somewhat advanced topic; those new to this document may wish to skip this section at first).

This section has information on the fetching mechanism known as both MASTER_SITES:n and MASTER_SITES_NN. We will refer to this mechanism as MASTER_SITES:n.

A little background first. OpenBSD has a neat feature inside DISTFILES and PATCHFILES which allows files and patches to be postfixed with :n identifiers. Here, n can be both [0-9] and denote a group designation. For example:

DISTFILES=	alpha:0 beta:1

In OpenBSD, distribution file alpha will be associated with variable MASTER_SITES0 instead of our common MASTER_SITES and beta with MASTER_SITES1.

This is a very interesting feature which can decrease that endless search for the correct download site.

Just picture 2 files in DISTFILES and 20 sites in MASTER_SITES, the sites slow as hell where beta is carried by all sites in MASTER_SITES, and alpha can only be found in the 20th site. It would be such a waste to check all of them if the maintainer knew this beforehand, would it not? Not a good start for that lovely weekend!

Now that you have the idea, just imagine more DISTFILES and more MASTER_SITES. Surely our distfiles survey meister would appreciate the relief to network strain that this would bring.

In the next sections, information will follow on the FreeBSD implementation of this idea. We improved a bit on OpenBSD's concept.

5.4.7.1. Simplified Information

This section explains how to quickly prepare fine grained fetching of multiple distribution files and patches from different sites and subdirectories. We describe here a case of simplified MASTER_SITES:n usage. This will be sufficient for most scenarios. More detailed information are available in Section 5.4.7.2, “Detailed Information”.

Some applications consist of multiple distribution files that must be downloaded from a number of different sites. For example, Ghostscript consists of the core of the program, and then a large number of driver files that are used depending on the user's printer. Some of these driver files are supplied with the core, but many others must be downloaded from a variety of different sites.

To support this, each entry in DISTFILES may be followed by a colon and a tag name. Each site listed in MASTER_SITES is then followed by a colon, and the tag that indicates which distribution files are downloaded from this site.

For example, consider an application with the source split in two parts, source1.tar.gz and source2.tar.gz, which must be downloaded from two different sites. The port's Makefile would include lines like Example 5.3, “Simplified Use of MASTER_SITES:n with One File Per Site”.

Example 5.3. Simplified Use of MASTER_SITES:n with One File Per Site
MASTER_SITES=	ftp://ftp.example1.com/:source1 \
	    ftp://ftp.example2.com/:source2
	    DISTFILES=	source1.tar.gz:source1 \
	    source2.tar.gz:source2

Multiple distribution files can have the same tag. Continuing the previous example, suppose that there was a third distfile, source3.tar.gz, that is downloaded from ftp.example2.com. The Makefile would then be written like Example 5.4, “Simplified Use of MASTER_SITES:n with More Than One File Per Site”.

Example 5.4. Simplified Use of MASTER_SITES:n with More Than One File Per Site
MASTER_SITES=	ftp://ftp.example1.com/:source1 \
	    ftp://ftp.example2.com/:source2
	    DISTFILES=	source1.tar.gz:source1 \
	    source2.tar.gz:source2 \
	    source3.tar.gz:source2

5.4.7.2. Detailed Information

Okay, so the previous example did not reflect the new port's needs? In this section we will explain in detail how the fine grained fetching mechanism MASTER_SITES:n works and how it can be used.

  1. Elements can be postfixed with :n where n is [^:,]+, that is, n could conceptually be any alphanumeric string but we will limit it to [a-zA-Z_][0-9a-zA-Z_]+ for now.

    Moreover, string matching is case sensitive; that is, n is different from N.

    However, these words cannot be used for postfixing purposes since they yield special meaning: default, all and ALL (they are used internally in item ii). Furthermore, DEFAULT is a special purpose word (check item 3).

  2. Elements postfixed with :n belong to the group n, :m belong to group m and so forth.

  3. Elements without a postfix are groupless, they all belong to the special group DEFAULT. Any elements postfixed with DEFAULT, is just being redundant unless an element belongs to both DEFAULT and other groups at the same time (check item 5).

    These examples are equivalent but the first one is preferred:

    MASTER_SITES=	alpha
    MASTER_SITES=	alpha:DEFAULT
  4. Groups are not exclusive, an element may belong to several different groups at the same time and a group can either have either several different elements or none at all.

  5. When an element belongs to several groups at the same time, use the comma operator (,).

    Instead of repeating it several times, each time with a different postfix, we can list several groups at once in a single postfix. For instance, :m,n,o marks an element that belongs to group m, n and o.

    All these examples are equivalent but the last one is preferred:

    MASTER_SITES=	alpha alpha:SOME_SITE
    MASTER_SITES=	alpha:DEFAULT alpha:SOME_SITE
    MASTER_SITES=	alpha:SOME_SITE,DEFAULT
    MASTER_SITES=	alpha:DEFAULT,SOME_SITE
  6. All sites within a given group are sorted according to MASTER_SORT_AWK. All groups within MASTER_SITES and PATCH_SITES are sorted as well.

  7. Group semantics can be used in any of the variables MASTER_SITES, PATCH_SITES, MASTER_SITE_SUBDIR, PATCH_SITE_SUBDIR, DISTFILES, and PATCHFILES according to this syntax:

    1. All MASTER_SITES, PATCH_SITES, MASTER_SITE_SUBDIR and PATCH_SITE_SUBDIR elements must be terminated with the forward slash / character. If any elements belong to any groups, the group postfix :n must come right after the terminator /. The MASTER_SITES:n mechanism relies on the existence of the terminator / to avoid confusing elements where a :n is a valid part of the element with occurrences where :n denotes group n. For compatibility purposes, since the / terminator was not required before in both MASTER_SITE_SUBDIR and PATCH_SITE_SUBDIR elements, if the postfix immediate preceding character is not a / then :n will be considered a valid part of the element instead of a group postfix even if an element is postfixed with :n. See both Example 5.5, “Detailed Use of MASTER_SITES:n in MASTER_SITE_SUBDIR and Example 5.6, “Detailed Use of MASTER_SITES:n with Comma Operator, Multiple Files, Multiple Sites and Multiple Subdirectories”.

      Example 5.5. Detailed Use of MASTER_SITES:n in MASTER_SITE_SUBDIR
      MASTER_SITE_SUBDIR=	old:n new/:NEW
      • Directories within group DEFAULT -> old:n

      • Directories within group NEW -> new


      Example 5.6. Detailed Use of MASTER_SITES:n with Comma Operator, Multiple Files, Multiple Sites and Multiple Subdirectories
      MASTER_SITES=	http://site1/%SUBDIR%/ http://site2/:DEFAULT \
      		    http://site3/:group3 http://site4/:group4 \
      		    http://site5/:group5 http://site6/:group6 \
      		    http://site7/:DEFAULT,group6 \
      		    http://site8/%SUBDIR%/:group6,group7 \
      		    http://site9/:group8
      		    DISTFILES=	file1 file2:DEFAULT file3:group3 \
      		    file4:group4,group5,group6 file5:grouping \
      		    file6:group7
      		    MASTER_SITE_SUBDIR=	directory-trial:1 directory-n/:groupn \
      		    directory-one/:group6,DEFAULT \
      		    directory

      The previous example results in this fine grained fetching. Sites are listed in the exact order they will be used.

      • file1 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site1/directory-trial:1/

        • http://site1/directory-one/

        • http://site1/directory/

        • http://site2/

        • http://site7/

        • MASTER_SITE_BACKUP

      • file2 will be fetched exactly as file1 since they both belong to the same group

        • MASTER_SITE_OVERRIDE

        • http://site1/directory-trial:1/

        • http://site1/directory-one/

        • http://site1/directory/

        • http://site2/

        • http://site7/

        • MASTER_SITE_BACKUP

      • file3 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site3/

        • MASTER_SITE_BACKUP

      • file4 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site4/

        • http://site5/

        • http://site6/

        • http://site7/

        • http://site8/directory-one/

        • MASTER_SITE_BACKUP

      • file5 will be fetched from

        • MASTER_SITE_OVERRIDE

        • MASTER_SITE_BACKUP

      • file6 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site8/

        • MASTER_SITE_BACKUP


  8. How do I group one of the special variables from bsd.sites.mk, for example, MASTER_SITE_SOURCEFORGE?

    See Example 5.7, “Detailed Use of MASTER_SITES:n with MASTER_SITE_SOURCEFORGE.

    Example 5.7. Detailed Use of MASTER_SITES:n with MASTER_SITE_SOURCEFORGE
    MASTER_SITES=	http://site1/ ${MASTER_SITE_SOURCEFORGE:S/$/:sourceforge,TEST/}
    		DISTFILES=	something.tar.gz:sourceforge

    something.tar.gz will be fetched from all sites within MASTER_SITE_SOURCEFORGE.

  9. How do I use this with PATCH*?

    All examples were done with MASTER* but they work exactly the same for PATCH* ones as can be seen in Example 5.8, “Simplified Use of MASTER_SITES:n with PATCH_SITES.

    Example 5.8. Simplified Use of MASTER_SITES:n with PATCH_SITES
    PATCH_SITES=	http://site1/ http://site2/:test
    		PATCHFILES=	patch1:test

5.4.7.3. What Does Change for Ports? What Does Not?

  1. All current ports remain the same. The MASTER_SITES:n feature code is only activated if there are elements postfixed with :n like elements according to the aforementioned syntax rules, especially as shown in item 7.

  2. The port targets remain the same: checksum, makesum, patch, configure, build, etc. With the obvious exceptions of do-fetch, fetch-list, master-sites and patch-sites.

    • do-fetch: deploys the new grouping postfixed DISTFILES and PATCHFILES with their matching group elements within both MASTER_SITES and PATCH_SITES which use matching group elements within both MASTER_SITE_SUBDIR and PATCH_SITE_SUBDIR. Check Example 5.6, “Detailed Use of MASTER_SITES:n with Comma Operator, Multiple Files, Multiple Sites and Multiple Subdirectories”.

    • fetch-list: works like old fetch-list with the exception that it groups just like do-fetch.

    • master-sites and patch-sites: (incompatible with older versions) only return the elements of group DEFAULT; in fact, they execute targets master-sites-default and patch-sites-default respectively.

      Furthermore, using target either master-sites-all or patch-sites-all is preferred to directly checking either MASTER_SITES or PATCH_SITES. Also, directly checking is not guaranteed to work in any future versions. Check item B for more information on these new port targets.

  3. New port targets

    1. There are master-sites-n and patch-sites-n targets which will list the elements of the respective group n within MASTER_SITES and PATCH_SITES respectively. For instance, both master-sites-DEFAULT and patch-sites-DEFAULT will return the elements of group DEFAULT, master-sites-test and patch-sites-test of group test, and thereon.

    2. There are new targets master-sites-all and patch-sites-all which do the work of the old master-sites and patch-sites ones. They return the elements of all groups as if they all belonged to the same group with the caveat that it lists as many MASTER_SITE_BACKUP and MASTER_SITE_OVERRIDE as there are groups defined within either DISTFILES or PATCHFILES; respectively for master-sites-all and patch-sites-all.

5.4.8. DIST_SUBDIR

Do not let the port clutter /usr/ports/distfiles. If the port requires a lot of files to be fetched, or contains a file that has a name that might conflict with other ports (for example, Makefile), set DIST_SUBDIR to the name of the port (${PORTNAME} or ${PKGNAMEPREFIX}${PORTNAME} are fine). This will change DISTDIR from the default /usr/ports/distfiles to /usr/ports/distfiles/DIST_SUBDIR, and in effect puts everything that is required for the port into that subdirectory.

It will also look at the subdirectory with the same name on the backup master site at ftp.FreeBSD.org. (Setting DISTDIR explicitly in Makefile will not accomplish this, so please use DIST_SUBDIR.)

Note:

This does not affect MASTER_SITES defined in the Makefile.

5.4.9. ALWAYS_KEEP_DISTFILES

If the port uses binary distfiles and has a license that requires that the source code is provided with packages distributed in binary form, like GPL, ALWAYS_KEEP_DISTFILES will instruct the FreeBSD build cluster to keep a copy of the files specified in DISTFILES. Users of these ports will generally not need these files, so it is a good idea to only add the source distfiles to DISTFILES when PACKAGE_BUILDING is defined.

Example 5.9. Use of ALWAYS_KEEP_DISTFILES
.if defined(PACKAGE_BUILDING)
	  DISTFILES+=		foo.tar.gz
	  ALWAYS_KEEP_DISTFILES=	yes
	  .endif

When adding extra files to DISTFILES, make sure to also add them to distinfo. Also, the additional files will normally be extracted into WRKDIR as well, which for some ports may lead to undesirable side effects and require special handling.

All FreeBSD documents are available for download at http://ftp.FreeBSD.org/pub/FreeBSD/doc/

Questions that are not answered by the documentation may be sent to <freebsd-questions@FreeBSD.org>.
Send questions about this document to <freebsd-doc@FreeBSD.org>.