Chapter 4. Slow Porting

Okay, so it was not that simple, and the port required some modifications to get it to work. In this section, we will explain, step by step, how to modify it to get it to work with the ports paradigm.

4.1. How Things Work

First, this is the sequence of events which occurs when the user first types make in the port’s directory. Having bsd.port.mk in another window while reading this really helps to understand it.

But do not worry, not many people understand exactly how bsd.port.mk is working…​ :-)

  1. The fetch target is run. The fetch target is responsible for making sure that the tarball exists locally in DISTDIR. If fetch cannot find the required files in DISTDIR it will look up the URL MASTER_SITES, which is set in the Makefile, as well as our FTP mirrors where we put distfiles as backup. It will then attempt to fetch the named distribution file with FETCH, assuming that the requesting site has direct access to the Internet. If that succeeds, it will save the file in DISTDIR for future use and proceed.

  2. The extract target is run. It looks for the port’s distribution file (typically a compressed tarball) in DISTDIR and unpacks it into a temporary subdirectory specified by WRKDIR (defaults to work).

  3. The patch target is run. First, any patches defined in PATCHFILES are applied. Second, if any patch files named patch-* are found in PATCHDIR (defaults to the files subdirectory), they are applied at this time in alphabetical order.

  4. The configure target is run. This can do any one of many different things.

    1. If it exists, scripts/configure is run.

    2. If HAS_CONFIGURE or GNU_CONFIGURE is set, WRKSRC/configure is run.

  5. The build target is run. This is responsible for descending into the port’s private working directory (WRKSRC) and building it.

  6. The stage target is run. This puts the final set of built files into a temporary directory (STAGEDIR, see Staging). The hierarchy of this directory mirrors that of the system on which the package will be installed.

  7. The package target is run. This creates a package using the files from the temporary directory created during the stage target and the port’s pkg-plist.

  8. The install target is run. This installs the package created during the package target into the host system.

The above are the default actions. In addition, define targets pre-something or post-something, or put scripts with those names, in the scripts subdirectory, and they will be run before or after the default actions are done.

For example, if there is a post-extract target defined in the Makefile, and a file pre-build in the scripts subdirectory, the post-extract target will be called after the regular extraction actions, and pre-build will be executed before the default build rules are done. It is recommended to use Makefile targets if the actions are simple enough, because it will be easier for someone to figure out what kind of non-default action the port requires.

The default actions are done by the do-something targets from bsd.port.mk. For example, the commands to extract a port are in the target do-extract. If the default target does not do the job right, redefine the do-something target in the Makefile.

The "main" targets (for example, extract, configure, etc.) do nothing more than make sure all the stages up to that one are completed and call the real targets or scripts, and they are not intended to be changed. To fix the extraction, fix do-extract, but never ever change the way extract operates! Additionally, the target post-deinstall is invalid and is not run by the ports infrastructure.

Now that what goes on when the user types make install is better understood, let us go through the recommended steps to create the perfect port.

4.2. Getting the Original Sources

Get the original sources (normally) as a compressed tarball (foo.tar.gz or foo.tar.bz2) and copy it into DISTDIR. Always use mainstream sources when and where possible.

Set the variable MASTER_SITES to reflect where the original tarball resides. Shorthand definitions exist for most mainstream sites in bsd.sites.mk. Please use these sites-and the associated definitions-if at all possible, to help avoid the problem of having the same information repeated over again many times in the source base. As these sites tend to change over time, this becomes a maintenance nightmare for everyone involved. See MASTER_SITES for details.

If there is no FTP/HTTP site that is well-connected to the net, or can only find sites that have irritatingly non-standard formats, put a copy on a reliable FTP or HTTP server (for example, a home page).

If a convenient and reliable place to put the distfile cannot be found, we can "house" it ourselves on ftp.FreeBSD.org; however, this is the least-preferred solution. The distfile must be placed into ~/public_distfiles/ of someone’s freefall account. Ask the person who commits the port to do this. This person will also set MASTER_SITES to LOCAL/username where username is their FreeBSD cluster login.

If the port’s distfile changes all the time without any kind of version update by the author, consider putting the distfile on a home page and listing it as the first MASTER_SITES. Try to talk the port author out of doing this; it really does help to establish some kind of source code control. Hosting a specific version will prevent users from getting checksum mismatch errors, and also reduce the workload of maintainers of our FTP site. Also, if there is only one master site for the port, it is recommended to house a backup on a home page and list it as the second MASTER_SITES.

If the port requires additional patches that are available on the Internet, fetch them too and put them in DISTDIR. Do not worry if they come from a site other than where the main source tarball comes, we have a way to handle these situations (see the description of PATCHFILES below).

4.3. Modifying the Port

Unpack a copy of the tarball in a private directory and make whatever changes are necessary to get the port to compile properly under the current version of FreeBSD. Keep careful track of steps, as they will be needed to automate the process shortly. Everything, including the deletion, addition, or modification of files has to be doable using an automated script or patch file when the port is finished.

If the port requires significant user interaction/customization to compile or install, take a look at one of Larry Wall’s classic Configure scripts and perhaps do something similar. The goal of the new ports collection is to make each port as "plug-and-play" as possible for the end-user while using a minimum of disk space.

Unless explicitly stated, patch files, scripts, and other files created and contributed to the FreeBSD ports collection are assumed to be covered by the standard BSD copyright conditions.

4.4. Patching

In the preparation of the port, files that have been added or changed can be recorded with diff(1) for later feeding to patch(1). Doing this with a typical file involves saving a copy of the original file before making any changes using a .orig suffix.

% cp file file.orig

After all changes have been made, cd back to the port directory. Use make makepatch to generate updated patch files in the files directory.

Use BINARY_ALIAS to substitute hardcoded commands during the build and avoid patching build files. See Use BINARY_ALIAS to Rename Commands Instead of Patching the Build for more information.

4.4.1. General Rules for Patching

Patch files are stored in PATCHDIR, usually files/, from where they will be automatically applied. All patches must be relative to WRKSRC. Typically WRKSRC is a subdirectory of WRKDIR, the directory where the distfile is extracted. Use make -V WRKSRC to see the actual path. The patch names are to follow these rules:

  • Avoid having more than one patch modify the same file. For example, having both patch-foobar.c and patch-foobar.c2 making changes to ${WRKSRC}/foobar.c makes them fragile and difficult to debug.

  • When creating names for patch files, replace each underscore (_) with two underscores (__) and each slash (/) with one underscore (_). For example, to patch a file named src/freeglut_joystick.c, name the corresponding patch patch-src_freeglut__joystick.c. Do not name patches like patch-aa or patch-ab. Always use the path and file name in patch names. Using make makepatch automatically generates the correct names.

  • A patch may modify multiple files if the changes are related and the patch is named appropriately. For example, patch-add-missing-stdlib.h.

  • Only use characters [-+._a-zA-Z0-9] for naming patches. In particular, do not use :: as a path separator, use _ instead.

Minimize the amount of non-functional whitespace changes in patches. It is common in the Open Source world for projects to share large amounts of a code base, but obey different style and indenting rules. When taking a working piece of functionality from one project to fix similar areas in another, please be careful: the resulting patch may be full of non-functional changes. It not only increases the size of the ports repository but makes it hard to find out what exactly caused the problem and what was changed at all.

If a file must be deleted, do it in the post-extract target rather than as part of the patch.

4.4.2. Manual Patch Generation

Manual patch creation is usually not necessary. Automatic patch generation as described earlier in this section is the preferred method. However, manual patching may be required occasionally.

Patches are saved into files named patch-* where * indicates the pathname of the file that is patched, such as patch-Imakefile or patch-src-config.h. Patches with file names which do not start with patch- will not be applied automatically.

After the file has been modified, diff(1) is used to record the differences between the original and the modified version. -u causes diff(1) to produce "unified" diffs, the preferred form.

% diff -u file.orig file > patch-pathname-file

When generating patches for new, added files, -N is used to tell diff(1) to treat the non-existent original file as if it existed but was empty:

% diff -u -N newfile.orig newfile > patch-pathname-newfile

Using the recurse (-r) option to diff(1) to generate patches is fine, but please look at the resulting patches to make sure there is no unnecessary junk in there. In particular, diffs between two backup files, Makefiles when the port uses Imake or GNU configure, etc., are unnecessary and have to be deleted. If it was necessary to edit configure.in and run autoconf to regenerate configure, do not take the diffs of configure (it often grows to a few thousand lines!). Instead, define USES=autoreconf and take the diffs of configure.in.

4.4.3. Simple Automatic Replacements

Simple replacements can be performed directly from the port Makefile using the in-place mode of sed(1). This is useful when changes use the value of a variable:

post-patch:
	@${REINPLACE_CMD} -e 's|/usr/local|${PREFIX}|g' ${WRKSRC}/Makefile

Only use sed(1) to replace variable content. You must use patch files instead of sed(1) to replace static content.

Quite often, software being ported uses the CR/LF convention in source files. This may cause problems with further patching, compiler warnings, or script execution (like /bin/sh^M not found.) To quickly convert all files from CR/LF to just LF, add this entry to the port Makefile:

USES=	dos2unix

A list of specific files to convert can be given:

USES=	dos2unix
DOS2UNIX_FILES=	util.c util.h

Use DOS2UNIX_REGEX to convert a group of files across subdirectories. Its argument is a find(1)-compatible regular expression. More on the format is in re_format(7). This option is useful for converting all files of a given extension. For example, convert all source code files, leaving binary files intact:

USES=	dos2unix
DOS2UNIX_REGEX=	.*\.([ch]|cpp)

A similar option is DOS2UNIX_GLOB, which runs find for each element listed in it.

USES=	dos2unix
DOS2UNIX_GLOB=	*.c *.cpp *.h

The base directory for the conversion can be set. This is useful when there are multiple distfiles and several contain files which require line-ending conversion.

USES=	dos2unix
DOS2UNIX_WRKSRC=	${WRKDIR}

4.4.4. Patching Conditionally

Some ports need patches that are only applied for specific FreeBSD versions or when a particular option is enabled or disabled. Conditional patches are specified by placing the full paths to the patch files in EXTRA_PATCHES. Conditional patch file names usually start with extra- although this is not necessary. However, their file names must not start with patch-. If they do, they are applied unconditionally by the framework which is undesired for conditional patches.

Example 1. Applying a Patch for a Specific FreeBSD Version
.include <bsd.port.options.mk>

# Patch in the iconv const qualifier before this
.if ${OPSYS} == FreeBSD && ${OSVERSION} < 1100069
EXTRA_PATCHES=	${PATCHDIR}/extra-patch-fbsd10
.endif

.include <bsd.port.mk>
Example 2. Optionally Applying a Patch

When an option requires a patch, use opt_EXTRA_PATCHES and opt_EXTRA_PATCHES_OFF to make the patch conditional on the opt option. See Generic Variables Replacement for more information.

OPTIONS_DEFINE=	  FOO BAR
FOO_EXTRA_PATCHES=  ${PATCHDIR}/extra-patch-foo
BAR_EXTRA_PATCHES_OFF=	${PATCHDIR}/extra-patch-bar.c \
		${PATCHDIR}/extra-patch-bar.h
Example 3. Using EXTRA_PATCHES With a Directory

Sometimes, there are many patches that are needed for a feature, in this case, it is possible to point EXTRA_PATCHES to a directory, and it will automatically apply all files named patch-* in it.

Create a subdirectory in ${PATCHDIR}, and move the patches in it. For example:

% ls -l files/foo-patches
-rw-r--r--  1 root  wheel    350 Jan 16 01:27 patch-Makefile.in
-rw-r--r--  1 root  wheel   3084 Jan 18 15:37 patch-configure.ac

Then add this to the Makefile:

OPTIONS_DEFINE=	FOO
FOO_EXTRA_PATCHES=	${PATCHDIR}/foo-patches

The framework will then use all the files named patch-* in that directory.

4.5. Configuring

Include any additional customization commands in the configure script and save it in the scripts subdirectory. As mentioned above, it is also possible do this with Makefile targets and/or scripts with the name pre-configure or post-configure.

4.6. Handling User Input

If the port requires user input to build, configure, or install, set IS_INTERACTIVE in the Makefile. This will allow "overnight builds" to skip it. If the user sets the variable BATCH in their environment (and if the user sets the variable INTERACTIVE, then only those ports requiring interaction are built). This will save a lot of wasted time on the set of machines that continually build ports (see below).

It is also recommended that if there are reasonable default answers to the questions, PACKAGE_BUILDING be used to turn off the interactive script when it is set. This will allow us to build the packages for CDROMs and FTP.


Last modified on: March 9, 2024 by Danilo G. Baio