3. How to Mirror FreeBSD

Ok, now you know the requirements and how to offer the services, but not how to get it. :-) This section explains how to actually mirror the various parts of FreeBSD, what tools to use, and where to mirror from.

3.1. Mirroring the FTP site

The FTP area is the largest amount of data that needs to be mirrored. It includes the distribution sets required for network installation, the branches which are actually snapshots of checked-out source trees, the ISO Images to write CD-ROMs with the installation distribution, a live file system, lots of packages, the ports tree, distfiles, and a huge amount of packages. All of course for various FreeBSD versions, and various architectures.

The best way to mirror the FTP area is rsync. You can install the port net/rsync and then use rsync to sync with your upstream host. rsync is already mentioned in Section 2.4.2, “Rsync (optional for FTP fileset)”. Since rsync access is not required, your preferred upstream site may not allow it. You may need to hunt around a little bit to find a site that allows rsync access.

Note:

Since the number of rsync clients will have a significant impact on the server machine, most admins impose limitations on their server. For a mirror, you should ask the site maintainer you are syncing from about their policy, and maybe an exception for your host (since you are a mirror).

A command line to mirror FreeBSD might look like:

% rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/
          

Consult the documentation for rsync, which is also available at http://rsync.samba.org/, about the various options to be used with rsync. If you sync the whole module (unlike subdirectories), be aware that the module-directory (here "FreeBSD") will not be created, so you cannot omit the target directory. Also you might want to set up a script framework that calls such a command via cron(8).

3.2. Mirroring the WWW pages

The FreeBSD website should only be mirrored via rsync.

A command line to mirror the FreeBSD web site might look like:

% rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/
      

3.3. How often should I mirror?

Every mirror should be updated on a regular basis. You will certainly need some script framework for it that will be called by cron(8). Since nearly every admin does this his own way, we cannot give specific instructions. It could work like this:

  1. Put the command to run your mirroring application in a script. Use of a plain /bin/sh script is recommended.

  2. Add some output redirections so diagnostic messages are logged to a file.

  3. Test if your script works. Check the logs.

  4. Use crontab(1) to add the script to the appropriate user's crontab(5). This should be a different user than what your FTP daemon runs as so that if file permissions inside your FTP area are not world-readable those files can not be accessed by anonymous FTP. This is used to stage releases — making sure all of the official mirror sites have all of the necessary release files on release day.

Here are some recommended schedules:

  • FTP fileset: daily

  • WWW pages: daily

All FreeBSD documents are available for download at http://ftp.FreeBSD.org/pub/FreeBSD/doc/

Questions that are not answered by the documentation may be sent to <freebsd-questions@FreeBSD.org>.
Send questions about this document to <freebsd-doc@FreeBSD.org>.