13. How to configure a new package building node

Before following these steps, please coordinate with portmgr.

13.1. Node requirements

Note:

This section is only of interest when considering tier-2 architectures.

Here are the requirement for what a node needs to be generally useful.

  • CPU capacity: anything less than 500MHz is generally not useful for package building.

    Note:

    We are able to adjust the number of jobs dispatched to each machine, and we generally tune the number to use 100% of CPU.

  • RAM: Less than 2G is not very useful; 8G or more is preferred. We have been tuning to one job per 512M of RAM.

  • disk: at least 20G is needed for filesystem; 32G is needed for swap. Best performance will be if multiple disks are used, and configured as geom stripes. Performance numbers are also TBA.

    Note:

    Package building will test disk drives to destruction. Be aware of what you are signing up for!

  • network bandwidth: TBA. However, an 8-job machine has been shown to saturate a cable modem line.

13.2. Preparation

  1. Pick a unique hostname. It does not have to be a publicly resolvable hostname (it can be a name on your internal network).

  2. By default, package building requires the following TCP ports to be accessible: 22 (ssh), 414 (infoseek), and 8649 (ganglia). If these are not accessible, pick others and ensure that an ssh tunnel is set up (see below).

    (Note: if you have more than one machine at your site, you will need an individual TCP port for each service on each machine, and thus ssh tunnels will be necessary. As such, you will probably need to configure port forwarding on your firewall.)

  3. Decide if you will be booting natively or via pxeboot. You will find that it is easier to keep up with changes to -current with the latter, especially if you have multiple machines at your site.

  4. Pick a directory to hold ports configuration and chroot subdirectories. It may be best to put it this on its own partition. (Example: /usr2/.)

    Note:

    The filename chroot is a historical remnant. The chroot command is no longer used.

  5. Decide if you will be using a local squid cache on the client, instead of the server. It is more efficient to run it on the server. If you are doing that, skip the "squid" steps below.)

13.3. Configuring src

  1. Create a directory to contain the latest -current source tree and check it out. (Since your machine will likely be asked to build packages for -current, the kernel it runs should be reasonably up-to-date with the bindist that will be exported by our scripts.)

  2. If you are using pxeboot: create a directory to contain the install bits. You will probably want to use a subdirectory of /pxeroot, e.g., /pxeroot/${arch}-${branch}. Export that as DESTDIR.

  3. If you are cross-building, export TARGET_ARCH=${arch}.

    Note:

    The procedure for cross-building ports is not yet defined.

  4. Generate a kernel config file. Include GENERIC (or, if on i386™, and you are using more than 3.5G, PAE).

    Required options:

    options         NULLFS
    options         TMPFS

    Suggested options:

    options         GEOM_CONCAT
    options         GEOM_STRIPE
    options         SHMMAXPGS=65536
    options         SEMMNI=40
    options         SEMMNS=240
    options         SEMUME=40
    options         SEMMNU=120

    If you are interested in debugging general problems, you may wish to use the following. However, for unattended operations, it is best to leave it out:

    options         ALT_BREAK_TO_DEBUGGER

    For PAE, it is not currently possible to load modules. Therefore, if you are running an architecture that supports Linux emulation, you will need to add:

    options         COMPAT_LINUX
    options         LINPROCFS

    Also for PAE, as of 20110912 you need the following. This needs to be investigated:

    nooption        NFSD                    # New Network Filesystem Server
    options         NFSCLIENT               # Network Filesystem Client
    options         NFSSERVER               # Network Filesystem Server
  5. As root, do the usual build steps, e.g.:

    # make -j4 buildworld
    # make buildkernel KERNCONF=${kernconf}
    # make installkernel KERNCONF=${kernconf}
    # make installworld

    The install steps use DESTDIR.

  6. Customize files in etc/. Whether you do this on the client itself, or another machine, will depend on whether you are using pxeboot.

    If you are using pxeboot: create a subdirectory of ${DESTDIR} called conf/. Create one subdirectory default/etc/, and (if your site will host multiple nodes), subdirectories ${ip-address}/etc/ to contain override files for individual hosts. (You may find it handy to symlink each of those directories to a hostname.) Copy the entire contents of ${DESTDIR}/etc/ to default/etc/; that is where you will edit your files. The by-ip-address etc/ directories will probably only need customized rc.conf files.

    In either case, apply the following steps:

    • Create a portbuild user and group. It can have the '*' password.

      Create /home/portbuild/.ssh/ and populate authorized_keys.

    • If you are using ganglia for monitoring, add the following user:

      ganglia:*:102:102::0:0:User &:/usr/local/ganglia:/bin/sh

      Add it to etc/group as well.

    • If you are using a local squid cache on the client, add the following user:

      squid:*:100:100::0:0:User &:/usr/local/squid:/bin/sh

      Add it to etc/group as well.

    • Create the appropriate files in etc/.ssh/.

    • In etc/crontab: add

      *       *       *       *       *       root    /var/portbuild/scripts/client-metrics
    • Create the appropriate etc/fstab. (If you have multiple, different, machines, you will need to put those in the override directories.)

    • In etc/inetd.conf: add

      infoseek        stream  tcp     nowait  nobody  /var/portbuild/scripts/reportload
    • You should run the cluster on UTC. If you have not set the clock to UTC:

      # cp -p /usr/share/zoneinfo/Etc/UTC etc/localtime
    • Create the appropriate etc/rc.conf. (If you are using pxeboot, and have multiple, different, machines, you will need to put those in the override directories.)

      Recommended entries for physical nodes:

      hostname="${hostname}"
      inetd_enable="YES"
      linux_enable="YES"
      nfs_client_enable="YES"
      ntpd_enable="YES"
      sendmail_enable="NONE"
      sshd_enable="YES"
      sshd_program="/usr/local/sbin/sshd"

      If you are using ganglia for monitoring, add the following

      gmond_enable="YES"

      If you are using a local squid cache on the client, add the following

      squid_enable="YES"
      squid_chdir="/a/squid/logs"
      squid_pidfile="/a/squid/logs/squid.pid"

      Required entries for VMWare-based nodes:

      vmware_guest_vmmemctl_enable="YES"
      vmware_guest_guestd_enable="YES"

      Recommended entries for VMWare-based nodes:

      hostname=""
      ifconfig_em0="DHCP"
      fsck_y_enable="YES"
      
      inetd_enable="YES"
      linux_enable="YES"
      nfs_client_enable="YES"
      sendmail_enable="NONE"
      sshd_enable="YES"
      sshd_program="/usr/local/sbin/sshd"
      
      gmond_enable="YES"
      squid_enable="YES"
      squid_chdir="/a/squid/logs"
      squid_pidfile="/a/squid/logs/squid.pid"

      ntpd(8) should not be enabled for VMWare instances.

      Also, it may be possible to leave squid disabled by default so as to not have /a persistent (which should save instantiation time.) Work is still ongoing.

    • Create etc/resolv.conf, if necessary.

    • Modify etc/sysctl.conf:

      9a10,30
      > kern.corefile=/a/%N.core
      > kern.sugid_coredump=1
      > #debug.witness_ddb=0
      > #debug.witness_watch=0
      >
      > # squid needs a lot of fds (leak?)
      > kern.maxfiles=40000
      > kern.maxfilesperproc=30000
      >
      > # Since the NFS root is static we do not need to check frequently for file changes
      > # This saves >75% of NFS traffic
      > vfs.nfs.access_cache_timeout=300
      > debug.debugger_on_panic=1
      >
      > # For jailing
      > security.jail.sysvipc_allowed=1
      > security.jail.allow_raw_sockets=1
      > security.jail.chflags_allowed=1
      > security.jail.enforce_statfs=1
      >
      > vfs.lookup_shared=1
    • If desired, modify etc/syslog.conf to change the logging destinations to @pointyhat.freebsd.org.

13.4. Configuring ports

  1. Install the following ports:

    net/rsync
    security/sudo

    You may also wish to install:

    security/openssh-portable (with HPN on)

    If you are using ganglia for monitoring, install the following:

    sysutils/ganglia-monitor-core (with GMETAD off)

    If you are using a local squid cache on the client, install the following

    www/squid31 (with SQUID_AUFS on)
  2. Customize files in usr/local/etc/. Whether you do this on the client itself, or another machine, will depend on whether you are using pxeboot.

    Note:

    The trick of using conf override subdirectories is less effective here, because you would need to copy over all subdirectories of usr/. This is an implementation detail of how the pxeboot works.

    Apply the following steps:

    • If you are using ganglia, modify usr/local/etc/gmond.conf:

      21,22c21,22
      <   name = "unspecified"
      <   owner = "unspecified"
      ---
      >   name = "${arch} package build cluster"
      >   owner = "portmgr@FreeBSD.org"
      24c24
      <   url = "unspecified"
      ---
      >   url = "http://pointyhat.freebsd.org"

      If there are machines from more than one cluster in the same multicast domain (basically = LAN) then change the multicast groups to different values (.71, .72, etc).

    • Create usr/local/etc/rc.d/portbuild.sh, using the appropriate value for scratchdir:

      #!/bin/sh
      #
      # Configure a package build system post-boot
      
      scratchdir=/a
      
      ln -sf ${scratchdir}/portbuild /var/
      
      # Identify builds ready for use
      cd /var/portbuild/arch
      for i in */builds/*; do
          if [ -f ${i}/.ready ]; then
              mkdir /tmp/.setup-${i##*/}
          fi
      done
      
      # Flag that we are ready to accept jobs
      touch /tmp/.boot_finished
    • If you are using a local squid cache, modify, usr/local/etc/squid/squid.conf:

      288,290c288,290
      < #auth_param basic children 5
      < #auth_param basic realm Squid proxy-caching web server
      < #auth_param basic credentialsttl 2 hours
      ---
      > auth_param basic children 5
      > auth_param basic realm Squid proxy-caching web server
      > auth_param basic credentialsttl 2 hours
      611a612
      > acl localnet src 127.0.0.0/255.0.0.0
      655a657
      > http_access allow localnet
      2007a2011
      > maximum_object_size 400 MB
      2828a2838
      > negative_ttl 0 minutes

      Also, change usr/local to usr2 in cache_dir, access_log, cache_log, cache_store_log, pid_filename, netdb_filename, coredump_dir.

      Finally, change the cache_dir storage scheme from ufs to aufs (offers better performance).

    • Configure ssh: copy etc/ssh to usr/local/etc/ssh and add NoneEnabled yes to sshd_config.

    • Note:

      This step is under review.

      Create usr/local/etc/sudoers/sudoers.d/portbuild:

      # local changes for package building
      portbuild     ALL=(ALL) NOPASSWD: ALL

13.5. Configuration on the client itself

  1. Change into the port/package directory you picked above, e.g., cd /usr2.

  2. As root:

    # mkdir portbuild
    # chown portbuild:portbuild portbuild
    # mkdir pkgbuild
    # chown portbuild:portbuild pkgbuild

    If you are using a local squid cache:

    # mkdir squid
    # mkdir squid/cache
    # mkdir squid/logs
    # chown -R squid:squid squid
  3. If clients preserve /var/portbuild between boots then they must either preserve their /tmp, or revalidate their available builds at boot time (see the script on the amd64 machines). They must also clean up stale jails from previous builds before creating /tmp/.boot_finished.

  4. Boot the client.

  5. If you are using a local squid cache, as root, initialize the squid directories:

    squid -z

13.6. Configuration on the server

These steps need to be taken by a portmgr acting as portbuild on the server.

  1. If any of the default TCP ports is not available (see above), you will need to create an ssh tunnel for them and include its invocation command in portbuild's crontab.

  2. Unless you can use the defaults, add an entry to /home/portbuild/.ssh/config to specify the public IP address, TCP port for ssh, username, and any other necessary information.

  3. Create /a/portbuild/${arch}/clients/bindist-${hostname}.tar.

    • Copy one of the existing ones as a template and unpack it in a temporary directory.

    • Customize etc/resolv.conf for the local site.

    • Customize etc/make.conf for FTP fetches for the local site. Note: the nulling-out of MASTER_SITE_BACKUP must be common to all nodes, but the first entry in MASTER_SITE_OVERRIDE should be the nearest local FTP mirror. Example:

      .if defined(FETCH_ORIGINAL)
      MASTER_SITE_BACKUP=
      .else
      MASTER_SITE_OVERRIDE= \
      	ftp://friendly-local-ftp-mirror/pub/FreeBSD/ports/distfiles/${DIST_SUBDIR}/ \
      	ftp://${BACKUP_FTP_SITE}/pub/FreeBSD/distfiles/${DIST_SUBDIR}/
      .endif
    • tar it up and move it to the right location.

    Hint: you will need one of these for each machine; however, if you have multiple machines at one site, you should create a site-specific one (e.g., in /a/portbuild/conf/clients/) and symlink to it.

  4. Create /a/portbuild/${arch}/portbuild-${hostname} using one of the existing ones as a guide. This file contains overrides to /a/portbuild/${arch}/portbuild.conf.

    Suggested values:

    disconnected=1
    scratchdir=/usr2/pkgbuild
    client_user=portbuild
    sudo_cmd="sudo -H"
    rsync_gzip=-z
    
    infoseek_host=localhost
    infoseek_port=${tunelled-tcp-port}

    If you will be using squid on the client:

    http_proxy="http://localhost:3128/"
    squid_dir=/usr2/squid

    If, instead, you will be using squid on the server:

    http_proxy="http://servername:3128/"

    Possible other values:

    use_md_swap=1
    md_size=9g
    use_zfs=1
    scp_cmd="/usr/local/bin/scp"
    ssh_cmd="/usr/local/bin/ssh"

These steps need to be taken by a portmgr acting as root on pointyhat.

  1. Add the public IP address to /etc/hosts.allow. (Remember, multiple machines can be on the same IP address.)

  2. If you are using ganglia, add an appropriate data_source entry to /usr/local/etc/gmetad.conf:

    data_source "arch/location Package Build Cluster" 30 hostname

    You will need to restart gmetad.

13.7. Enabling the node

These steps need to be taken by a portmgr acting as portbuild:

  1. Ensure that ssh to the client is working by executing ssh hostname uname -a. The actual command is not important; what is important is to confirm the setup, and also add an entry into known_hosts, once you have confirmed the node's identity.

  2. Populate the client's copy of /var/portbuild/scripts/ by something like /a/portbuild/scripts/dosetupnode arch major latest hostname. Verify that you now have files in that directory.

  3. Test the other TCP ports by executing telnet hostname portnumber. 414 (or its tunnel) should give you a few lines of status information including arch and osversion; 8649 should give you an XML response from ganglia.

This step needs to be taken by a portmgr acting as portbuild:

  • Tell qmanager about the node. Example:

    python path/qmanager/qclient add name=uniquename arch=arch osversion=osversion numcpus=number haszfs=0 online=1 domain=domain primarypool=package pools="package all" maxjobs=1 acl="ports-arch,deny_all"

Finally, again as portmgr acting as portbuild:

  • Once you are sure that the client is working, tell pollmachine about it by adding it to /a/portbuild/${arch}/mlist.

All FreeBSD documents are available for download at http://ftp.FreeBSD.org/pub/FreeBSD/doc/

Questions that are not answered by the documentation may be sent to <freebsd-questions@FreeBSD.org>.
Send questions about this document to <freebsd-doc@FreeBSD.org>.