Using poudriere to create a custom FreeBSD repository for package installation

I just showed you how I installed ezjail 3.3 on FreeBSD 9.2 and how it uses ZFS for each jail. Today I’m writing about using poudriere to create custom packages for use by pkng.

In this post, I will be creating packages on a FreeBSD 9.2 server. Those packages are for running Bacula on FreeBSD 9.2. To accomplish this, we will need:

1. A list of ports to install for running Bacula. In poudriere terminology, this is known as a set. Read the CUSTOMISATION section of the man page for more information.
2. The options you want to use for those ports.

We will make the customized packages will be available via an http interface (provided by nginx), but we will do that in a future post, not this one.

Why not just use the FreeBSD packages?

FreeBSD already provides packages for you. Why would you want to use custom packages? The answer is: to get what you want.

For example, I want PostgreSQL with Bacula. I want an Apache module with my PHP.

This is why you want to compile from source. poudriere can automate this and do it much faster than you can. It can build ports in parallel, using the options you specify, and leave you with packages which you can install later.

Why install from packages? It’s faster than compiling from source. It also means less downtime for your server.

poudriere lets you maintain different lists of packages for different servers (e.g. your mail server versus your database server). You can compile for multiple versions of FreeBSD (you can compile for systems older than your poudriere server, but not newer; e.g your poudriere server runs on FreeBSD 9.2, you can compile for 9.2, 9.1, 8.4, etc).

A bit about how this works

This is a simplification. If you read the man page, you’ll learn more.

poudriere builds stuff in a jail. That jail is specify to the version of FreeBSD you are building for. I am building for 9.2, so I will install a 9.2 jail. poudriere needs to know the list of ports you want to build. You can put that list in a file and any dependencies will be built automatically. You can also specify the options you want for a port.

There are many options, but we are going to keep this very simple. poudriere is very feature rich, and it is easy to be overwhelmed by everything it can do.

Installing poudriere

You can install poudriere from either ports or from package. Take your pick. Do not install it into a jail. You’ll have unsurmountable problems if you do.

Configuring poudriere

There are a few configurations to consider:

1. The poudriere configuration file.
2. The SET to use for this build.
3. The make.conf to use when building those ports.
4. The options to use when building those ports.

The poudriere configuration file

Here is my /usr/local/etc/pourdriere.conf file:

#### ZFS
# The pool where poudriere will create all the filesystems it needs
# poudriere will use tank/${ZROOTFS} as its root
#
# You need at least 7GB of free space in this pool to have a working
# poudriere.
#
ZPOOL=system

# root of the poudriere zfs filesystem, by default /poudriere
ZROOTFS=/poudriere

# the host where to download sets  for the jails setup
# You can specify here a host or an IP
# replace _PROTO_ by http or ftp
# replace _CHANGE_THIS_ by the hostname of the mirrors where you want to fetch
# by default: ftp://ftp.freebsd.org
#
# Also not that every protocols supported by fetch(1) are supported here, even
# file:///
FREEBSD_HOST=ftp://ftp.freebsd.org

# By default the jails have no /etc/resolv.conf, you will need to set
# REVOLV_CONF to a file on your hosts system that will be copied has
# /etc/resolv.conf for the jail, except if you don't need it (using an http
# proxy for example)
RESOLV_CONF=/etc/resolv.conf

# The directory where poudriere will store jails and ports
BASEFS=/poudriere

# The directory where the jail will store the packages and logs
# by default a zfs filesystem will be created and set to
# ${BASEFS}/data
#
POUDRIERE_DATA=${BASEFS}/data

# Use portlint to check ports sanity
USE_PORTLINT=no

# Use tmpfs(5)
# This can be a space-separated list of options:
# wrkdir    - Use tmpfs(5) for port building WRKDIRPREFIX
# data      - Use tmpfs(5) for poudriere cache/temp build data
# localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing)
# all       - Run the entire build in memory, including builder jails.
# yes       - Only enables tmpfs(5) for wrkdir
# EXAMPLE: USE_TMPFS="wrkdir data"
USE_TMPFS="wrkdir data localbase"

# If set the given directory will be used for the distfiles this allow the share
# the distfiles between jails and ports tree
DISTFILES_CACHE=/usr/ports/distfiles

# If set the ports tree or source tree marked to use svn will use the defined
# mirror (default: svn0.us-west.FreeBSD.org)
# The full mirror list is available here:
# http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn-mirrors.html
#SVN_HOST=svn0.us-west.FreeBSD.org
SVN_HOST=svn0.us-east.FreeBSD.org

# Disable linux support
NOLINUX=yes

# Define as the URL that your POUDRIERE_DATA/logs is hosted at
# This will be used for giving URL hints to the HTML output when
# scheduling and starting builds
#URL_BASE=http://yourdomain.com/poudriere/
URL_BASE=http://slocum.example.com/logs/

Keep in mind that value for URL_BASE. This is the URL we will use later in our jail host.

The list of ports to build

I am putting my lists into the /usr/local/etc/poudriere.d/buildlists/ directory. Here is the bacula list:

$ cat bacula 
ports-mgmt/pkg
sysutils/bacula-server
sysutils/bacula-client
databases/postgresql93-server

These are the ports I want to compile when I am building for a Bacula server.

The make.conf to use

I also have /usr/local/etc/poudriere.d/releng_9_2-bacula-make.conf (I will explain the filename later):

# Needed prior to FBSD 10
WITH_PKGNG=yes

WANT_PGSQL_VER=93
USE_PGSQL_VER=93

The filename is significant:

• releng_9_2 is the name of the poudreire jail we will create later.
• bacula is the name of the SET we will use for building our ports

The options to use when building those ports

When you set the configuration items for port, those options are [usually] stored at /var/db/ports/. Here are the options I selected for sysutils/bacula-server:

$ cat /var/db/ports/sysutils_bacula-server/options 
# This file is auto-generated by 'make config'.
# Options for bacula-server-5.2.12_3
_OPTIONS_READ=bacula-server-5.2.12_3
_FILE_COMPLETE_OPTIONS_LIST=MTX MYSQL NLS OPENSSL PGSQL PYTHON SQLITE3
OPTIONS_FILE_UNSET+=MTX
OPTIONS_FILE_UNSET+=MYSQL
OPTIONS_FILE_SET+=NLS
OPTIONS_FILE_SET+=OPENSSL
OPTIONS_FILE_SET+=PGSQL
OPTIONS_FILE_UNSET+=PYTHON
OPTIONS_FILE_UNSET+=SQLITE3

At /usr/local/etc/poudriere.d/, I created a directory, bacula-options. This will contain the options specific to my bacula build. Into this directory, I copied /var/db/ports/sysutils_bacula-server from above. I repeated this for each port for which I had the options.

You can also specify those options on the command line via the poudriere options command.

Prepare a ports tree

You can use a number of ports tree with poudriere. Say, one from 8.4 one from 9.2, or your own. I will be using a portsnap provided tree. This command sets up my ports tree, which I will call HEAD. This tree will be updated on a regular basis as required.

poudriere ports -c -p HEAD

This will create a ports tree at /usr/local/poudriere/ports/HEAD. The contents should look similar to this:

CHANGES         converters      net-p2p
CONTRIBUTING.md databases       news
COPYRIGHT       deskutils       palm
GIDs            devel           polish
INDEX-7         dns             ports-mgmt
INDEX-8         editors         portuguese
INDEX-9         emulators       print
Keywords        finance         russian
LEGAL           french          science
MOVED           ftp             security
Makefile        games           shells
Mk              german          sysutils
README          graphics        textproc
Templates       hebrew          ukrainian
Tools           hungarian       vietnamese
UIDs            irc             www
UPDATING        japanese        x11
accessibility   java            x11-clocks
arabic          korean          x11-drivers
archivers       lang            x11-fm
astro           mail            x11-fonts
audio           math            x11-servers
benchmarks      misc            x11-themes
biology         multimedia      x11-toolkits
cad             net             x11-wm
chinese         net-im
comms           net-mgmt

Create the jail for poudriere

This step creates the jail which poudriere uses for building ports.

poudriere jails -c -j releng_9_2 -v 9.2-RELEASE

This will install, via ftp, an instance of FreeBSD 9.2-RELEASE into /poudriere/jails/releng_9_2. The jail is named releng_9_2, a name chosen to reflect the version installed.

Build the packages

This command will build all the ports listed in /usr/local/etc/poudriere.d/buildlists/bacula

poudriere bulk -v -j releng_9_2 -z bacula -p HEAD -f /usr/local/etc/poudriere.d/buildlists/bacula

More information:

• -v : verborse
• -j : the jail to use (i.e. build ports for this release)
• -z : the SET to use (i.e. build these ports)
• -p : the ports tree to use
• -f : the list of ports to build

You will see that not all of the configuration items (mentioned earlier in this post) appear above. This is because of the build options described in the CUSTOMISATION section of the man page. Specifically:

• We created /usr/local/etc/poudriere.d/releng_9_2-bacula-make.conf, which conforms to the /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf optional make.conf file.
• We created /usr/local/etc/poudriere.d/bacula-options, which conforms to the custom build options file /usr/local/etc/poudriere.d/<setname>-options.

This command will run for a while. At the end, and during the build, you can view the output of the logs via the web interface (see my net post for how to set that up).

Did all this work for you?

Serve up the files by http

Sorry, that will be done in a future post. For now, you’ve got your packages built. Congratulations.