Fixing vulns in poudriere jails – Dan Langille's Other Diary

This post was originally a few tweets. It’s not really about upgrading vulnerabilities in poudriere jails either. Read this as if each paragraph was a tweet.

When a FreeBSD security alert comes out, or a package is marked as vulnerable, I try to get that fixed as soon as I can. Even if not using the feature. Sometimes a vuln can be leverages against something you are using. Patch it.

When it comes to my poudriere jails, I don’t.

I forget about them. Yesterday I remembered. I updated them and you can see the timestamps in this screen shot.

When a poudriere bulk is run, if the jail has been updated since the last run, all packages get rebuilt. This is by design. I don’t know if you can bypass that.

This is what my build server looks like now:

That’s 82 jails running in this jail (known as pkg01). On the host itself, there are 6 other jails running on this Dell R720 (hardware details here).

The cronjob

The build is started from a cron job which starts at 0408 UTC:

[dan@pkg01:~] $ sudo crontab -l -u root
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/home/dan/bin
# use /bin/sh to run commands, overriding the default set by cron
SHELL=/bin/sh
# mail any output to `dan', no matter whose crontab this is
MAILTO=dan

18  4 * * * /usr/bin/lockf -t 0 /tmp/.poudriere.build /root/bin/poudriere-builds.sh
[dan@pkg01:~] $

lockf?

You might ask, why lockf?

What’s lockf you might ask?

Any time you have a script which should never run concurrently with itself, lockf can help. See also cron is running all jobs twice – solved.

The script

What’s that job do?

This:

[dan@pkg01:~] $ cat /root/bin/poudriere-builds.sh
#!/bin/sh

/root/bin/poudriere-refresh-trees           2>&1 >> /usr/local/poudriere/data/cronjob-logs/poudriere-refresh-trees.log
#/root/bin/poudriere-builds-121-amd64.sh    2>&1 >> /usr/local/poudriere/data/cronjob-logs/poudriere-builds-121-amd64.log
#/root/bin/poudriere-builds-121-i386.sh     2>&1 >> /usr/local/poudriere/data/cronjob-logs/poudriere-builds-121-i386.log
/root/bin/poudriere-builds-122-amd64.sh     2>&1 >> /usr/local/poudriere/data/cronjob-logs/poudriere-builds-122-amd64.log
/root/bin/poudriere-builds-122-i386.sh      2>&1 >> /usr/local/poudriere/data/cronjob-logs/poudriere-builds-122-i386.log
/root/bin/poudriere-builds-13-amd64.sh      2>&1 >> /usr/local/poudriere/data/cronjob-logs/poudriere-builds-13-amd64.log
/root/bin/poudriere-builds-13-i386.sh       2>&1 >> /usr/local/poudriere/data/cronjob-logs/poudriere-builds-13-i386.log
[dan@pkg01:~] $

First, it refreshes the ports tree, all of them:

[dan@pkg01:~] $ cat /root/bin/poudriere-refresh-trees
#!/bin/sh

. /root/bin/config.sh

# updates the ports trees
for tree in ${TREES}
do
  ${POUDRIERE} ports -u -p ${tree}
done
[dan@pkg01:~] $

config.sh

What’s this config.sh eh?

[dan@pkg01:~] $ cat /root/bin/config.sh
#!/bin/sh

POUDRIERE="/usr/local/bin/poudriere"

TREES="default testing"

I386_SETS="master-list-i386"

NEW_SIMPLE_SETS="master-list mysql57 mysql80 pg96 pg10 pg11 pg12 pg13 unifi"
[dan@pkg01:~] $

We’ll come back to those sets later

The ports trees

I don’t use the testing ports tree for building, but it is nice to get it updated from time to time.

These are the ports trees I have:

[dan@pkg01:~] $ poudriere ports -l
PORTSTREE METHOD    TIMESTAMP           PATH
default   git       2021-07-09 04:18:05 /usr/local/poudriere/ports/default
dvl       null      2021-01-01 15:02:13 /usr/home/dan/ports/head
dvl-git   null      2021-04-07 15:15:02 /usr/home/dan/ports/ports
testing   git+https 2021-07-09 04:18:08 /usr/local/poudriere/ports/testing
[dan@pkg01:~] $

A build script

Let’s look at one of the build scripts:

[dan@pkg01:~] $ cat /root/bin/poudriere-builds-122-amd64.sh
#!/bin/sh

. /root/bin/config.sh

for set in ${NEW_SIMPLE_SETS}
do
  ${POUDRIERE} bulk -j 122amd64 -p default -z ${set} -f /usr/local/etc/poudriere.d/buildlists/${set}
done
[dan@pkg01:~] $

This builds several sets of ports. The main list, used for general consumption is master-list. See NEW_SIMPLE_SETS in config.sh above.

The mysql57 mysql80 pg96 pg10 pg11 pg12 pg13 sets all correspond to various Bacula regression tests that run daily. Those sets ensure that each of the 7 jails (each of the same name as their corresponding poudriere set) get what they need and that the packages are built against the corresponding version of the database.

The unifi set is for the Unifi Controller which I run in a jail. The theory was: don’t build all that stuff with all the other stuff I use, since it’s Java etc. This might not be much of a time/space saver I thought it might be.

These builds have been running for 17 hours as I type this. That’s a lot of building. 7 sets, on four different architectures, that’s 28 builds. Each with between 150 and 980 packages. It can take a while.

This is what top looked like earlier:

CPU

This is the CPU usage. It tops out that 100% from time to time, but not all the time.

Power consumption

This graph covers the last 24 hours. The build is still going and it’s on its last set now.

The power consumption varies considerable and is directly related to CPU usage as shown above.

I have the server configured to let one power supply idle when not required.

The other CPU is pretty standard:

Disk I/O

The disk I/O is for the same period as above. Disk IO is not that high compared to non-build times.

EDIT: The next day – 2021.07.10

In comparison, the next days builds were completed by 04:51. Given they start at 04:18, that’s 33 minutes of building for 28 poudriere sets.

[dan@pkg01:/usr/local/poudriere/data/cronjob-logs] $ ls -l *.log
-rw-r--r--  1 root  wheel  28975 Jul 10 04:34 poudriere-builds-122-amd64.log
-rw-r--r--  1 root  wheel   2487 Jul 10 04:34 poudriere-builds-122-i386.log
-rw-r--r--  1 root  wheel  28708 Jul 10 04:51 poudriere-builds-13-amd64.log
-rw-r--r--  1 root  wheel   2420 Jul 10 04:51 poudriere-builds-13-i386.log
-rw-r--r--  1 root  wheel    120 Jul 10 04:18 poudriere-refresh-trees.log
[dan@pkg01:/usr/local/poudriere/data/cronjob-logs] $