While I was moving jails from one host to another, I decided to start using /etc/jail.conf.d/. The idea is each jail gets its own configuration file in that directory, instead of using /etc/jail.conf.
In this post:
- FreeBSD 13.1
But first, you should know
Let’s get some things out in the open before you jump into the change.
The main advantage of /etc/jail.conf.d/ is each jail gets its own configuration file. Easy to find. Easy to update.
If you have global jail configuration items at the top of your existing /etc/jail.conf, they will need to be duplicated in each jail configuration file.
What?
That led me to my ansible solution.
The other issue: there is no using: depend = pg01
If you need order, make the order via jail_list in /etc/rc.conf
The playbook
I created an ansible playbook/template: https://git.langille.org/dvl/ansible/src/branch/main/jail.conf.d
Let’s try the pkg01 jail on the new host.
[r730-01 dvl ~] % sudo service jail start pkg01 Starting jails: cannot start jail "pkg01": 12 ifconfig: interface e0b_pkg01 does not exist jail: pkg01: /sbin/ifconfig e0b_pkg01 vnet pkg01: failed . [r730-01 dvl ~] %
The 12 you see is the jail id which tried to start. Let’s check out e0b and vnet.
Order matters
Why did this happen? I messed up the jail configuration.
By convention, the top of /etc/jail.conf will have common settings which apply to all jails. This allows the jail-specific settings to override them. The key there being: the jail-specific settings are after the common settings. I was not doing that in my /etc/jail.conf.d/pkg01.conf file. Changing that around fixed the above problem and the jail started.
When you look at the resulting configuration, you can see how putting the default stuff at the end would not work.
[r730-01 dvl ~] % cat /etc/jail.conf.d/pkg01.conf
pkg01 {
#
# start of standard settings for each jail
#
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
exec.consolelog="/var/tmp/jail-console-$name.log";
mount.devfs;
path = /jails/$name;
allow.raw_sockets;
#securelevel = 2;
exec.prestart = "logger trying to start jail $name...";
exec.poststart = "logger jail $name has started";
exec.prestop = "logger shutting down jail $name";
exec.poststop = "logger jail $name has shut down";
host.hostname = "$name.int.unixathome.org";
persist;
#
# end of standard settings for each jail
#
allow.chflags;
allow.mount.devfs;
allow.mount.fdescfs;
allow.mount.linprocfs;
allow.mount.nullfs;
allow.mount.procfs;
allow.mount.tmpfs;
allow.mount.zfs=true;
allow.mount=true;
allow.raw_sockets;
allow.socket_af;
children.max=200;
enforce_statfs=1;
exec.created+="zfs jail $name data01/poudriere";
exec.created+="zfs set jailed=on data01/poudriere";
exec.poststart += "jail -m allow.mount.linprocfs=1 name=$name";
exec.poststop += "/usr/local/sbin/jib destroy $name";
exec.prestart += "/usr/local/sbin/jib addm $name igb0";
host.domainname=none;
sysvmsg=new;
sysvsem=new;
sysvshm=new;
vnet.interface = "e0b_$name";
vnet;
}
The exec.prestart directives would be incomplete. In the original error-filled order, only one of the directives would be run:
exec.prestart += "/usr/local/sbin/jib addm $name igb0"; exec.poststart = "logger jail $name has started";
The jib command would never be run and the vnet would not ge configured. The details of what that does and why it is needed is outside scope.
Other stuff
Other things I had to modify for this poudriere jail:
Some zfs file system references from the old zpool name to the new zpool name (e.g. in /usr/local/etc/poudriere.conf and /usr/local/etc/poudriere.d/jails/*/fs
I then had to set mountpoints:
sudo zfs set mountpoint=/usr/local/poudriere data01/poudriere sudo zfs set mountpoint=/usr/ports/distfiles data01/poudriere/distfiles
That then allowed the web interface for poudriere to work. These are the file systems in that jail:
[pkg01 dan ~] % zfs list NAME USED AVAIL REFER MOUNTPOINT data01 1.92T 2.09T 205K none data01/poudriere 258G 2.09T 239K /usr/local/poudriere data01/poudriere/ccache.13amd64 205K 2.09T 205K /usr/local/poudriere/ccache.13amd64 data01/poudriere/ccache.13i386 205K 2.09T 205K /usr/local/poudriere/ccache.13i386 data01/poudriere/ccache.amd64 52.8G 2.09T 47.3G /usr/local/poudriere/ccache.amd64 data01/poudriere/ccache.i386 9.27G 2.09T 6.07G /usr/local/poudriere/ccache.i386 data01/poudriere/data 93.7G 2.09T 29.0G /usr/local/poudriere/data data01/poudriere/data/cache 198M 2.09T 68.6M /usr/local/poudriere/data/cache data01/poudriere/data/cronjob-logs 9.68M 2.09T 3.03M /usr/local/poudriere/data/cronjob-logs data01/poudriere/data/packages 62.9G 2.09T 54.0G /usr/local/poudriere/data/packages data01/poudriere/distfiles 67.6G 2.09T 67.5G /usr/ports/distfiles data01/poudriere/jails 13.0G 2.09T 239K /usr/local/poudriere/jails data01/poudriere/jails/114R 1.76G 2.09T 1.76G /usr/local/poudriere/jails/114R data01/poudriere/jails/121amd64 2.00G 2.09T 2.00G /usr/local/poudriere/jails/121amd64 data01/poudriere/jails/121i386 1.73G 2.09T 1.73G /usr/local/poudriere/jails/121i386 data01/poudriere/jails/131amd64 2.04G 2.09T 2.04G /usr/local/poudriere/jails/131amd64 data01/poudriere/jails/131i386 1.72G 2.09T 1.72G /usr/local/poudriere/jails/131i386 data01/poudriere/jails/13amd64 2.04G 2.09T 2.04G /usr/local/poudriere/jails/13amd64 data01/poudriere/jails/13i386 1.72G 2.09T 1.72G /usr/local/poudriere/jails/13i386 data01/poudriere/ports 21.9G 2.09T 188K /usr/local/poudriere/ports data01/poudriere/ports/default 5.12G 2.09T 3.08G /usr/local/poudriere/ports/default data01/poudriere/ports/main 2.55G 2.09T 2.40G /usr/local/poudriere/ports/main data01/poudriere/ports/testing 14.2G 2.09T 3.06G /usr/local/poudriere/ports/testing data01/poudriere/test 478K 2.09T 205K /usr/local/poudriere/test
When I moved on to the next jail, it complained about the dev directory. The solution: mount the jail. It had just been copied over and was not yet mounted.
r730-01 dvl ~] % sudo service jail start besser Starting jails: cannot start jail "besser": jail: besser: mount.devfs: /jails/besser/dev: No such file or directory . [r730-01 dvl ~] % zfs list | grep besser data02/jails/besser 21.4G 1.38T 18.2G /jails/besser [r730-01 dvl ~] % sudo zfs mount data02/jails/besser [r730-01 dvl ~] % sudo service jail start besse Starting jails: besser. [r730-01 dvl ~] %
I’ve been using this script for each new jail I move. So far, so good.
