Creating jails in a jail for testing /etc/jail.conf

Leave a Comment / Jails / By

This is work from April 2021, and sometimes there is a need to do a jail in a jail. This shows how.

I want to testing jail.conf, in a jail. I don’t want to test this on a host, because that might interfere with existing jails. This is related to a FreeBSD code review which will add support for jail.d – that means you can have something like /etc/jail.d.conf/foo.conf which is the configuration file for the jail foo.

In this post:

  1. FreeBSD 12.2
  2. mkjail

Creating the jail

The first step, is creating the jail. The following sections show those commands.

The same steps are using WITHIN the jail, after creation, to create more jails in that jail.

getting the mkjail scripts

I’m creating the jail with mkjail. If you are using your own method, please down to The jail configuration.

Here is how I got started with that:

$ mkdir ~/src
$ cd ~/src
$ git clone https://github.com/mkjail/mkjail.git
$ cd mkjail
$ sudo mkdir -p /var/db/mkjail/flavours/default/etc

You should inspect src/etc/mkjail.conf and see if it meets your needs.

[dan@slocum:~/src/mkjail] $ cat src/etc/mkjail.conf
# mkjail config file

# Set your zpool name
ZPOOL="system"

# Set jail root filesystem path
JAILROOT="/jails"

# Sets you want extracted into new jail
# options include: base, doc, games (deprecated), kernel, lib32, ports, src
SETS="base"
[dan@slocum:~/src/mkjail] $

Specifically, adjust ZPOOL if required.

Creating the jail

$ sudo cp /etc/resolv.conf /var/db/mkjail/flavours/default/etc
$ ./src/bin/mkjail create -v 12.2-RELEASE -j jail-testing -f default
Creating system/jails/jail-testing...
Extracting base into /jails/jail-testing...
Copying in default flavor...
Updating jail-testing jail...

src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 12.2-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.
The following files will be removed as part of updating to
12.2-RELEASE-p6:
/etc/ssl/certs/2c543cd1.0
/etc/ssl/certs/2e4eed3c.0
....
/usr/share/zoneinfo/zone1970.tab
Installing updates...Scanning /jails/jail-testing/usr/share/certs/blacklisted for certificates...
Scanning /jails/jail-testing/usr/share/certs/trusted for certificates...
 done.
...
[dan@slocum:~/src/mkjail] $

The jail configuration

This is my jail configuration, only for testing.

exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
exec.consolelog="/var/tmp/$name";
mount.devfs;
path = /jails/$name;
# depreacted; see man jail
#allow.sysvipc;
allow.raw_sockets;
#securelevel = 2;

host.hostname = "$name.int.unixathome.org";

persist;

jail-testing {
    ip4.addr = "10.0.0.17";
}

Starting the jail

[dan@slocum:~/src/mkjail] $ sudo service jail start jail-testing
Starting jails: jail-testing.

Getting into the jail

[dan@slocum:~/src/mkjail] $ sudo jexec jail-testing
root@jail-testing:/ # ls -l
total 324
-rw-r--r--   2 root  wheel  1089 Oct 23 05:32 .cshrc
-rw-r--r--   2 root  wheel   470 Oct 23 05:32 .profile
-r--r--r--   1 root  wheel  6177 Oct 23 05:39 COPYRIGHT
drwxr-xr-x   2 root  wheel    46 Apr 22 22:49 bin
drwxr-xr-x  10 root  wheel    63 Apr 22 22:49 boot
dr-xr-xr-x  15 root  wheel   512 Apr 23 01:18 dev
drwxr-xr-x  25 root  wheel   104 Apr 23 01:18 etc
drwxr-xr-x   5 root  wheel    62 Apr 22 22:49 lib
drwxr-xr-x   3 root  wheel     4 Oct 23 05:32 libexec
drwxr-xr-x   2 root  wheel     2 Oct 23 05:30 media
drwxr-xr-x   2 root  wheel     2 Oct 23 05:30 mnt
drwxr-xr-x   2 root  wheel     2 Oct 23 05:30 net
dr-xr-xr-x   2 root  wheel     2 Oct 23 05:30 proc
drwxr-xr-x   2 root  wheel   150 Apr 22 22:49 rescue
drwxr-xr-x   2 root  wheel     6 Oct 23 05:39 root
drwxr-xr-x   2 root  wheel   137 Apr 22 22:49 sbin
lrwxr-xr-x   1 root  wheel    11 Oct 23 05:30 sys -> usr/src/sys
drwxrwxrwt   6 root  wheel     6 Apr 23 01:18 tmp
drwxr-xr-x  14 root  wheel    14 Oct 23 05:30 usr
drwxr-xr-x  24 root  wheel    24 Apr 23 01:18 var
root@jail-testing:/ # ifconfig
ix0: flags=8802 metric 0 mtu 1500
	options=e53fbb
	ether 00:25:90:fa:0f:10
	media: Ethernet autoselect
	status: no carrier
ix1: flags=8802 metric 0 mtu 1500
	options=e53fbb
	ether 00:25:90:fa:0f:11
	media: Ethernet autoselect
	status: no carrier
ix2: flags=8843 metric 0 mtu 1500
	options=e53fbb
	ether 00:1b:21:39:a9:c5
	media: Ethernet autoselect (10Gbase-SR )
	status: active
lo0: flags=8049 metric 0 mtu 16384
	options=680003
	groups: lo
pflog0: flags=141 metric 0 mtu 33160
	groups: pflog
wg0: flags=8051 metric 0 mtu 1420
	options=80000
	groups: tun
	Opened by PID 2681
root@jail-testing:/ #

There, that’s the bulk of the work done. That’s easy and straight forward. Now for the complex stuff.

ZFS file system

I want this jail to have control over a ZFS file system. This is where that jail will create jails.

First, I create this jail on the host system:

[dan@slocum:~] $ sudo zfs create -o canmount=off system/data/jail-testing

Jail that filesystem

I amended the /etc/jail.conf entry:

jail-testing {
    ip4.addr = "10.55.0.17";
    allow.mount
    allow.mount.zfs;
    exec.created+="zfs set jailed=on system/data/jail-testing";
    exec.created+="zfs jail $name    system/data/jail-testing";
}

About the above:

  1. allow.mount – allows you to do zfs set mountpoint
  2. allow.mount.zfs – allows you to do zfs mount
  3. zfs set jailed=on – jails the filesystem so you can use it inside the jail
  4. zfs jail $name – associates the jailed filesystem with the given jail

After restarting the jail:

[dan@slocum:~] $ sudo service jail restart jail-testing
Stopping jails: jail-testing.
Starting jails: jail-testing.
[dan@slocum:~] $

And going into the jail, I could see this:

[dan@slocum:~/src/mkjail] $ sudo jexec jail-testing
root@jail-testing:/ # zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
system                    5.34T  12.2T   288K  /
system/data                802G  12.2T   911K  /data
system/data/jail-testing   192K  12.2T   192K  /data/jail-testing
root@jail-testing:/ #

That’s not what I’m looking for. I want a different mountpoint.

root@jail-testing:/ # zfs set mountpoint=/jails system/data/jail-testing
root@jail-testing:/ # zfs mount system/data/jail-testing
root@jail-testing:/ # zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
system                    5.34T  12.2T   288K  /
system/data                801G  12.2T   911K  /data
system/data/jail-testing   192K  12.2T   192K  /jails
root@jail-testing:/ #

That’s better!

Create the first jail inside the jail

Here, I go back up to getting the mkjail scripts and do this:

  1. install the mkjail scripts
  2. configure the src/etc/mkjail.conf zroot value to reflect the filesystem mounted in my jail-testing jail: i.e. ZPOOL=”system/data/jail-testing”
  3. create my new jail in a jail: ./src/bin/mkjail create -v 12.2-RELEASE -j jail1 -f default

jail1 configuration

I’m calling my first jail-in-a-jail jail1 – with a dead simple configuration, which is similar to the one on the host:

exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
path = /jails/$name;
allow.sysvipc     = 1;
allow.raw_sockets = 1;
#securelevel = 2;
 
host.hostname = "$name.jail-testing.int.unixathome.org";

jail1 {
}

Notice how host.hostname uses the name of this jail’s hostname.

Configure for jails within jails

This jail is not configured for jails. Yet. Let’s start with:

root@jail-testing:~/src/mkjail # sysrc jail_enable="YES"
awk: can't open file /etc/rc.conf
 source line number 1
jail_enable: NO -> YES

That enables the jail service. Now, let’s start a jail:

root@jail-testing:/ # jls
   JID  IP Address      Hostname                      Path
root@jail-testing:/ # service jail start jail1
Starting jails: cannot start jail  "jail1": 
jail: jail1: jail_set: Operation not permitted
.
root@jail-testing:/ #

I expected this because I had purposely not copied all of the settings I had used for my poudriere jail.

The above issue was fixed by adding this to the jail in /etc/jail.conf:

children.max=6;

After restarting the jail, I tried again:

[dan@slocum:~] $ sudo jexec jail-testing
root@jail-testing:/ # jls
   JID  IP Address      Hostname                      Path
root@jail-testing:/ # service jail start jail1
Starting jails: jail1.
root@jail-testing:/ # jls
   JID  IP Address      Hostname                      Path
   107  10.55.0.17      jail1.jail-testing.int.unixat /jails/jail1
root@jail-testing:/ #

Success! Can I get into the jail?

root@jail-testing:/ # jexec jail1
root@jail1:/ # ls
.cshrc		bin		etc		media		proc		sbin		usr
.profile	boot		lib		mnt		rescue		sys		var
COPYRIGHT	dev		libexec		net		root		tmp
root@jail1:/ # ps auwwx 
ps: empty file: Invalid argument
root@jail1:/ #

Yes, but you can see that ps does not work.

That is devfs related, I’m sure of it.

Getting devfs working in the jail

First, let’s create a jail within this jail. In this jail, I’m going to repeat the above steps.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Leave a Comment

Scroll to Top