Are FreeBSD jails being shutdown properly?

This was done on FreeBSD 9.2-RELEASE-p10.

After my boss told me about losing qmail logs from a jail some time ago when a server was shutdown, I started looking at some tests to ensure that jails are shutdown properly. I am beginning to suspect they are not.

Consider this test. I added this to /usr/local/etc/rc.d/nullmailer in the jail named ‘testing’:

echo $(date) $0 $1 >> /var/log/shutdown.log

And then did this on the jail host system:

# touch /usr/jails/testing/var/log/shutdown.log

Then I restarted the jail:

$ sudo ezjail-admin ail-admin restart testing
Stopping jails: testing.
Configuring jails:.
Starting jails: testing.

The log contained:

Sat Jul 26 20:07:19 UTC 2014 /etc/rc quietstart

That is, shutdown, just a start up.

The full script is:

[root@testing /usr/local/etc/rc.d]# less nullmailer
#!/bin/sh

# $FreeBSD: head/mail/nullmailer/files/nullmailer.in 340872 2014-01-24 00:14:07Z mat $
#
# PROVIDE: nullmailer
# REQUIRE: LOGIN
# KEYWORD: shutdown
#
# Add the following line to /etc/rc.conf to enable nullmailer:
#
# nullmailer_enable="YES"
#

echo $(date) $0 $1 >> /var/log/shutdown.log

. /etc/rc.subr

name=nullmailer
rcvar=nullmailer_enable

command=/usr/sbin/daemon
procname=/usr/local/sbin/nullmailer-send
command_args=" -cf -u nullmail /bin/sh -c '$procname | /usr/bin/logger -i -p mail.info -t $name'"

load_rc_config "$name"

: ${nullmailer_enable="NO"}

run_rc_command "$1"

Addenda 6:36pm

After some searching and testing, I discovered this setting in /etc/rc.conf ensures that each jail is properly shutdown:

jail_exec_stop="/bin/sh /etc/rc.shutdown"

I am astounded this is not the default setting. It breaks POLA in my not so humble opinion.

NOTE: this setting can be applied to an individual jail via:

jail_JAILNAME_exec_stop="/bin/sh /etc/rc.shutdown"

where JAILNAME relates to the jail in question.

Addenda 2014-Jun-27 11:29am

Allan Jude pointed out on both Twitter and IRC that ezjail has a configuration setting for this very thing. For my jail, testing, it was at /usr/local/etc/ezjail/testing:

export jail_testing_exec_stop="/bin/sh /etc/rc.shutdown"

With that setting, the jail is shutdown properly.

Patching FreeBSD

Allan Jude suggested this patch to /etc/rc.d/jail around line 92. The current code is:

91			#   flexible execution
92			if [ -z "${_exec_start}" ]; then
93				_exec_start="/bin/sh /etc/rc"
94				if [ -z "${_exec_stop}" ]; then
95					_exec_stop="/bin/sh /etc/rc.shutdown"
96				fi
97			fi
98		fi

The amended code:

91			#   flexible execution
92			if [ -z "${_exec_start}" ]; then
93				_exec_start="/bin/sh /etc/rc"
94			fi
95			if [ -z "${_exec_stop}" ]; then
96				_exec_stop="/bin/sh /etc/rc.shutdown"
97			fi
98		fi

This patch sets a default stop command if one is not already set, rather than only doing so if the start command is not set.

Other discussions

I also posted this question on Google+ where Dag-Erling Smørgrav mentioned this issue will not occur in FreeBSD 10.x and 11.x: “The rc script in 10 and 11 automatically generates a jail.conf(5) based on your existing configuration, and uses the correct defaults for exec.start and exec.stop.?”

He also mentioned the rc script from 10.x will work on 9.x but I have not tried that.