jail: execv: /bin/sh: No such file or directory

Recently, I was moving some jails from one server to another.

First, I created the jail using the -x switch, which indicates the jail already exists, don’t create it, just add entries for it:

# ezjail-admin create -f bacula -x mysql41.example.org 10.35.0.100
Warning: Some services already seem to be listening on IP 10.35.0.100
  This may cause some confusion, here they are:
root     ntpd       63822 25 udp4   10.35.0.100:123       *:*
Warning: Some services already seem to be listening on all IP, (including 10.35.0.100)
  This may cause some confusion, here they are:
root     ntpd       63822 20 udp4   *:123                 *:*
#

Then, I tried starting that jail:

# /usr/local/etc/rc.d/ezjail.sh start
 ezjailConfiguring jails:.
Starting jails: cannot start jail "mysql41_unixathome_org":
jail: execvp: /bin/sh: No such file or directory
.
[root@kraken /usr/local/etc/ezjail]#

DOH! Why?

Eventually, I read through my old notes and realized: I have never run on jails on this new server.

I had not created the basejail…

ezjail-admin update -ip

This step takes quite some time. Be prepared.

But that was not quite enough. ezjail uses symlinks. Extensively. They are missing. Compare this newly created jail with my old jail:

[root@kraken /storage/compressed/jails]# ls -l pg74.unixathome.org XXXXXXXXXXX.unixathome.org
XXXXXXXXXXX.unixathome.org:
total 41
-rw-r--r--   2 root  wheel   794 Mar 30 18:25 .cshrc
-rw-r--r--   2 root  wheel   261 Mar 30 18:25 .profile
-r--r--r--   1 root  wheel  6196 Mar 30 18:25 COPYRIGHT
drwxr-xr-x   2 root  wheel     2 Mar 30 18:25 basejail
lrwxr-xr-x   1 root  wheel    13 Mar 30 18:25 bin -> /basejail/bin
lrwxr-xr-x   1 root  wheel    14 Mar 30 18:25 boot -> /basejail/boot
dr-xr-xr-x   2 root  wheel     2 Mar 30 18:25 dev
drwxr-xr-x  20 root  wheel   101 Mar 30 18:25 etc
lrwxr-xr-x   1 root  wheel    13 Mar 30 18:25 lib -> /basejail/lib
lrwxr-xr-x   1 root  wheel    17 Mar 30 18:25 libexec -> /basejail/libexec
drwxr-xr-x   2 root  wheel     2 Mar 30 18:25 media
drwxr-xr-x   2 root  wheel     2 Mar 30 18:25 mnt
dr-xr-xr-x   2 root  wheel     2 Mar 30 18:25 proc
lrwxr-xr-x   1 root  wheel    16 Mar 30 18:25 rescue -> /basejail/rescue
drwxr-xr-x   2 root  wheel     6 Mar 30 18:25 root
lrwxr-xr-x   1 root  wheel    14 Mar 30 18:25 sbin -> /basejail/sbin
lrwxr-xr-x   1 root  wheel    11 Mar 30 18:25 sys -> usr/src/sys
drwxrwxrwt   6 root  wheel     6 Mar 30 18:28 tmp
drwxr-xr-x   5 root  wheel    16 Mar 30 18:25 usr
drwxr-xr-x  24 root  wheel    24 Mar 30 18:28 var

pg74.unixathome.org:
total 47
-rw-r--r--   1 root  wheel   794 Jan 22  2010 .cshrc
-rw-r--r--   1 root  wheel   261 Jan 22  2010 .profile
-r--r--r--   1 root  wheel  6202 Jan 22  2010 COPYRIGHT
drwxr-xr-x   2 root  wheel     2 Jan 22  2010 basejail
dr-xr-xr-x   2 root  wheel     2 Jan 22  2010 dev
drwxr-xr-x  21 root  wheel   106 Aug 21  2010 etc
drwxr-xr-x   3 root  wheel     3 Jan 22  2010 home
drwxr-xr-x   2 root  wheel     2 Jan 22  2010 media
drwxr-xr-x   2 root  wheel     2 Jan 22  2010 mnt
dr-xr-xr-x   2 root  wheel     2 Jan 22  2010 proc
drwxr-xr-x   2 root  wheel     9 Jan 23  2010 root
drwxrwxrwt   6 root  wheel   183 Jul 28  2011 tmp
drwxr-xr-x   5 root  wheel     5 Jan 22  2010 usr
drwxr-xr-x  24 root  wheel    24 Jul 27  2011 var
[root@kraken /storage/compressed/jails]#

After creating those symlinks by hand, the jail still would not start:

# /usr/local/etc/rc.d/ezjail.sh start
 ezjailConfiguring jails:.
Starting jails: mysql41.unixathome.org.
# jls
   JID  IP Address      Hostname                      Path

Hmm, starting the jail manually yields a clue:

# jail /storage/compressed/jails/mysql41.unixathome.org mysql41.unixathome.org 10.55.0.100 /bin/sh
Cannot read termcap database;
using dumb terminal settings.
# 

The solution: create symlinks by hand:

cd /storage/compressed/jails/mysql41.unixathome.org
ln -s /basejail/bin
ln -s /basejail/boot
ln -s /basejail/lib
ln -s /basejail/libexec
ln -s /basejail/rescue
ln -s /basejail/sbin
ln -s usr/src/sys

cd /storage/compressed/jails/mysql41.unixathome.org/usr
ln -s /basejail/usr/bin
ln -s /basejail/usr/games
ln -s /basejail/usr/include
ln -s /basejail/usr/lib
ln -s /basejail/usr/lib32
ln -s /basejail/usr/libdata
ln -s /basejail/usr/libexec
ln -s /basejail/usr/ports
ln -s /basejail/usr/sbin
ln -s /basejail/usr/share
ln -s /basejail/usr/src

Now the jail starts.