ezjail-admin: moving a jail between hosts with archive

I’m in the process of moving a system from one server to another. Today, I’m going to try moving a jail using ezjail-admin archive.

The two systems

I am copying jails between these two systems:

source: FreeBSD 8.4-RELEASE
destination: FreeBSD 9.1-RELEASE-p4

I really am interested to see if this works.

Creating the archive

Best practice is to stop the jail. You can archive a running jail, but I content that you will get best results from a stopped jail.

# ezjail-admin stop building.example.org
Stopping jails: building.example.org.
# ezjail-admin archive building.example.org

Copying the archive

The archive is automatically placed in the ezjail_archivedir directory (see /usr/local/etc/ezjail.conf):

# cd /usr/jails/ezjail_archives
# ls -lh
total 687079
-rw-r--r--  1 root  wheel   670M Jul 26 12:26 building_example_org-201307261222.01.tar.gz
#

I copied that file to the new server:

$ scp building_example_org-201307261222.01.tar.gz knew:

Using ZFS? Create a new filesystem

Since I am using ZFS, it’s a good idea to create a new filesystem for the jails. Why? Because you can. Snapshots. Control. It’s all good.

# zfs create system/usr/jails
# zfs set compression=lzjb system/usr/jails
# zfs get compression system/usr/jails
NAME              PROPERTY     VALUE     SOURCE
system/usr/jails  compression  lzjb      local

I also set compression on.

And then, I took a snapshot, so I can rollback, should I mess up.

# zfs snapshot system/usr/jails@BeforeInstall

Setting up ezjail

Please ignore this step if you already have ezjail installed and configured. I hadn’t run an install yet, and did not have /usr/jails create, nor any basejails or jail flavors.

Because I am using ZFS, I set these items in /usr/local/etc/ezjail.conf:

ezjail_use_zfs="YES"
ezjail_use_zfs_for_jails="YES"
ezjail_jailzfs="system/usr/jails"

The value for ezjail_jailzfs relates to the ‘zfs create’ I did above.

I already had ezjail installed. I had not created any jails, but I don’t think that’s relevant.

I am running ZFS, so I decided to take a snapshot before I proceeded:

# zfs snapshot system/usr@beforeFirstJail

The filesystem name is specific to my configuration. Yours may vary, and it probably will.

With that snapshot taken, I’m ready to install my first jail, from the archive I just copied over.

First attempt fails

Here is what happened when I first tried. The -p option will also fetch a ports tree and install it into the base jail.

$ sudo ezjail-admin install -p
Password:
Your system is 9.1-RELEASE-p4. Normally FTP-servers don't provide non-RELEASE-builds.
Querying your ftp-server... The ftp server you specified (ftp.freebsd.org) seems to provide the following builds:
lrwxr-xr-x    1 520      100            14 Feb 12 08:32 8.3-RELEASE -> ../8.3-RELEASE
drwxrwxr-x    2 980      100           512 May 22 12:59 9.1-RELEASE
drwxrwxr-x    2 1237     100           512 Jul 21 21:47 9.2-BETA1
drwxrwxr-x    5 980      100           512 Jul 21 23:19 ISO-IMAGES
Release to fetch [ 9.1-RELEASE-p4 ]:
fetch: ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/9.1-RELEASE-p4/base.txz: File unavailable (e.g., file not found, no access)
fetch: ftp://ftp.freebsd.org/pub/FreeBSD/snapshot/amd64/amd64/9.1-RELEASE-p4/base.txz: File unavailable (e.g., file not found, no access)
fetch: ftp://ftp.freebsd.org/pub/FreeBSD/amd64/amd64/9.1-RELEASE-p4/base.txz: File unavailable (e.g., file not found, no access)
fetch: ftp://ftp.freebsd.org/releases/amd64/amd64/9.1-RELEASE-p4/base.txz: File unavailable (e.g., file not found, no access)
fetch: ftp://ftp.freebsd.org/snapshots/amd64/amd64/9.1-RELEASE-p4/base.txz: File unavailable (e.g., file not found, no access)
fetch: ftp://ftp.freebsd.org/pub/FreeBSD-Archive/old-releases/amd64/amd64/9.1-RELEASE-p4/base.txz: File unavailable (e.g., file not found, no access)

Could not fetch base from ftp://ftp.freebsd.org.
  Maybe your release (9.1-RELEASE-p4) is specified incorrectly or the host ftp.freebsd.org does not provide that release build.
  Use the -r option to specify an existing release or the -h option to specify an alternative ftp server.

Second attempt

My second attempt was more successful. This time, I specified the release on the command line. I could have entered it at the prompt, as shown in my first attempt, where I just hit ENTER).

$ sudo ezjail-admin install -p -r 9.1-RELEASE
Password:
base.txz                                      100% of   57 MB 4002 kBps 00m00s
lib32.txz                                     100% of 9515 kB  734 kBps 00m00s
/usr/jails/basejail/bin
/usr/jails/basejail/bin/stty
/usr/jails/basejail/bin/domainname
/usr/jails/basejail/bin/mv
/usr/jails/basejail/bin/kenv
....
/usr/jails/basejail/usr/ports/x11/yalias/
/usr/jails/basejail/usr/ports/x11/yeahconsole/
/usr/jails/basejail/usr/ports/x11/yelp/
/usr/jails/basejail/usr/ports/x11/zenity/
Building new INDEX files... done.
$

That succeeded. The build was fetched, and the ports tree installed.

What are the implications of running 9.1-RELEASE jails on a 9.1-RELEASE-p4 system? Not much I bet. I will also try upgrading the jails after this initial install. Complicating the issue, I’m about to install a jail from an 8.4 system.

Using the archive

ezjail-admin install will look for the archive file in the archives directory. The default location for this is /usr/jails/ezjail_archives. Since I do not have that directory, let me set that up, and move the archive I copied earlier.

# zfs create /usr/jails/ezjail_archives
# mv ~dan/building_example_org-201307261222.01.tar.gz /usr/jails/ezjail_archives/

IP address?

If you don’t already have an IP address for this about-to-be-created jail, let’s create one now. I’m going to use the loopback device, and an address of 127.0.0.10. This jail still exists on the other server (but not running). I will eventually assign the real IP address to this jail.

# ifconfig lo0 alias 127.0.0.10 netmask 255.255.255.255
# ifconfig lo0
lo0: flags=8049 metric 0 mtu 16384
        options=600003
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
        inet 127.0.0.1 netmask 0xff000000
        inet 127.0.0.10 netmask 0xffffffff
        nd6 options=21

Now we create the jail:

$ sudo ezjail-admin create -a building_example_org-201307261222.01.tar.gz building.example.org 127.0.0.10
Error: No archive found at building_example_org-201307261222.01.tar.gz.

Oh. So on the archive create, it puts the jail in the ezjail_archives directory, but on the install, it wants a full path. OK fair enough.

 $ sudo ezjail-admin create -a /usr/jails/ezjail_archives/building_example_org-201307261222.01.tar.gz building.example.org 127.0.0.10
pax: ./tmp/ssh-kGzw5MTnF3/agent.55026 skipped. Sockets cannot be copied or extracted
Warning: Some services already seem to be listening on IP 127.0.0.10
  This may cause some confusion, here they are:
root     ntpd       1335  28 udp4   127.0.0.10:123        *:*
Warning: Some services already seem to be listening on all IP, (including 127.0.0.10)
  This may cause some confusion, here they are:
root     sshd       1451  3  tcp6   *:22                  *:*
root     sshd       1451  4  tcp4   *:22                  *:*
root     ntpd       1335  20 udp4   *:123                 *:*
root     ntpd       1335  21 udp6   *:123                 *:*

Yes, I should fix up sshd. ntpd cannot be changed from what I’ve read in the past. It always listens on all interfaces. The actual fix is outside scope. Short answer: set ListenAddress in /etc/ssh/sshd_config and restart sshd.

Start the jail

Let’s start that jail!

$ sudo ezjail-admin start building.example.org
Configuring jails:.
Starting jails: building.example.org.
$ jls
   JID  IP Address      Hostname                      Path
     1  127.0.0.10      building.example.org       /usr/jails/building.example.org
$

Clearly, it is running. But can I connect to it?

Connect to the jail

From the host server, I attempted to ssh into the newly created jail:

$ ssh -A 127.0.0.10
The authenticity of host '127.0.0.10 (127.0.0.10)' can't be established.
ECDSA key fingerprint is 67:7a:66:99:fc:00:be:6d:06:75:f4:be:e6:0b:1d:74.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '127.0.0.10' (ECDSA) to the list of known hosts.
FreeBSD 9.1-RELEASE-p4 (GENERIC) #0: Mon Jun 17 11:42:37 UTC 2013

Welcome to FreeBSD!

Before seeking technical support, please use the following resources:

o  Security advisories and updated errata information for all releases are
   at http://www.FreeBSD.org/releases/ - always consult the ERRATA section
   for your release first as it's updated frequently.

o  The Handbook and FAQ documents are at http://www.FreeBSD.org/ and,
   along with the mailing lists, can be searched by going to
   http://www.FreeBSD.org/search/.  If the doc distribution has
   been installed, they're also available formatted in /usr/share/doc.

If you still have a question or problem, please take the output of
`uname -a', along with any relevant error messages, and email it
as a question to the questions@FreeBSD.org mailing list.  If you are
unfamiliar with FreeBSD's directory layout, please refer to the hier(7)
manual page.  If you are not familiar with manual pages, type `man man'.

You may also use sysinstall(8) to re-enter the installation and
configuration utility.  Edit /etc/motd to change this login announcement.

$ ps auwx
USER     PID %CPU %MEM   VSZ  RSS TT  STAT STARTED    TIME COMMAND
root   66205  0.0  0.0 12052 1248 ??  IsJ   1:43PM 0:00.00 /usr/sbin/syslogd -ss
nagios 66324  0.0  0.0 19144 2564 ??  SsJ   1:43PM 0:00.01 /usr/local/sbin/nrpe2 -c /usr/local/etc/nrpe.cfg -d
root   66358  0.0  0.0 46744 3464 ??  IsJ   1:43PM 0:00.00 /usr/sbin/sshd
root   66369  0.0  0.0 14128 1436 ??  IsJ   1:43PM 0:00.00 /usr/sbin/cron -J 15 -s
root   66424  0.0  0.0 67884 4132 ??  IsJ   1:43PM 0:00.02 sshd: dan [priv] (sshd)
dan    66426  0.0  0.0 67884 4160 ??  SJ    1:43PM 0:00.00 sshd: dan@pts/3 (sshd)
dan    66427  0.0  0.0 14504 1860  3  SsJ   1:43PM 0:00.00 -sh (sh)
dan    66457  0.0  0.0 14188 1348  3  R+J   1:44PM 0:00.00 ps auwx
$ psql -l
psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
$

Yes, I’m in. Some things are running, but not everything, such as PostgreSQL. But I am not shocked. Everything in this jail should be recomplied. That is, all the ports should be refreshed.