Converting an iocage vnet jail to a vnet jail

In this post I will be creating a ‘new’ jail which uses vnet. I say ‘new’ because I am actually Converting an iocage jail to a vanilla jail which happens to use vnet.

NOTE: I originally failed to get this to work. You can read about it.

I’m going to repost this article soon with a working solution.

I won’t describe the process of creating the jail. Sorry.

For the record:

• FreeBSD 12.2
• No firewall rules on this host

I will outline the settings to get vnet running. I am reading FreBSD Mastery Jails by Michael W Lucas. I’m on page 159 of the 1st edition.

What I did different this time

This time I did a few things before starting the below:

• disabled the iocage jail: iocage_enable=”NO”
• I disabled the existing clone and bridge directives:

  #cloned_interfaces="lo1 bridge0"
  cloned_interfaces="lo1"
  #ifconfig_bridge0="addm ix0 up"
  

  These were related to iocage.

• I rebooted the server – this cleared up the two bridges I had configured. I could have done this by hand if I’d known the exact commands, but sometimes you just want to get it done.

Getting jib to the right place

Next, take the script from share and installing it in a better location:

[dan@r720-01:~] $ sudo cp -ip /usr/share/examples/jails/jib /usr/local/sbin/

[dan@r720-01:~] $ sudo chmod +x /usr/local/sbin/jib

[dan@r720-01:~] $ ls -l /usr/local/sbin/jib
-r-xr-xr-x  1 root  wheel  11928 Dec  8 22:06 /usr/local/sbin/jib

This will be used when the jail starts/stops.

jail configuration

This is my jail configuration:

$ cat /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
exec.consolelog="/var/tmp/$name";
mount.devfs;
path = /jails/$name;
# depreacted; see man jail
#allow.sysvipc;
allow.raw_sockets;
#securelevel = 2;

exec.prestart  = "logger trying to start jail $name...";
exec.poststart = "logger jail $name has started";
exec.prestop   = "logger shutting down jail $name";
exec.poststop  = "logger jail $name has shut down";

host.hostname = "$name.int.unixathome.org";

persist;
pkg01 {
    # because I renamed the jail from pkg01 to pkg
    path = /jails/pkg01;
    vnet;
    vnet.interface   = "e0b_$name";
    exec.prestart   += "/usr/local/sbin/jib addm    $name ix0";
    exec.poststop   += "/usr/local/sbin/jib destroy $name";
#    exec.poststart  += "jail -m allow.mount.linprocfs=1 name=$name";
    exec.consolelog  = "/var/tmp/$name";
}

NOTE that ix0 is the primary NIC on this server.

Start the jail:

[dan@r720-01:~] $ sudo service jail start pkg01
Starting jails: pkg01.

What’s in the jail?

Let’s jump into the jail via jexec:

[dan@r720-01:~] $ sudo jexec pkg01
root@pkg01:/ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_pkg01: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:99:5a:9c:1e:0b
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@pkg01:/ # 

Notice there is no network configured which is expected:

root@pkg01:/ # grep ifconfig /etc/rc.conf
root@pkg01:/ # 

While here, I added this to the jails /etc/rc.conf:

ifconfig_e0b_pkg01="10.55.0.76/24"
defaultrouter="10.55.0.1"

Make the jail boot faster

To make the jail boot faster, because I am debugging, I disabled most applications via /etc/rc.conf.

Restarting the jail

I restarted the jail:

[dan@r720-01:~] $ sudo service jail restart pkg01
Stopping jails: pkg01.
Starting jails: pkg01.

The stop was fast, the start took much longer than the first time.

Networking

Here is what I saw on the host after restarting the jail:

ix0bridge: flags=8843 metric 0 mtu 1500
	ether 02:fe:ca:4a:7a:00
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: e0a_pkg01 flags=143
	        ifmaxaddr 0 port 7 priority 128 path cost 2000
	member: ix0 flags=143
	        ifmaxaddr 0 port 1 priority 128 path cost 2000
	groups: bridge
	nd6 options=1
e0a_pkg01: flags=8943 metric 0 mtu 1500
	options=8
	ether 02:20:77:c0:bd:a0
	hwaddr 02:6e:19:e8:d5:0a
	inet6 fe80::6e:19ff:fee8:d50a%e0a_pkg01 prefixlen 64 scopeid 0x7
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T )
	status: active
	nd6 options=21

The jail networking

Let’s see what the jail has now:

[dan@r720-01:~] $ sudo jexecxec pkg01
root@pkg01:/ # ifconfig
lo0: flags=8049 metric 0 mtu 16384
	options=680003
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21
e0b_pkg01: flags=8843 metric 0 mtu 1500
	options=8
	ether 0e:20:77:c0:bd:a0
	hwaddr 02:6e:19:e8:d5:0b
	inet 10.55.0.76 netmask 0xffffff00 broadcast 10.55.0.255
	inet6 fe80::c20:77ff:fec0:bda0%e0b_pkg01 prefixlen 64 scopeid 0x2
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T )
	status: active
	nd6 options=21
root@pkg01:/ # ping google.ca
PING google.ca (172.217.12.195): 56 data bytes
64 bytes from 172.217.12.195: icmp_seq=0 ttl=119 time=10.304 ms
^C
--- google.ca ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 10.304/10.304/10.304/0.000 ms
root@pkg01:/ # netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.55.0.1          UGS    e0b_pkg0
10.55.0.0/24       link#2             U      e0b_pkg0
10.55.0.76         link#2             UHS         lo0
127.0.0.1          link#1             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#1                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
fe80::%e0b_pkg01/64               link#2                        U      e0b_pkg0
fe80::c20:77ff:fec0:bda0%e0b_pkg01 link#2                       UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
root@pkg01:/ # 

What’s that under Netif? Why is that e0b_pkg0 and not e0b_pkg01?

That’s it. Done.

No big deal.

Another way

This also worked for me, as taken from VNET Jail HowTo Part 2: Networking:

pkg01 {
    # because I renamed the jail from pkg01 to pkg
    path = /jails/pkg01;
    vnet;
    allow.raw_sockets;
    vnet.interface = "epair${id}b";

    exec.prestart  += "ifconfig epair${id} create up";
    exec.prestart  += "ifconfig epair${id}a up descr vnet-${name}";
    exec.prestart  += "ifconfig bridge0 addm epair${id}a up";

    exec.poststop   = "ifconfig bridge0 deletem epair${id}a";
    exec.poststop  += "ifconfig epair${id}a destroy";
}

Both worked.