Creating a vnet jail

Leave a Comment / Networks, Open Source / By

In this post I will be creating a ‘new’ jail which uses vnet. I say ‘new’ because I am actually Converting an iocage jail to a vanilla jail which happens to use vnet.

NOTE: HEADS UP: I failed to get this to work. I’m still going to post it though.

What I did wrong:

  • I should not have used ix1 – that’s the wrong NIC to use. This needs to use an active NIC. In that box, ix1 is not connected.
  • I should not have used ifconfig_ix1_name – nothing wrong with that feature, it’s just chaning too many things at once.

I’m going to repost this article soon with a working solution.

I won’t describe the process of creating the jail. Sorry.

For the record:

  • FreeBSD 12.2
  • No firewall rules on this host

I will outline the settings to get vnet running. I am reading FreBSD Mastery Jails by Michael W Lucas. I’m on page 159 of the 1st edition.

Configure the “fake” NIC

My server (r720-01) is using ix0 as the primary NIC. All the IP addresses are there. ix1 is unused.

I added these to /etc/rc.conf to configure this at boot time:

ifconfig_ix1_name=”jailether”
ifconfig_jailether=”up”

I issued these commands to implement the above:

[dan@r720-01:~] $ ifconfig ix1 
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500	options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ec:f4:bb:c0:bd:a2
	media: Ethernet autoselect
	status: no carrier
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

[dan@r720-01:~] $ sudo ifconfig ix1 name jailether
jailether

[dan@r720-01:~] $ sudo ifconfig jailether up

[dan@r720-01:~] $ sudo ifconfig ix1
ifconfig: interface ix1 does not exist

[dan@r720-01:~] $ sudo ifconfig jailether
jailether: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500	options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ec:f4:bb:c0:bd:a2
	inet6 fe80::eef4:bbff:fec0:bda2%jailether prefixlen 64 scopeid 0x2
	media: Ethernet autoselect
	status: no carrier
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[dan@r720-01:~] $

There, that looks right.

Getting jib to the right place

Next, take the script from share and installing it in a better location:

[dan@r720-01:~] $ sudo cp -ip /usr/share/examples/jails/jib /usr/local/sbin/

[dan@r720-01:~] $ sudo chmod +x /usr/local/sbin/jib

[dan@r720-01:~] $ ls -l /usr/local/sbin/jib
-r-xr-xr-x  1 root  wheel  11928 Dec  8 22:06 /usr/local/sbin/jib

This will be used when the jail starts/stops.

Other NICs

Before I start on the new jail, I wanted to document the other NICs used by the existing pkg01 jail, which already uses vnet.

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:fe:ca:4a:7a:00
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vnet0.1724 flags=143
	        ifmaxaddr 0 port 8 priority 128 path cost 2000
	member: ix0 flags=143
	        ifmaxaddr 0 port 1 priority 128 path cost 2000
	groups: bridge
	nd6 options=1<PERFORMNUD>:
vnet0.1724: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: pkg01 as nic: epair0b
	options=8<VLAN_MTU>
	ether 02:ff:60:b8:73:5b
	hwaddr 02:8b:81:ee:b9:0a
	inet6 fe80::ff:60ff:feb8:735b%vnet0.1724 prefixlen 64 scopeid 0x8
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

jail configuration

This is my jail configuration:

$ cat /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
exec.consolelog="/var/tmp/$name";
mount.devfs;
path = /jails/$name;
allow.sysvipc     = 1;
allow.raw_sockets = 1;
#securelevel = 2;

host.hostname = "$name.int.unixathome.org";

persist;

pkg01 {
    vnet;
    allow.raw_sockets;
    vnet.interface  = "e0b_$name";
    exec.prestart  += "/usr/local/sbin/jib addm    $name jailether";
    exec.poststop  += "/usr/local/sbin/jib destroy $name";
}

Start the jail:

[dan@r720-01:~] $ sudo service jail start pkg01
Starting jails: pkg01.

Two jails same name

Right now, I have two identically named jails:

[dan@r720-01:~] $ jls
   JID  IP Address      Hostname                      Path
  1718  10.55.0.32      pg02.int.unixathome.org       /jails/pg02
  1719  10.55.0.10      mqtt01.int.unixathome.org     /jails/mqtt01
  1720  10.55.0.35      dev-pgeu.int.unixathome.org   /jails/dev-pgeu
  1721  10.55.0.33      bacula-sd-02.int.unixathome.o /jails/bacula-sd-02
  1722  10.55.0.23      keycloak.int.unixathome.org   /jails/keycloak
  1723  10.55.0.34      pg03.int.unixathome.org       /jails/pg03
  1724                  pkg01.int.unixathome.org      /iocage/jails/pkg01/root
  1725                  pkg01.int.unixathome.org      /jails/pkg01
[dan@r720-01:~] $

What’s in the jail?

But we can get to the one I want:

[dan@r720-01:~] $ sudo jexec pkg01
root@pkg01:/ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_pkg01: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:99:5a:9c:1e:0b
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@pkg01:/ #

Notice there is no network configured which is expected:

root@pkg01:/ # grep ifconfig /etc/rc.conf
root@pkg01:/ #

While here, I added this to the jails /etc/rc.conf:

ifconfig_e0b_pkg01="10.55.0.76/24"
defaultrouter="10.55.0.1"

And I restarted the jail:

[dan@r720-01:~] $ sudo service jail restartart pkg01
Stopping jails: pkg01.
Starting jails: pkg01.

The stop was fast, the start took much longer than the first time.

Networking

Here is what I saw on the host after restarting the jail:

jailetherbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:fe:ca:4a:7a:01
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: e0a_pkg01 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 10 priority 128 path cost 2000
	member: jailether flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 2 priority 128 path cost 2000
	groups: bridge
	nd6 options=1<PERFORMNUD>
e0a_pkg01: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:88:f0:5f:fd:0a
	inet6 fe80::88:f0ff:fe5f:fd0a%e0a_pkg01 prefixlen 64 scopeid 0xa
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T )
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The jail networking

Let’s see what the jail has now:

[dan@r720-01:~] $ sudo jexecxec pkg01
root@pkg01:/ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_pkg01: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:88:f0:5f:fd:0b
	inet 10.55.0.76 netmask 0xffffff00 broadcast 10.55.0.255
	inet6 fe80::88:f0ff:fe5f:fd0b%e0b_pkg01 prefixlen 64 scopeid 0x2
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T )
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@pkg01:/ # ping google.ca
^C
root@pkg01:/ # ping 10.55.0.1
PING 10.55.0.1 (10.55.0.1): 56 data bytes
^C
--- 10.55.0.1 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
root@pkg01:/ # 

root@pkg01:/ # netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.55.0.1          UGS    e0b_pkg0
10.55.0.0/24       link#2             U      e0b_pkg0
10.55.0.76         link#2             UHS         lo0
127.0.0.1          link#1             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#1                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
fe80::%e0b_pkg01/64               link#2                        U      e0b_pkg0
fe80::88:f0ff:fe5f:fd0b%e0b_pkg01 link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
root@pkg01:/ #

What’s that under Netif? Why is that e0b_pkg0 and not e0b_pkg01?

Make the jail boot faster

To make the jail boot faster, because I am debugging, I disabled most applications via /etc/rc.conf.

Why doesn’t it work?

Let’s look at the output of jls -n -j JID where JID is the jail id of the jail in question. You can find that in the output of jls.

Here are the main differences I found. The first values are the ones from the working jail.

  • sysvmsg=new vs sysvmsg=inherit
  • sysvsem=inherit vs sysvsem=new
  • sysvshm=inherit vs sysvshm=new
  • allow.mount.linprocfs vs allow.mount.nolinprocfs
  • allow.mount.nullfs vs allow.mount.nonullfs
  • allow.mount.procfs vs allow.mount.noprocfs
  • allow.mount.tmpfs vs allow.mount.notmpfs
  • allow.mount.zfs vs allow.mount.nozfs
  • allow.socket_af vs allow.nosocket_af
  • allow.sysvipc vs allow.nosysvipc

I’ve added all them into the new jail. This is the current configuration:

[dan@r720-01:~] $ cat /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
exec.consolelog="/var/tmp/$name";
mount.devfs;
path = /jails/$name;
# depreacted; see man jail
#allow.sysvipc     = 1;
allow.raw_sockets = 1;
#securelevel = 2;

host.hostname = "$name.int.unixathome.org";

pkg01 {
    vnet;
    allow.raw_sockets;
    allow.socket_af;
    allow.mount.linprocfs;
    allow.mount.nullfs;
    allow.mount.procfs;
    allow.mount.tmpfs;
    allow.mount.zfs;
    sysvmsg=new;
    sysvsem=new;
    sysvshm=new;
    allow.mount;
    allow.mount.devfs;
    host.domainname=none;
    allow.chflags;
    enforce_statfs=1;
    children.max=100;
    vnet.interface   = "e0b_$name";
    exec.prestart   += "/usr/local/sbin/jib addm    $name jailether";
    exec.poststop   += "/usr/local/sbin/jib destroy $name";
}
Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Leave a Comment

Scroll to Top