Accessing FreeBSD Jails over OpenVPN – Dan Langille's Other Diary

With this new server, I am taking a new approach. Each jail will have at least three IP addresses:

The public IP4 address, used by internet facing services (e.g. http or https)
The public IPv6 address, similar to the above
A VPN address, used for system administration and private services (e.g. nrpe)

In this article, I will assume you are familiar with ezjail, FreeBSD jails, basic networking, OpenVPN, and ZFS. That is, this is not a post for those new to FreeBSD. This is an advanced topic.

The goal is to access the jail, and services in that jail, from within the VPN. This private network will be used for system administration, monitoring, and for backups. This means I can install nrpe and bacula-fd into each jail and access those services over OpenVPN.

Create a jail

Let’s start with the jail creation. We will give it three IP addresses:

162.208.116.67 – the main IP address and the one you’ll connect to once I get a website running on it
2610:1c1:0:4::3 – an IPv6 address for the same place
10.80.0.67 – an IP address for private services, not accessible by the public (e.g. nrpe, bacula-client)

Here is the command to create that jail. I am using a flavour which sets up this jail as an Ansible client / node.

$ sudo ezjail-admin create -f ansible-slave bsdcan 162.208.116.67,2610:1c1:0:4::3,10.80.0.67
/usr/jails/bsdcan/.
/usr/jails/bsdcan/./etc
/usr/jails/bsdcan/./etc/locate.rc
/usr/jails/bsdcan/./etc/crontab
/usr/jails/bsdcan/./etc/gss
...
/usr/jails/bsdcan/./root/.bash_profile
/usr/jails/bsdcan/./root/.bashrc
19 blocks
find: /usr/jails/bsdcan/pkg/: No such file or directory
Warning: IP 162.208.116.67 not configured on a local interface.
Warning: Some services already seem to be listening on all IP, (including 162.208.116.67)
  This may cause some confusion, here they are:
openvpn  openvpn    43169 4  udp4   *:12510               *:*
Warning: IP 2610:1c1:0:4::3 not configured on a local interface.
Warning: Some services already seem to be listening on all IP, (including 2610:1c1:0:4::3)
  This may cause some confusion, here they are:
openvpn  openvpn    43169 4  udp4   *:12510               *:*
Warning: IP 10.80.0.67 not configured on a local interface.
Warning: Some services already seem to be listening on all IP, (including 10.80.0.67)
  This may cause some confusion, here they are:
openvpn  openvpn    43169 4  udp4   *:12510               *:*

I have no idea why they think OpenVPN is on that port.

Next, I will manually add the IP addresses to the system:

ifconfig em0 alias 162.208.116.67/32
ifconfig em0 alias 10.80.0.67/32
ifconfig em0 inet6 2610:1c1:0:4::3 prefixlen 64 alias

To make sure these addresses are assigned at boot time, I added these corresponding entries to /etc/rc.conf:

ifconfig_em0_alias0="inet  162.208.116.67/32"            # bsdcan
ifconfig_em0_alias1="inet  10.80.0.67/32"                # bsdcan
ifconfig_em0_alias2="inet6 2610:1c1:0:4::3 prefixlen 64" # bsdcan

I added a comment to the end of each line so I know what jail belongs to that particular alias.

At this point, this is what the NIC looks like on the jail host:

$ ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
	ether 74:d0:2b:98:19:22
	inet 162.208.116.66 netmask 0xffffff00 broadcast 162.208.116.255
	inet6 fe80::76d0:2bff:fe98:1922%em0 prefixlen 64 scopeid 0x2 
	inet6 2610:1c1:0:4::2 prefixlen 64 
	inet 162.208.116.67 netmask 0xffffffff broadcast 162.208.116.67
	inet6 2610:1c1:0:4::3 prefixlen 64 
	inet 10.80.0.67 netmask 0xffffffff broadcast 10.80.0.67
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)

Start the jail

Let’s start that jail:

$ sudo ezjail-admin onestart bsdcan
Configuring jails:.
Starting jails: bsdcan.
$ jls
   JID  IP Address      Hostname                      Path
     1  162.208.116.67  bsdcan                        /usr/jails/bsdcan

Connect into the jail

Let’s connect into that jail via the console:

$ sudo ezjail-admin in console bsdcan
FreeBSD 9.2-RELEASE (GENERIC) #0 r255898: Thu Sep 26 22:50:31 UTC 2013

Welcome to FreeBSD!

Before seeking technical support, please use the following resources:

o  Security advisories and updated errata information for all releases are
   at http://www.FreeBSD.org/releases/ - always consult the ERRATA section
   for your release first as it's updated frequently.

o  The Handbook and FAQ documents are at http://www.FreeBSD.org/ and,
   along with the mailing lists, can be searched by going to
   http://www.FreeBSD.org/search/.  If the doc package has been installed
   (or fetched via pkg_add -r lang-freebsd-doc, where lang is the
   2-letter language code, e.g. en), they are also available formatted
   in /usr/local/share/doc/freebsd.

If you still have a question or problem, please take the output of
`uname -a', along with any relevant error messages, and email it
as a question to the questions@FreeBSD.org mailing list.  If you are
unfamiliar with FreeBSD's directory layout, please refer to the hier(7)
manual page.  If you are not familiar with manual pages, type `man man'.

Edit /etc/motd to change this login announcement.

root@bsdcan:~ # pkg info
gettext-0.18.3.1               GNU gettext package
libiconv-1.14_1                A character set conversion library
pam_ssh_agent_auth-0.9.5       PAM module which permits authentication via ssh-agent
perl5-5.16.3_4                 Practical Extraction and Report Language
pkg-1.2.3                      New generation package manager
python-2.7_1,2                 The \"meta-port\" for the default version of Python interpreter
python2-2_1                    The \"meta-port\" for version 2 of the Python interpreter
python27-2.7.6_1               Interpreted object-oriented programming language
sudo-1.8.8                     Allow others to run commands as root
root@bsdcan:~ #

Ahh, good. All the right stuff was installed for this particular flavour. Those packages will me to configure this jail via Ansible. But more on that in a later blog post.

Looking at the jail network, it looks as expected:

# ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
	ether 74:d0:2b:98:19:22
	inet 162.208.116.67 netmask 0xffffffff broadcast 162.208.116.67
	inet 10.80.0.67 netmask 0xffffffff broadcast 10.80.0.67
	inet6 2610:1c1:0:4::3 prefixlen 64 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active

Making the Client LAN accessible from the VPN

Let us review the networking we have on this server:

main IP address: 162.208.116.66
OpenVPN tun0: inet 10.8.1.80 –> 10.8.1.81 netmask 0xffffffff
Each jail will have an IP address in the 10.80.0.0/24 subnet (e.g. 10.80.0.66). I will refer to this as the Client LAN.

To route the Client LAN across OpenVPN, we modify the OpenVPN server. No change occurs on the client.

The best thing to read (and I mean read, not skim) is this post on OpenVPN/Routing, specifically Lans behind OpenVPN

These two lines go into openvpn.conf on the server:

route 10.80.0.0 255.255.255.0
push "route 10.80.0.0 255.255.255.0"

This line gets added to the ccd/zuul.example.org file (if you don’t know what ccd refers to here, you skimmed).

iroute 10.80.0.0 255.255.255.0

Restarting both the client and server enables everyone on this VPN to access the Client LAN.

There are a lot of moving parts here, especially with respect to the network traffic over tun0. But this is the first step. You may want to restrict network traffic on your VPN more than I am.

Create a jail

Start the jail

Connect into the jail

Making the Client LAN accessible from the VPN

Leave a Comment Cancel Reply