FreeBSD jails on non-routable IP addresses

One of my goals with the server I’m setting up is putting non-public services into jails with non-routable IP addresses. Today, I’ve been working on getting PostgreSQL into a jail. The problem I have been grappling with is not putting PostgreSQL into a jail but routing. It took me a while to figure out where I was going wrong. Turns out, it was my NAT rules and, perhaps, my IP address strategy.

The layout

I’m creating a jail with two IP addresses:

  1. 127.1.0.128 – the address on which local jails will contact the PostgreSQL server
  2. 10.80.0.128 – the VPN address, which will be used for system administration

The jail was created with this statement:

ezjail-admin create -f ansible-slave postgresql 'lo1|127.1.0.128,lo1|10.80.0.128'

The PF rules

In order for a non-routable address to access public services (such as DNS, pkg repositories, etc), we will need to do some NAT. Since I’m using pf, I initially added this rule, which fails to do the right thing.

EXT_IF="em0"
nat on $EXT_IF from 127.1.0.0/24 to any -> ($EXT_IF)

Within the jail

After creating and starting the jail, I went in via the console and found this:

[dan@zuul:~] $ sudo ezjail-admin ail-admin console postgresql

root@postgresql:~ # host google.ca
google.ca has address 74.125.226.248
google.ca has address 74.125.226.247
google.ca has IPv6 address 2607:f8b0:4006:801::1018
;; connection timed out; no servers could be reached

root@postgresql:~ # ifconfig lo1
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 127.1.0.128 netmask 0xffffffff 
	inet 10.80.0.128 netmask 0xffffffff 

The tcpdump output from the jail host showed:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes
21:25:58.595139 IP 162.208.116.66.55361 > 96.47.64.10.53: 63107+ A? google.ca. (27)
21:25:58.595519 IP 96.47.64.10.53 > 162.208.116.66.55361: 63107 2/0/0 A 74.125.226.248, A 74.125.226.247 (59)
21:25:58.595707 IP 162.208.116.67.56659 > 96.47.64.10.53: 46894+ AAAA? google.ca. (27)
21:25:58.596104 IP 96.47.64.10.53 > 162.208.116.67.56659: 46894 1/0/0 AAAA 2607:f8b0:4006:801::1018 (55)
21:25:58.596217 IP 10.80.0.66.56776 > 96.47.64.10.53: 17806+ MX? google.ca. (27)
21:25:59.597023 IP 10.80.0.67.56292 > 96.47.65.90.53: 17806+ MX? google.ca. (27)
21:26:04.597955 IP 10.80.0.66.56776 > 96.47.64.10.53: 17806+ MX? google.ca. (27)
21:26:05.598939 IP 10.80.0.67.56292 > 96.47.65.90.53: 17806+ MX? google.ca. (27)

I have no idea why we’re setting 10.80.0.66 in this output. But 162.208.116.66 is the primary IP address of the jail host.

I tried various pf rule changes but failed to get this working.

Bad NAT

Faced with several failed attempts at fixing PF rules, I searched around and found FreeBSD Jail with Single IP. That is where I found my main clue. Looking at the NAT rule on that page, I found:

nat pass on em0 from $NET_JAIL to any -> $IP_PUB

Compare that to my rule:

nat on $EXT_IF from 127.1.0.0/24 to any -> ($EXT_IF)

Ahh, I’m using the NIC, not the IP address. I changed my rule to:

MYSELF="162.208.116.66"
nat on $EXT_IF from 127.1.0.0/24 to any -> $MYSELF

Let’s try the jail again:

The jail

Here’s what I got in the jail this time:

root@postgresql:~ # host google.ca
google.ca has address 74.125.226.248
google.ca has address 74.125.226.247
google.ca has IPv6 address 2607:f8b0:4006:801::1018
;; connection timed out; no servers could be reached
root@postgresql:~ # host google.ca
google.ca has address 74.125.226.247
google.ca has address 74.125.226.248
google.ca has IPv6 address 2607:f8b0:4006:801::1018
google.ca mail is handled by 10 aspmx.l.google.com.
google.ca mail is handled by 50 alt4.aspmx.l.google.com.
google.ca mail is handled by 20 alt1.aspmx.l.google.com.
google.ca mail is handled by 30 alt2.aspmx.l.google.com.
google.ca mail is handled by 40 alt3.aspmx.l.google.com.
root@postgresql:~ # 

Success! NAT is now working.

Better rules

To be strict, I could beef up the rules on this. Not sure if I will, or if it’s advisable.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Leave a Comment

Scroll to Top