One of my goals with the server I’m setting up is putting non-public services into jails with non-routable IP addresses. Today, I’ve been working on getting PostgreSQL into a jail. The problem I have been grappling with is not putting PostgreSQL into a jail but routing. It took me a while to figure out where I was going wrong. Turns out, it was my NAT rules and, perhaps, my IP address strategy.
The layout
I’m creating a jail with two IP addresses:
- 127.1.0.128 – the address on which local jails will contact the PostgreSQL server
- 10.80.0.128 – the VPN address, which will be used for system administration
The jail was created with this statement:
ezjail-admin create -f ansible-slave postgresql 'lo1|127.1.0.128,lo1|10.80.0.128'
The PF rules
In order for a non-routable address to access public services (such as DNS, pkg repositories, etc), we will need to do some NAT. Since I’m using pf, I initially added this rule, which fails to do the right thing.
EXT_IF="em0" nat on $EXT_IF from 127.1.0.0/24 to any -> ($EXT_IF)
Within the jail
After creating and starting the jail, I went in via the console and found this:
[dan@zuul:~] $ sudo ezjail-admin ail-admin console postgresql root@postgresql:~ # host google.ca google.ca has address 74.125.226.248 google.ca has address 74.125.226.247 google.ca has IPv6 address 2607:f8b0:4006:801::1018 ;; connection timed out; no servers could be reached root@postgresql:~ # ifconfig lo1 lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.1.0.128 netmask 0xffffffff inet 10.80.0.128 netmask 0xffffffff
The tcpdump output from the jail host showed:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes 21:25:58.595139 IP 162.208.116.66.55361 > 96.47.64.10.53: 63107+ A? google.ca. (27) 21:25:58.595519 IP 96.47.64.10.53 > 162.208.116.66.55361: 63107 2/0/0 A 74.125.226.248, A 74.125.226.247 (59) 21:25:58.595707 IP 162.208.116.67.56659 > 96.47.64.10.53: 46894+ AAAA? google.ca. (27) 21:25:58.596104 IP 96.47.64.10.53 > 162.208.116.67.56659: 46894 1/0/0 AAAA 2607:f8b0:4006:801::1018 (55) 21:25:58.596217 IP 10.80.0.66.56776 > 96.47.64.10.53: 17806+ MX? google.ca. (27) 21:25:59.597023 IP 10.80.0.67.56292 > 96.47.65.90.53: 17806+ MX? google.ca. (27) 21:26:04.597955 IP 10.80.0.66.56776 > 96.47.64.10.53: 17806+ MX? google.ca. (27) 21:26:05.598939 IP 10.80.0.67.56292 > 96.47.65.90.53: 17806+ MX? google.ca. (27)
I have no idea why we’re setting 10.80.0.66 in this output. But 162.208.116.66 is the primary IP address of the jail host.
I tried various pf rule changes but failed to get this working.
Bad NAT
Faced with several failed attempts at fixing PF rules, I searched around and found FreeBSD Jail with Single IP. That is where I found my main clue. Looking at the NAT rule on that page, I found:
nat pass on em0 from $NET_JAIL to any -> $IP_PUB
Compare that to my rule:
nat on $EXT_IF from 127.1.0.0/24 to any -> ($EXT_IF)
Ahh, I’m using the NIC, not the IP address. I changed my rule to:
MYSELF="162.208.116.66" nat on $EXT_IF from 127.1.0.0/24 to any -> $MYSELF
Let’s try the jail again:
The jail
Here’s what I got in the jail this time:
root@postgresql:~ # host google.ca google.ca has address 74.125.226.248 google.ca has address 74.125.226.247 google.ca has IPv6 address 2607:f8b0:4006:801::1018 ;; connection timed out; no servers could be reached root@postgresql:~ # host google.ca google.ca has address 74.125.226.247 google.ca has address 74.125.226.248 google.ca has IPv6 address 2607:f8b0:4006:801::1018 google.ca mail is handled by 10 aspmx.l.google.com. google.ca mail is handled by 50 alt4.aspmx.l.google.com. google.ca mail is handled by 20 alt1.aspmx.l.google.com. google.ca mail is handled by 30 alt2.aspmx.l.google.com. google.ca mail is handled by 40 alt3.aspmx.l.google.com. root@postgresql:~ #
Success! NAT is now working.
Better rules
To be strict, I could beef up the rules on this. Not sure if I will, or if it’s advisable.
