Adding IPv6 to an Nginx website on FreeBSD / FreshPorts

FreshPorts recently moved to an IPv6-capable server but until today, that capability has not been utilized.

There were a number of things I had to configure, but this will not necessarily be an exhaustive list for you to follow. Some steps might be missing, and it might not apply to your situation.

All of this took about 3 hours.

We are using:

FreeBSD 11.1
Bind 9.9.11
nginx 1.12.2

The server configuration

This is the server configuration:

Incoming http and https connections arrive at the jail host
The connections are redirected to a FreeBSD jail running nginx listening on lo1
nginx listens only on non-routable addresses.

IPv6 gateway

On the FreshPorts server, I had to set the upstream IPv6 gateway using the ipv6_defaultrouter directive in /etc/rc.conf.

Configuration of IPv6 addresses for nginx

At present, nginx listens on 127.1.0.201 via lo1. Note this is lo1, not lo0. I configure lo1 via these directives in /etc/rc.conf:

cloned_interfaces="lo1"
ifconfig_lo1_alias2="inet  127.1.0.201/32" # x8dtu-nginx01

2019-07-31 EDIT: I came back today to see how I did this. I see I am missing part of the command line setup. If you want to configure this before rebooting, and you should reboot to verify your /etc/rc.conf changes, I did this:

[dan@r710-01:~] $ grep lo1 /etc/rc.conf
ifconfig_lo1_alias0="inet  127.1.0.201/32" # ot-recorder will listen on this in mqtt01 jail
cloned_interfaces="lo1"

[dan@r710-01:~] $ ifconfig lo1
ifconfig: interface lo1 does not exist

[dan@r710-01:~] $ sudo ifconfig lo1 create

[dan@r710-01:~] $ ifconfig lo1
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 127.1.0.201 netmask 0xffffffff 
	inet6 fe80::1%lo1 prefixlen 64 scopeid 0x7 
	groups: lo 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[dan@r710-01:~] $

NOTE: the above is on another host, not related to this post, but it shows you how to create lo1 first. I like how the ifconfig lo1 create grabs stuff from /etc/rc.conf to configure it.

Unique local address for the IPv6 address for nginx on this server. I chose fd00::201 and this is what I added to /etc/rc.conf:

ifconfig_lo1_ipv6="inet6 fd00::201 prefixlen 64"         # x8dtu-nginx01

jail configuration

The jail needs to be configured to work with the previously mentioned address on lo1. I am using iocage. This is the command I issued to configure the IPv6 address for that jail:

sudo iocage set ip6_addr="fd00::201" x8dtu-nginx01

After restarting the jail:

sudo iocage restart x8dtu-nginx01

… I saw this inside the jail:

$ ifconfig lo1
lo1: flags=8049 metric 0 mtu 16384
	options=600003
	inet 127.1.0.201 netmask 0xffffffff 
	inet6 fd00::201 prefixlen 64 
	nd6 options=21
	groups: lo 
[dan@x8dtu-nginx01:/usr/local/etc/freshports] $

nginx configuration

I added IPv6 listen directives to my nginx configuration:

server {
  listen 127.1.0.201:80;
  listen [fd00::201]:80;

...
}

server {
  listen 127.1.0.201:443 ssl http2;
  listen [fd00::201]:443 ssl http2;

...
}

Then I tried an nginx configtest:

$ sudo service nginx configtest
Performing sanity check on nginx configuration:
nginx: [emerg] the INET6 sockets are not supported on this platform in "[fd00::201]:80" of the "listen" directive in /usr/local/etc/nginx/includes/freshports.org.conf:3
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed

Oh.

I need to rebuild nginx with IPv6 enabled.

I reverted the nginx configuration changes for now.

The poudriere change

I use poudriere for building my packages. In order to enable the correct configuration item, I consulted the nginx FreshPorts page where I found:

IPV6=on: Enable IPv6 support

That’s odd. It is on by default. But the installed nginx definitely has it turned off:

[dan@x8dtu-nginx01:~] $ pkg info nginx | grep -i ipv6
	IPV6           : off
[dan@x8dtu-nginx01:~] $

I didn’t notice the default until after I had added this directive to /usr/local/etc/poudriere.d/make.conf and rebuilt nginx:

www_nginx_SET+=IPV6

I am not sure how/why that option had been disabled.

After installing the newly built nginx package, I saw:

$ pkg info nginx | grep -i ipv6
Categories     : ipv6 www
	IPV6           : on

The old version was 1.12.2_3,2 and the new version was 1.12.2_4,2; not a significant upgrade.

Restarting nginx

After reconfiguring IPv6 in my nginx configuration, the configtest was clean:

$ sudo service nginx configtest
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

Before the nginx restart, I saw nothing listening on port 443 for IPv6:

$ sockstat -p 443 -6
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS

After the restart, I saw:

$ sockstat -p 443 -6
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
www      nginx      22231 17 tcp6   fd00::201:443         *:*
www      nginx      22230 17 tcp6   fd00::201:443         *:*
www      nginx      22229 17 tcp6   fd00::201:443         *:*
www      nginx      22228 17 tcp6   fd00::201:443         *:*
www      nginx      22227 17 tcp6   fd00::201:443         *:*
www      nginx      22226 17 tcp6   fd00::201:443         *:*
www      nginx      22225 17 tcp6   fd00::201:443         *:*
www      nginx      22224 17 tcp6   fd00::201:443         *:*
root     nginx      22223 17 tcp6   fd00::201:443         *:*

Firewall / packet filter redirect

The next step is getting the host to redirect incoming IPv6 to the jail. I am using pf, my preferred firewall, which originates in OpenBSD.

Here is what I added to /etc/pf.conf:

FRESHPORTS_IPV6="2610:1c1:0:4::124"
...
FRESHPORTS_WWW_JAIL_IPV6="fd00::201"
...
table <WEBSITES_IPV6> { $FRESHPORTS_IPV6 }
...
pass in on $PUBLIC inet6 proto tcp from any to <WEBSITES_IPV6> port http  flags S/SA synproxy state
pass in on $PUBLIC inet6 proto tcp from any to <WEBSITES_IPV6> port https flags S/SA synproxy state
...
rdr pass on $PUBLIC inet6 proto tcp from any to $FRESHPORTS_IPV6 port = http  -> $FRESHPORTS_WWW_JAIL_IPV6
rdr pass on $PUBLIC inet6 proto tcp from any to $FRESHPORTS_IPV6 port = https -> $FRESHPORTS_WWW_JAIL_IPV6

I first tested and then invoked the above changes with these commands:

$ sudo pfctl -f /etc/pf.conf -n
$ sudo pfctl -f /etc/pf.conf

The test

My first test was to browse to https://[2610:1c1:0:4::124]

I got the expected SSL warning because the hostname didn’t match the SSL certificate, and I was redirected to https://www.freshports.org/….

Checking the nginx logs, I saw the incoming IPv6 request, but then I was redirected to an IP4 website, because of this nginx directive:

return 301 https://$server_name$request_uri;

My browser responded appropriately because there was no IPv6 address for FreshPorts. Yet.

Good. This is good. On to the next stage.

Adding the IPv6 address

I’m using a hidden dns master. I update it via nsupdate.

Before I did the following work, here is what DNS found:

$ dig www.freshports.org A www.freshports.org AAAA +short
162.208.116.125

Here is the command I issued to add the AAAA record for FreshPorts.

[dan@[redacted]:~] $ nsupdate -k ~/Kdan.dns.hidden.master.[redacted].key
> server [redacted]
> zone freshports.org.
> update add        freshports.org. 3600 IN AAAA 2610:1c1:0:4::124
> update add    www.freshports.org. 3600 IN AAAA 2610:1c1:0:4::124
> send
> quit
[dan@[redacted]:~] $

After issuing the above command, I used https://www.whatsmydns.net/ to check the DNS, and it showed everything was great.

From a host outside my network, I found the expected results:

dvl@wolfman $ dig www.freshports.org A www.freshports.org AAAA +short
162.208.116.125
2610:1c1:0:4::124
dvl@wolfman $

Good. Looking good.

Eventually, after dns propagation, at home I saw:

$ dig www.freshports.org A www.freshports.org AAAA +short
162.208.116.125
2610:1c1:0:4::124

Looking for traffic

I started looking in the nginx logs for incoming IPv6.

Yep. They were there.

Fallout

I expect some monitoring fallout from this change. I suspect some of my monitoring assumes IP4 and now that IPv6 is available, I need to monitor both IP addresses.