Is your jail not getting an IPv6 address soon enough? Blame DAD.

I have jails with both IP4 and IPv6 addresses. Lately, I’ve been noticing this in /var/log/messages:

Mar 10 13:39:08 besser sshd[50340]: error: Bind to port 22 on :: failed: Can't assign requested address.
Mar 10 13:39:08 besser nrpe[50359]: Starting up daemon
Mar 10 13:39:08 besser nrpe[50359]: Bind to port 5666 on :: failed: Can't assign requested address.

And this in console messages:

(49)Can't assign requested address: AH00072: make_sock: could not bind to address [::]:80
(49)Can't assign requested address: AH00072: make_sock: could not bind to address [::]:443

Looking in the jail, it had IPv6 addresses. What gives?

In this post:

• FreeBDS 13.1

The jail example

I had a jail, similar to this:

besser {
    ip4.addr = "igb0| 203.0.113.23";
    ip6.addr = "igb0| 2001:DB8:8abf::1";
}

After the jail started, I could see both IP addresses on the specified NIC:

[besser dan ~] % ifconfig igb0
igb0: flags=8963 metric 0 mtu 1500
	options=4a500b9
	ether ec:f4:bb:ef:c9:54
	inet 203.0.113.23 netmask 0xffffffff broadcast 10.55.0.27
	inet6 2001:DB8:8abf::1 prefixlen 128
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=23

Checking with sockstat confirmed the service was not listening on IPv6:

[13:33 r730-01 dvl /etc/jail.conf.d] % sudo jexec besser sockstat -p 5666 -6
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      

My theory: the IP addresses are not being assigned fast enough. Let’s try delaying them.

What I tried

I tried adding this to the jail’s /etc/rc.conf:

[besser dan ~] % grep netwait /etc/rc.conf
netwait_enable="YES"
netwait_if="igb0"
netwait_ip="fe80::12:32ad:5324:9281%igb0"

I also tried a non-link-local IPv6 address.

That didn’t help.

I posted on Mastodon.

That helped.

The fix

The fix was DAD-related. However, see the end of this post for a solution which does not disable DAD.

DAD is Duplicate Address Detection, an IPv6 feature, as documented in RFC 4429

dch suggested trying net.inet6.ip6.dad_count=0. So I did:

[13:42 r730-01 dvl /etc/jail.conf.d] % sudo sysctl net.inet6.ip6.dad_count=0
net.inet6.ip6.dad_count: 1 -> 0
[15:37 r730-01 dvl /etc/jail.conf.d] % sudo service jail restart besser                    
Stopping jails: besser.
Starting jails: besser.

That worked. No more messages as described at the top of this post.

Better still, the services were listening on IPv6:

[besser dan ~] % sockstat -p 5666,22 -46
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
dan      sshd       19026 4  tcp4   203.0.113.23:22       203.0.113.23:54464
root     sshd       19023 4  tcp4   203.0.113.23:22       203.0.113.23:54464
nagios   nrpe3      17207 5  tcp6   2001:DB8:8abf::1:5666 *:*
nagios   nrpe3      17207 6  tcp4   203.0.113.23:5666      *:*
root     sshd       17195 3  tcp6   2001:DB8:8abf::1:22 *:*
root     sshd       17195 4  tcp4   203.0.113.23:22        *:*
?        ?          ?     ?  tcp6   2001:DB8:8abf::1:5666 2001:DB8:8abf::80:19385
?        ?          ?     ?  tcp6   2001:DB8:8abf::1:5666 2001:DB8:8abf::80:19386
?        ?          ?     ?  tcp6   2001:DB8:8abf::1:5666 2001:DB8:8abf::80:19387

Making it stick

This change to /etc/sysctl.conf will make sure that sysctl is set upon reboot.

# to allow jails to get their IP addresses installed and running before the
# daemons start up
#
# (49)Can't assign requested address: AH00072: make_sock: could not bind to address [::]:80
# (49)Can't assign requested address: AH00072: make_sock: could not bind to address [::]:443
# Mar 10 13:30:45 besser sshd[19183]: error: Bind to port 22 on :: failed: Can't assign requested address.
# Mar 10 13:30:45 besser nrpe[19194]: Bind to port 5666 on :: failed: Can't assign requested address.

net.inet6.ip6.dad_count=0

Fixing it without disabling DAD

DAD can be useful, especially when you do not maintain tight control over all directly attached networks. (so said crest). He suggest a pause when starting the jail. Here is a pause:

  exec.start = "/bin/sleep 5";
  exec.start += "/bin/sh /etc/rc";

I added that to my jail configuration file. It worked. I’ll keep it. I’ll also reenable DAD.

crest also told me that DAD can be disabled for a single interface with ifconfig $iface inet6 no_dad.