vnet jail on FreeBSD not letting pings through from another non-vnet jail

NOTE: this problem was fixed by moving all the IP addresses to the bridge. See Changing how I use IP address with FreeBSD’s vnet – so IPv6 works

I just restarted my pkg01 jail. Now Nagios can’t see it on IPv6.

In fact, none of the jails on this host can now ping pkg01.

This is FreeBSD 13.1-RELEASE-p6

I did try DAD (based on this previous post, but that’s no help here).

[17:19 r730-01 dvl /etc/jail.conf.d] % sysctl -a | grep -i dad   
net.inet6.ip6.dad_count: 0
net.inet6.ip6.dad_enhanced: 1

pkg01

This is the vnet jail.

[17:08 r730-01 dvl /etc/jail.conf.d] % cat pkg01.conf
pkg01 {

  #
  # start of standard settings for each jail
  #

#  exec.start  = "/bin/sleep 5";
  exec.start += "/bin/sh /etc/rc";
  exec.stop  = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  path = /jails/$name;

  allow.raw_sockets;
  #securelevel = 2;
  
  exec.prestart  = "logger trying to start jail $name...";
  exec.poststart = "logger jail $name has started";
  exec.prestop   = "logger shutting down jail $name";
  exec.poststop  = "logger jail $name has shut down";
  
  host.hostname = "$name.int.unixathome.org";
  exec.consolelog="/var/tmp/jail-console-$name.log";
 
  persist;

  #
  # end of standard settings for each jail
  #

  allow.chflags;

  allow.mount.devfs;
  allow.mount.fdescfs;
  allow.mount.linprocfs;
  allow.mount.nullfs;
  allow.mount.procfs;
  allow.mount.tmpfs;
  allow.mount.zfs=true;
  allow.mount=true;

  allow.raw_sockets;
  allow.socket_af;

  children.max=200;

  enforce_statfs=1;

  exec.created+="zfs jail $name  data03/poudriere";
  exec.created+="zfs set jailed=on data03/poudriere";

  exec.poststart  += "jail -m allow.mount.linprocfs=1 name=$name";

  exec.poststop   += "/usr/local/sbin/jib destroy $name";

  exec.prestart   += "/usr/local/sbin/jib addm  $name igb0";

  host.domainname=none;

  sysvmsg=new;
  sysvsem=new;
  sysvshm=new;

  vnet.interface   = "e0b_$name";
  vnet;
}

Inside /etc/rc.conf of that jail:

[17:16 pkg01 dan ~] % cat /etc/rc.conf
cron_flags="$cron_flags -J 15"

# Disable Sendmail by default
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Run secure syslog
syslogd_flags="-c -ss"

# Enable IPv6
ipv6_activate_all_interfaces="YES"
#hostname="pkg01"
rtsold_enable="NO"
sshd_enable="YES"
daily_clean_hoststat_enable="NO"
postfix_enable="YES"
nginx_enable="YES"

ifconfig_e0b_pkg01="10.55.0.29/24"
ifconfig_e0b_pkg01_ipv6="inet6 2001:470:8abf:7055:c348:9dc1:0:29 prefixlen 64 accept_rtadv"

defaultrouter="10.55.0.1"
#resolv_enable="NO"
zfs_enable="YES"
update_motd="NO"
daily_queuerun_enable="NO"
nrpe_enable="YES"
[17:16 pkg01 dan ~] % 

[17:23 pkg01 dan ~] % ifconfig
lo0: flags=8049 metric 0 mtu 16384
	options=680003
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21
pflog0: flags=0<> metric 0 mtu 33160
	groups: pflog
e0b_pkg01: flags=8863 metric 0 mtu 1500
	options=8
	ether 0e:20:77:ef:c9:54
	hwaddr 02:54:fe:54:2e:0b
	inet 10.55.0.29 netmask 0xffffff00 broadcast 10.55.0.255
	inet6 fe80::c20:77ff:feef:c954%e0b_pkg01 prefixlen 64 scopeid 0x3
	inet6 2001:470:8abf:7055:c348:9dc1:0:29 prefixlen 64
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T )
	status: active
	nd6 options=23
[17:24 pkg01 dan ~] % 

The Nagios jail

The jail for Nagios:

[17:09 r730-01 dvl /etc/jail.conf.d] % cat webserver.conf 
webserver {

  #
  # start of standard settings for each jail
  #

  exec.start = "/bin/sh /etc/rc";
  exec.stop  = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  path = /jails/$name;

  allow.raw_sockets;
  #securelevel = 2;
  
  exec.prestart  = "logger trying to start jail $name...";
  exec.poststart = "logger jail $name has started";
  exec.prestop   = "logger shutting down jail $name";
  exec.poststop  = "logger jail $name has shut down";
  
  host.hostname = "$name.int.unixathome.org";
  exec.consolelog="/var/tmp/jail-console-$name.log";
 
  persist;

  #
  # end of standard settings for each jail
  #

    ip4.addr = "igb0|10.55.0.3";
    ip6.addr = "igb0|2001:470:8abf:7055:b6f9:d572:6622:ea2d";
}
[17:17 r730-01 dvl /etc/jail.conf.d] % 

sample pings

From my laptop:

[pro02 dan ~] % ping6 pkg01
PING6(56=40+8+8 bytes) 2001:470:8abf:4054:9ce6:15d9:82a3:9e87 --> 2001:470:8abf:7055:c348:9dc1:0:29
16 bytes from 2001:470:8abf:7055:c348:9dc1:0:29, icmp_seq=0 hlim=63 time=1791.941 ms
16 bytes from 2001:470:8abf:7055:c348:9dc1:0:29, icmp_seq=1 hlim=63 time=791.876 ms
16 bytes from 2001:470:8abf:7055:c348:9dc1:0:29, icmp_seq=2 hlim=63 time=1.259 ms
...

From the Nagios jail:

[17:07 webserver dan ~] % ping 2001:470:8abf:7055:c348:9dc1:0:29
PING6(56=40+8+8 bytes) 2001:470:8abf:7055:b6f9:d572:6622:ea2d --> 2001:470:8abf:7055:c348:9dc1:0:29
^C
--- 2001:470:8abf:7055:c348:9dc1:0:29 ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

Inside the pkg01 jail

[17:16 pkg01 dan ~] % ping6 google.ca
PING6(56=40+8+8 bytes) 2001:470:8abf:7055:c348:9dc1:0:29 --> 2607:f8b0:4006:80b::2003
16 bytes from 2607:f8b0:4006:80b::2003, icmp_seq=0 hlim=119 time=32.254 ms
16 bytes from 2607:f8b0:4006:80b::2003, icmp_seq=1 hlim=119 time=8.247 ms
16 bytes from 2607:f8b0:4006:80b::2003, icmp_seq=2 hlim=119 time=9.666 ms
^C
--- google.ca ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 8.247/16.722/32.254/10.997 ms
[17:23 pkg01 dan ~] % ping6 pkg01    
PING6(56=40+8+8 bytes) 2001:470:8abf:7055:c348:9dc1:0:29 --> 2001:470:8abf:7055:c348:9dc1:0:29
16 bytes from 2001:470:8abf:7055:c348:9dc1:0:29, icmp_seq=0 hlim=64 time=0.046 ms
^C
--- pkg01.int.unixathome.org ping6 statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.046/0.046/0.046/0.000 ms
[17:23 pkg01 dan ~] %