Why don’t my hosts get IPv6 routes?

Leave a Comment / IPv6, Networks / By /

I started this off as a post on FreeBSD Forums, but as I composed it, I decided it was better as a blog post.

This post is most likely not useful to anyone. I solved the problem by rebooting the gateway. I never did find out the cause of the routing issue.

Astute readers will recognized the IP4 and IPv6 addresses in this post – they are taken from RFCs which designate ranges “Reserved for Documentation”:


This post is not meant as instruction. It is only for documentation of the things I tried. I recommend you don’t try what I did.

I’m in the process of setting up a new gateway box. Fortunately, I have the old gateway as a working example and (for better or worse) I’m trying to replicate what it was doing.

This could be as simple as firewall rules.

In this post

In this post, we’re dealing with:

  • gw01 – gateway host
  • r730-01 – a host behind the gateway
  • r730-03 – another host behind the gateway
  • FreeBSD 14.1-RELEASE (r730-03)
  • FreeBSD 14.2-RELEASE (r730-01, gw01)
  • These hosts are on vlan7, which is based upon igc3 on gw01

For example on gw01:

[15:38 gw01 dvl ~] % ifconfig vlan7                                
vlan7: flags=1008943 metric 0 mtu 1500
	description: servers
	options=4600703
	ether 20:7c:14:f5:8e:53
	inet 10.55.0.1 netmask 0xffffff00 broadcast 10.55.0.255
	inet6 fe80::227c:14ff:fef5:8e53%vlan7 prefixlen 64 scopeid 0xe
	inet6 2001:db8:1:7055:: prefixlen 64
	groups: vlan
	vlan: 7 vlanproto: 802.1q vlanpcp: 0 parent interface: igc3
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=23
[16:08 gw01 dvl ~] %

The IPv6 addresses on vlan7 are:

  • gw01 – 2001:db8:1:7055::
  • r730-01 – 2001:db8:1:7055:c348::141
  • r730-03 – 2001:db8:1:7055:6006::1

gw01 is the gateway, running FreeBSD 14.2 – it has a Hurricane Electric IPv6 tunnel configured – it works. (blog post on those details)

rstsold is running on both r730-01 and r730-03:

[16:31 r730-01 dvl ~] % grep rtsold /etc/rc.conf
rtsold_enable="YES"
rtsold_flags='-i -m bridge0'

[16:31 r730-03 dvl ~] % grep rtsold /etc/rc.conf
# enable rtsold and configure it to use the bridge interface
rtsold_enable="YES"
rtsold_flags='-i -m bridge0'

Both of those hosts also have the appropriate default route to gw01:

[16:31 r730-01 dvl ~] % netstat -nr -6 | grep -i default
default                           2001:db8:1:7055::             UGS         bridge0
default                           fe80::227c:14ff:fef5:8e53%bridge0 UG      bridge0
[16:33 r730-01 dvl ~] % 

[16:31 r730-03 dvl ~] % netstat -nr -6 | grep -i default
default                           fe80::227c:14ff:fef5:8e53%bridge0 UG  bridge0

Well, seems to be right.

Outgoing connections

For example, ping6 google.com] works as expected on gw01 – it also works for apple.com & facebook.com – I’m concluding the tunnel and gateway are properly configured for IPv6.

[11:41 gw01 dvl ~] % ping6 google.ca                          
PING(56=40+8+8 bytes) 2001:DB8:1f06:9ea::2 --> 2607:f8b0:4006:807::2003
16 bytes from 2607:f8b0:4006:807::2003, icmp_seq=0 hlim=121 time=6.828 ms
16 bytes from 2607:f8b0:4006:807::2003, icmp_seq=1 hlim=121 time=6.807 ms
^C
--- google.ca ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 6.807/6.818/6.828/0.010 ms
[11:42 gw01 dvl ~] % ping6 apple.com
PING(56=40+8+8 bytes) 2001:DB8:1f06:9ea::2 --> 2620:149:af0::10
16 bytes from 2620:149:af0::10, icmp_seq=0 hlim=60 time=7.483 ms
^C
--- apple.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 7.483/7.483/7.483/0.000 ms
[11:42 gw01 dvl ~] % ping6 facebook.com
PING(56=40+8+8 bytes) 2001:DB8:1f06:9ea::2 --> 2a03:2880:f112:83:face:b00c:0:25de
16 bytes from 2a03:2880:f112:83:face:b00c:0:25de, icmp_seq=0 hlim=54 time=7.752 ms
^C
--- facebook.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 7.752/7.752/7.752/0.000 ms
[11:42 gw01 dvl ~] %

forwarding

Forwarding is enabled:

[11:23 gw01 dvl ~] % sysctl net.inet.ip.forwarding net.inet6.ip6.forwarding
net.inet.ip.forwarding: 1
net.inet6.ip6.forwarding: 1

The gateway is running radvd.

The gateway routes for several vlans. Let’s take the main one as an example, vlan7.

The primary NIC on the r730-01 host accepts router advertisements.

[11:33 r730-01 dvl ~] % ifconfig bridge0 | grep ACC
    nd6 options=23

bridge0? What’s up with that? Read details.

router advertisements

While writing the above, I started a tcpdump on the gateway:

[11:30 gw01 dvl ~] % sudo tcpdump -n -e -ttt -i pflog0 "icmp6 and ip6[40] == 134"        
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), snapshot length 262144 bytes
^C
0 packets captured
476 packets received by filter
0 packets dropped by kernel

Nothing. So I tried to be more accepting:

[11:40 gw01 dvl ~] % sudo tcpdump -n -e -ttt -i pflog0 icmp6                     
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), snapshot length 262144 bytes
 00:00:00.000000 rule 22/0(match): block in on vlan7: fe80::5a9c:fcff:fe10:8c57 > ff02::2: ICMP6, router solicitation, length 16
 00:00:00.419791 rule 22/0(match): block in on vlan7: fe80::21b:21ff:fe41:2ffa > ff02::2: ICMP6, router solicitation, length 16
 00:00:03.591294 rule 22/0(match): block in on vlan7: fe80::5a9c:fcff:fe10:8c57 > ff02::2: ICMP6, router solicitation, length 16
 00:00:00.419533 rule 22/0(match): block in on vlan7: fe80::21b:21ff:fe41:2ffa > ff02::2: ICMP6, router solicitation, length 16
 00:00:03.589754 rule 22/0(match): block in on vlan7: fe80::5a9c:fcff:fe10:8c57 > ff02::2: ICMP6, router solicitation, length 16

Oh, well, that’s a thing. That should not be blocked.

I fixed that with a new firewall rule. In fact, I changed these:

pass inet6 proto icmp6 icmp6-type $icmp6_types

to

pass inet6 proto icmp6 all

Added a default route

I decided to add a default route on r730-01 (often referred to as r730).

[20:32 r730-01 dvl ~] % sudo route -6 add default 2001:db8:1:7055::

I have this:

[20:32 r730-01 dvl ~] % netstat -nr -6 | head                         
Routing tables

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::/96                             link#7                        URS             lo0
default                           2001:db8:1:7055::             UGS         bridge0
default                           fe80::227c:14ff:fef5:8e53%bridge0 UG      bridge0
::1                               link#7                        UHS             lo0
::ffff:0.0.0.0/96                 link#7                        URS             lo0
2001:470:1f07:9ea::/64            link#11                       U           bridge0

However, stuff isn’t getting to where it needs to get. Pings are failing:

[21:16 r730-01 dvl ~] % ping6 2001:db8:1:7055::0
PING(56=40+8+8 bytes) 2001:db8:1:7055:2c42:3ac2:b990:7a0d --> 2001:db8:1:7055::
^C
--- 2001:db8:1:7055::0 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
[21:17 r730-01 dvl ~] %

Next, I found this:

[21:17 r730-01 dvl ~] % sudo tcpdump -n -e -ttt -i bridge0 ip6
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
 00:00:00.000000 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:01.001271 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:01.013978 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
^C
3 packets captured
433 packets received by filter
0 packets dropped by kernel

That’s r730 asking for information and getting no replies.

I see router solicitations as well, with a reply:

00:00:00.479631 58:9c:fc:10:8c:57 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: fe80::5a9c:fcff:fe10:8c57 > ff02::2: ICMP6, router solicitation, length 16
00:00:00.000441 20:7c:14:f5:8e:53 > 58:9c:fc:10:8c:57, ethertype IPv6 (0x86dd), length 190: fe80::227c:14ff:fef5:8e53 > fe80::5a9c:fcff:fe10:8c57: ICMP6, router advertisement, length 136

That reply came from gw01

igc3: flags=1008943 metric 0 mtu 1500
        options=4e427bb
        ether 20:7c:14:f5:8e:53
        inet 10.55.1.1 netmask 0xffffff00 broadcast 10.55.1.255
        inet6 fe80::227c:14ff:fef5:8e53%igc3 prefixlen 64 scopeid 0x4
        media: Ethernet autoselect (1000baseT )
        status: active

Oh, there’s no public IPv6 address on that. Checking notes… I see the old gateway had an IP address on there. Let’s add one. … and I added it to the wrong interface.

[14:49 gw01 dvl ~] % ifconfig igc0
igc0: flags=1008843 metric 0 mtu 1500
	options=4e427bb
	ether 20:7c:14:f5:8e:50
	inet 233.252.103.35 netmask 0xffffff00 broadcast 108.52.204.255
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=29
[14:59 gw01 dvl ~] % sudo ifconfig igc0 inet6 2001:db8:1:1055::
[15:00 gw01 dvl ~] % ifconfig igc0                                             
igc0: flags=1008843 metric 0 mtu 1500
	options=4e427bb
	ether 20:7c:14:f5:8e:50
	inet 233.252.103.35 netmask 0xffffff00 broadcast 108.52.204.255
	inet6 2001:db8:1:1055:: prefixlen 64
	inet6 fe80::227c:14ff:fef5:8e50%igc0 prefixlen 64 scopeid 0x1
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=21
[15:00 gw01 dvl ~] %

Let’s fix that:

[16:44 gw01 dvl ~] % sudo ifconfig igc0 inet6 2001:db8:1:1055:: -alias 
[16:45 gw01 dvl ~] % sudo ifconfig igc3 inet6 2001:db8:1:1055:: prefixlen 64
[16:45 gw01 dvl ~] % sudo ifconfig igc3                                        
igc3: flags=1008943 metric 0 mtu 1500
	options=4e427bb
	ether 20:7c:14:f5:8e:53
	inet 10.55.1.1 netmask 0xffffff00 broadcast 10.55.1.255
	inet6 fe80::227c:14ff:fef5:8e53%igc3 prefixlen 64 scopeid 0x4
	inet6 2001:db8:1:1055:: prefixlen 64
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=23
[16:45 gw01 dvl ~] %

Hmm, let’s add this option (although I see now, this option was not present on the old host):

[15:02 gw01 dvl ~] % sudo ifconfig igc0 inet6 accept_rtadv
[15:03 gw01 dvl ~] % ifconfig igc0                        
igc0: flags=1008843 metric 0 mtu 1500
	options=4e427bb
	ether 20:7c:14:f5:8e:50
	inet 233.252.103.35 netmask 0xffffff00 broadcast 108.52.204.255
	inet6 2001:db8:1:1055:: prefixlen 64
	inet6 fe80::227c:14ff:fef5:8e50%igc0 prefixlen 64 scopeid 0x1
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=23

I later removed that via:

[16:46 gw01 dvl ~] % sudo ifconfig igc0 inet6 -accept_rtadv
[16:47 gw01 dvl ~] % sudo ifconfig igc0                    
igc0: flags=1008843 metric 0 mtu 1500
	options=4e427bb
	ether 20:7c:14:f5:8e:50
	inet 233.252.103.35 netmask 0xffffff00 broadcast 108.52.204.255
	inet6 fe80::227c:14ff:fef5:8e50%igc0 prefixlen 64 scopeid 0x1
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=21
[16:47 gw01 dvl ~] %

Later that day

The grandkids were here, and just left. Let’s start fresh.

r730-01 and r730-03 can ping each other:

[15:53 r730-01 dvl ~] % ping6 -c 5 2001:db8:1:7055:6006::1
PING(56=40+8+8 bytes) 2001:db8:1:7055:2c42:3ac2:b990:7a0d --> 2001:db8:1:7055:6006::1
16 bytes from 2001:db8:1:7055:6006::1, icmp_seq=0 hlim=64 time=0.157 ms
16 bytes from 2001:db8:1:7055:6006::1, icmp_seq=1 hlim=64 time=0.165 ms
16 bytes from 2001:db8:1:7055:6006::1, icmp_seq=2 hlim=64 time=0.171 ms
16 bytes from 2001:db8:1:7055:6006::1, icmp_seq=3 hlim=64 time=0.173 ms
16 bytes from 2001:db8:1:7055:6006::1, icmp_seq=4 hlim=64 time=0.135 ms

--- 2001:db8:1:7055:6006::1 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.135/0.160/0.173/0.014 ms
[15:53 r730-01 dvl ~] %

Careful observation will reveal that the address we are pinging from is not the one shown for r730-01, but instead is for a jail on that host (serpico).

span class=”file”>r730-03 can ping the other host, on both IP addresses:

[15:55 r730-03 dvl ~] % ping6 -c 5 2001:db8:1:7055:2c42:3ac2:b990:7a0d
PING(56=40+8+8 bytes) 2001:db8:1:7055:31da:7e45:0:53 –> 2001:db8:1:7055:2c42:3ac2:b990:7a0d
16 bytes from 2001:db8:1:7055:2c42:3ac2:b990:7a0d, icmp_seq=0 hlim=64 time=0.209 ms
16 bytes from 2001:db8:1:7055:2c42:3ac2:b990:7a0d, icmp_seq=1 hlim=64 time=0.213 ms
16 bytes from 2001:db8:1:7055:2c42:3ac2:b990:7a0d, icmp_seq=2 hlim=64 time=0.165 ms
16 bytes from 2001:db8:1:7055:2c42:3ac2:b990:7a0d, icmp_seq=3 hlim=64 time=0.154 ms
16 bytes from 2001:db8:1:7055:2c42:3ac2:b990:7a0d, icmp_seq=4 hlim=64 time=0.155 ms

— 2001:db8:1:7055:2c42:3ac2:b990:7a0d ping statistics —
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.154/0.179/0.213/0.026 ms
[15:55 r730-03 dvl ~] % 

[15:55 r730-03 dvl ~] % ping6 -c 5 2001:db8:1:7055:c348::141
PING(56=40+8+8 bytes) 2001:db8:1:7055:6006::1 --> 2001:db8:1:7055:c348::141
16 bytes from 2001:db8:1:7055:c348::141, icmp_seq=0 hlim=64 time=0.185 ms
16 bytes from 2001:db8:1:7055:c348::141, icmp_seq=1 hlim=64 time=0.215 ms
16 bytes from 2001:db8:1:7055:c348::141, icmp_seq=2 hlim=64 time=0.183 ms
16 bytes from 2001:db8:1:7055:c348::141, icmp_seq=3 hlim=64 time=0.192 ms
16 bytes from 2001:db8:1:7055:c348::141, icmp_seq=4 hlim=64 time=0.221 ms

--- 2001:db8:1:7055:c348::141 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.183/0.199/0.221/0.016 ms
[16:01 r730-03 dvl ~] %

I’m concluding there is no physical network issue stopping IPv6 from working. I’m sure it’s routing or firewall.

Ping the firewall / gateway / gw01

Let’s try pinging the firewall:

[16:04 r730-01 dvl ~] % ping6 -c 5 2001:db8:1:7055::
PING(56=40+8+8 bytes) 2001:db8:1:7055:2c42:3ac2:b990:7a0d --> 2001:db8:1:7055::

--- 2001:db8:1:7055:: ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss
[16:05 r730-01 dvl ~] %

It is not happening.

Meanwhile, on gw01, I saw this:

[16:04 gw01 dvl ~] % sudo tcpdump -n -e -ttt -i igc3 | grep 2001:db8:1:7055:2c42:3ac2:b990:7a0d
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on igc3, link-type EN10MB (Ethernet), snapshot length 262144 bytes

 00:00:00.000019 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype 802.1Q (0x8100), length 90: vlan 7, p 0, ethertype IPv6 (0x86dd), 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:00.000139 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype 802.1Q (0x8100), length 90: vlan 7, p 0, ethertype IPv6 (0x86dd), 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:00.002941 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype 802.1Q (0x8100), length 90: vlan 7, p 0, ethertype IPv6 (0x86dd), 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:00.002227 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype 802.1Q (0x8100), length 90: vlan 7, p 0, ethertype IPv6 (0x86dd), 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:00.000022 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype 802.1Q (0x8100), length 90: vlan 7, p 0, ethertype IPv6 (0x86dd), 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:00.000459 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype 802.1Q (0x8100), length 90: vlan 7, p 0, ethertype IPv6 (0x86dd), 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32

As show, that’s coming in on vlan7 and nobody is replying to it.

Pinging outside from inside

When I try to ping google.ca:

[17:31 r730-01 dvl ~] % ping google.ca               
PING(56=40+8+8 bytes) 2001:db8:1:7055:c348::141 --> 2607:f8b0:4006:807::2003
^C
--- google.ca ping statistics ---
15 packets transmitted, 0 packets received, 100.0% packet loss

Yet, over on the gateway, I see the replies:

[17:27 gw01 dvl ~] % sudo tcpdump -n -e -ttt -i gif0 host 2607:f8b0:4006:807::2003
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on gif0, link-type NULL (BSD loopback), snapshot length 262144 bytes
 00:00:00.000000 AF IPv6 (28), length 108: 2001:DB8:1f06:9ea::2 > 2607:f8b0:4006:807::2003: ICMP6, destination unreachable, unreachable address 2001:db8:1:7055:c348::141, length 64
 00:00:00.044395 AF IPv6 (28), length 60: 2001:db8:1:7055:c348::141 > 2607:f8b0:4006:807::2003: ICMP6, echo request, id 34583, seq 10, length 16
 00:00:00.007997 AF IPv6 (28), length 60: 2001:db8:1:7055:c348::141 > 2607:f8b0:4006:807::2003: ICMP6, echo request, id 29549, seq 359, length 16
 00:00:00.017310 AF IPv6 (28), length 60: 2607:f8b0:4006:807::2003 > 2001:db8:1:7055:c348::141: ICMP6, echo reply, id 34583, seq 10, length 16
 00:00:00.000379 AF IPv6 (28), length 60: 2607:f8b0:4006:807::2003 > 2001:db8:1:7055:c348::141: ICMP6, echo reply, id 29549, seq 359, length 16
 00:00:00.975324 AF IPv6 (28), length 60: 2001:db8:1:7055:c348::141 > 2607:f8b0:4006:807::2003: ICMP6, echo request, id 34583, seq 11, length 16
 00:00:00.008014 AF IPv6 (28), length 60: 2001:db8:1:7055:c348::141 > 2607:f8b0:4006:807::2003: ICMP6, echo request, id 29549, seq 360, length 16
 00:00:00.000107 AF IPv6 (28), length 60: 2607:f8b0:4006:807::2003 > 2001:db8:1:7055:c348::141: ICMP6, echo reply, id 34583, seq 11, length 16
 00:00:00.008493 AF IPv6 (28), length 60: 2607:f8b0:4006:807::2003 > 2001:db8:1:7055:c348::141: ICMP6, echo reply, id 29549, seq 360, length 16
 00:00:00.984394 AF IPv6 (28), length 60: 2001:db8:1:7055:c348::141 > 2607:f8b0:4006:807::2003: ICMP6, echo request, id 34583, seq 12, length 16
 00:00:00.005976 AF IPv6 (28), length 60: 2607:f8b0:4006:807::2003 > 2001:db8:1:7055:c348::141: ICMP6, echo reply, id 34583, seq 12, length 16

To me, this is gw01 not knowing where to send the replies.

Let’s check routing:

[17:34 gw01 dvl ~] % netstat -nr -6 | grep 2001:db8:1:7055:c348::141              
[17:35 gw01 dvl ~] % netstat -nr -6 | grep 2001:db8:1:7055:c348     
[17:35 gw01 dvl ~] % netstat -nr -6 | grep 2001:db8:1:7055     
2001:db8:1:7055::              link#10                       UHS             lo0
2001:db8:1:7055::/64           2001:db8:1:7055::          UGS           vlan7
2001:db8:1:7055:2c42:3ac2:b990:7a0d 2001:db8:1:7055::     UGHS          vlan7

So there is a route.

And we allow forwarding (as shown earlier):

[17:35 gw01 dvl ~] % sysctl net.inet.ip.forwarding net.inet6.ip6.forwarding
net.inet.ip.forwarding: 1
net.inet6.ip6.forwarding: 1

The base problem

The base problem is the gateway can’t ping hosts and the host can’t ping the gateway.

Let’s try this”

[17:45 r730-01 dvl ~] % traceroute6 2001:db8:1:7055::
traceroute6 to 2001:db8:1:7055:: (2001:db8:1:7055::) from 2001:db8:1:7055:2c42:3ac2:b990:7a0d, 64 hops max, 28 byte packets
 1  2001:db8:1:7055:2c42:3ac2:b990:7a0d  3031.257 ms !A  3000.070 ms !A  3018.985 ms !A

According to the manpage, the !A means Destination Unreachable – Address Unreachable

So, a routing problem?

Perhaps not, dumping shows:

[17:55 r730-01 dvl ~] % sudo tcpdump -n -e -ttt -i bridge0  ip6
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
 00:00:00.000000 20:7c:14:f5:8e:53 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: fe80::227c:14ff:fef5:8e53 > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:00.657337 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:00.344283 20:7c:14:f5:8e:53 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: fe80::227c:14ff:fef5:8e53 > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:00.711584 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
^C

So we’re back to the who has this address problem.

And that request is arriving at the gateway:

[17:45 gw01 dvl ~] % sudo tcpdump -n -e -ttt -i vlan7 ip6   
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vlan7, link-type EN10MB (Ethernet), snapshot length 262144 bytes
 00:00:00.000000 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:01.005543 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32
 00:00:01.011350 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: 2001:db8:1:7055:2c42:3ac2:b990:7a0d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 2001:db8:1:7055::, length 32

Why is gw01 not answering?

It has a route right there, to the whole IP address:

[17:57 gw01 dvl ~] % netstat -nr -6 | grep vlan7
2001:db8:1:7055::/64           2001:db8:1:7055::          UGS           vlan7
2001:db8:1:7055:2c42:3ac2:b990:7a0d 2001:db8:1:7055::     UGHS          vlan7
fe80::%vlan7/64                   link#14                       U             vlan7

However, as pointed out by ivy, that’s a route I added. “i strongly suggest letting the kernel add it rather than doing it by hand, just to be sure” – OK

2025-03-23

Yesterday, I concluded I should reboot the gateway. I’ll do that next time I’m in/near the basement.

Today, I’m observing some tcpdump output.

I saw my webserver ask for directions:

[12:23 gw01 dvl ~] % sudo tcpdump -n -e -ttt -i vlan7 ip6 
...
 00:00:00.018760 58:9c:fc:10:8c:57 > 33:33:ff:00:00:00, ethertype IPv6 (0x86dd), length 86: 
2001:db8:1:7055:b6f9:d572:6622:ea2d > ff02::1:ff00:0: ICMP6, neighbor solicitation, who has 
2001:db8:1:7055::, length 32

This was also seen:

[12:21 gw01 dvl ~] % sudo tcpdump -n -e -ttt -i gif0 ip6
 00:00:00.006716 AF IPv6 (28), length 76: 2001:DB8:1f06:9ea::2 > 2001:db8:1:7055:b6f9:d572:6622:ea2d: ICMP6, 
neighbor advertisement, tgt is 2001:db8:1:7055::, length 32
 00:00:00.000006 AF IPv6 (28), length 76: 2001:DB8:1f06:9ea::2 > 2001:db8:1:7055:b6f9:d572:6622:ea2d: ICMP6, 
neighbor advertisement, tgt is 2001:db8:1:7055::, length 32
 00:00:00.006640 AF IPv6 (28), length 76: 2001:DB8:1f06:9ea::2 > 2001:db8:1:7055:b6f9:d572:6622:ea2d: ICMP6, 
neighbor advertisement, tgt is 2001:db8:1:7055::, length 32
 00:00:00.000007 AF IPv6 (28), length 76: 2001:DB8:1f06:9ea::2 > 2001:db8:1:7055:b6f9:d572:6622:ea2d: ICMP6, 
neighbor advertisement, tgt is 2001:db8:1:7055::, length 32

That is a reply to my webserver, heading out the HE tunnel:

gif0: flags=1008151 metric 0 mtu 1280
        options=80000
        tunnel inet 233.252.103.35 --> 203.0.113.14
        inet6 2001:DB8:1f06:9ea::2 --> 2001:DB8:1f06:9ea::1 prefixlen 128
        inet6 fe80::227c:14ff:fef5:8e50%gif0 prefixlen 64 scopeid 0x12
        groups: gif
        nd6 options=21

I’m convinced it’s a routing problem. I’m back to the title I gave this post.

Next, I found https://freebsdfoundation.org/wp-content/uploads/2022/08/sato_IPv6.pdf where they say:

The ipv6_defaultrouter variable specifies the default router as defaultrouter for IPv4 does. You need an IPv6 address of the router. Usually, this information is not provided explicitly. You can find the router’s address by using ping6(8) utility:

[12:42 gw01 dvl ~] %  ping6 ff02::2%vlan4
PING(56=40+8+8 bytes) fe80::227c:14ff:fef5:8e53%vlan4 --> ff02::2%vlan4
16 bytes from fe80::92ca:faff:fecc:f982%vlan4, icmp_seq=0 hlim=64 time=212.949 ms
16 bytes from fe80::92ca:faff:fecc:f982%vlan4, icmp_seq=1 hlim=64 time=114.028 ms
16 bytes from fe80::92ca:faff:fecc:f982%vlan4, icmp_seq=2 hlim=64 time=31.761 ms
16 bytes from fe80::92ca:faff:fecc:f982%vlan4, icmp_seq=3 hlim=64 time=258.640 ms
16 bytes from fe80::92ca:faff:fecc:f982%vlan4, icmp_seq=4 hlim=64 time=177.360 ms
^C
--- ff02::2%vlan4 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 31.761/158.948/258.640/79.231 ms
[12:43 gw01 dvl ~] % 


[12:38 r730-01 dvl ~] % ping6 ff02::2%bridge0
PING(56=40+8+8 bytes) fe80::5a9c:fcff:fe10:8c57%bridge0 --> ff02::2%bridge0
16 bytes from fe80::3f7:a5e:4649:ad46%bridge0, icmp_seq=0 hlim=64 time=0.360 ms
16 bytes from fe80::3f7:a5e:4649:ad46%bridge0, icmp_seq=1 hlim=64 time=0.458 ms
16 bytes from fe80::3f7:a5e:4649:ad46%bridge0, icmp_seq=2 hlim=64 time=0.383 ms
^C
--- ff02::2%bridge0 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.360/0.400/0.458/0.042 ms

[20:08 r730-03 dvl ~] % ping6 ff02::2%bridge0
PING(56=40+8+8 bytes) fe80::21b:21ff:fe41:2ffa%bridge0 --> ff02::2%bridge0
16 bytes from fe80::3f7:a5e:4649:ad46%bridge0, icmp_seq=0 hlim=64 time=0.411 ms
16 bytes from fe80::3f7:a5e:4649:ad46%bridge0, icmp_seq=1 hlim=64 time=0.424 ms
16 bytes from fe80::3f7:a5e:4649:ad46%bridge0, icmp_seq=2 hlim=64 time=0.424 ms
16 bytes from fe80::3f7:a5e:4649:ad46%bridge0, icmp_seq=3 hlim=64 time=0.434 ms
^C
--- ff02::2%bridge0 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.411/0.423/0.434/0.008 ms
[12:43 r730-03 dvl ~] %

They also say:

ff02::2 is the all-routers multicast address. ICMPv6 echo request packets sent by the ping6(8) utility to this address will be received by routers on the network, and you will re- ceive ICMPv6 echo reply packets. You can find the addresses by observing the replies.

For gw01, on vlan7, that value is fe80::92ca:faff:fecc:f982.

On both r730-01 and r730-03, that value is fe80::3f7:a5e:4649:ad46.

They also mentioned:

You need to be aware that a router receives no RA message. In IPv6 specification, IPv6 nodes are categorized into hosts and routers. A host is a leaf node of the network and does not forward IPv6 packets, and a router is a multi-homed node that forwards IPv6 packets across the networks. RA messages are defined as ones sent by a router and received by a host.

This means that we cannot configure a router by using the automatic configuration ca- pability explained in the previous section. You must not specify “inet6_accept_rtadv” on a router, and you need to configure the network parameter manually as an example shown above. If you specify ipv6_gateway_enable=”YES”, the FreeBSD kernel will ignore RA messages even if “inet6_accept_rtadv” is specified.

So… interesting. But more:

However, this model is too restrictive under some circumstances. For example, this host- and-router model does not work well for the IPv6 router provided by the ISP. This router must be automatically configured, but there is no way to configure the default router if it does not receive RA messages. On the other hand, if a router receives RA messages to con- figure itself, the configuration will be screwed up quickly because of messages from other routers. Another router will change the router’s default route.

To mitigate this problem, FreeBSD has adopted the following concepts:

  • The “host or router” is determined on each interface, not the system-wide property,
  • if the interface accepts RA messages, it is seen as “host” from other nodes.

Following this, the accept_rtadv flag can be configured on a per-interface basis. While
thepacketforwardingcapabilitycannotbeconfiguredsimilarly,asysctl net.inet6. ip6.rfc6204w3 is provided. When it is set to 1, the kernel receives RA messages even if the packet forwarding is enabled. While these knobs are difficult to understand, the details and concrete examples will be covered in later columns.

Reading further, I see references to the rtadvd(8) daemon – and I’m using net/radvd

So, let’s try that. I’ve stopped radvd and added this to my rc configuration (for me, that was /etc/rc.conf.d/radvd):

rtadvd_enable="YES"
rtadvd_interfaces="vlan2 vlan4 vlan3 vlan7 vlan219"

Then started it:

[13:05 gw01 dvl ~] % sudo service rtadvd onestart 
Starting rtadvd.
[13:05 gw01 dvl ~] % ps auwwx | grep dvd
root    16187   0.2  0.0  13908  2376  -  Ss   13:05       0:00.00 /usr/sbin/rtadvd
dvl     16189   0.0  0.0  13836  2380  8  S+   13:06       0:00.00 grep dvd
[13:06 gw01 dvl ~] %

Let’s try reboot

I did a reboot. My routes are much better:

after reboot, hosts inside can ping both the gateway and internet hosts.

We are running radvd, not convinced we need that.

routing table on gw01:

Routing tables

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::/96                             link#10                       URS             lo0
default                           2001:DB8:1f06:9ea::1          UGS            gif0
::1                               link#10                       UHS             lo0
::ffff:0.0.0.0/96                 link#10                       URS             lo0
2001:DB8:1f06:9ea::1              link#18                       UH             gif0
2001:DB8:1f06:9ea::2              link#10                       UHS             lo0
2001:db8:1:9e::                   link#10                       UHS             lo0
2001:db8:1:9e::/64                link#16                       U           vlan219
2001:db8:1:2052::                 link#10                       UHS             lo0
2001:db8:1:2052::/64              link#11                       U             vlan2
2001:db8:1:4054::                 link#10                       UHS             lo0
2001:db8:1:4054::/64              link#13                       U             vlan4
2001:db8:1:7055::                 link#10                       UHS             lo0
2001:db8:1:7055::/64              link#14                       U             vlan7
fe80::%lo0/10                     link#10                       URS             lo0
fe80::%igc3/64                    link#4                        U              igc3
fe80::227c:14ff:fef5:8e53%lo0     link#10                       UHS             lo0
fe80::%lo0/64                     link#10                       U               lo0
fe80::1%lo0                       link#10                       UHS             lo0
fe80::%vlan2/64                   link#11                       U             vlan2
fe80::227c:14ff:fef5:8e53%lo0     link#10                       UHS             lo0
fe80::%vlan4/64                   link#13                       U             vlan4
fe80::227c:14ff:fef5:8e53%lo0     link#10                       UHS             lo0
fe80::%vlan7/64                   link#14                       U             vlan7
fe80::227c:14ff:fef5:8e53%lo0     link#10                       UHS             lo0
fe80::%vlan219/64                 link#16                       U           vlan219
fe80::227c:14ff:fef5:8e53%lo0     link#10                       UHS             lo0
fe80::%gif0/64                    link#18                       U              gif0
fe80::227c:14ff:fef5:8e50%lo0     link#10                       UHS             lo0
ff02::/16                         link#10                       URS             lo0




[17:32 gw01 dvl ~] % ifconfig vlan3
vlan3: flags=1008843 metric 0 mtu 1500
	description: entertainment
	options=4600703
	ether 20:7c:14:f5:8e:53
	inet 10.53.0.1 netmask 0xffffff00 broadcast 10.53.0.255
	groups: vlan
	vlan: 3 vlanproto: 802.1q vlanpcp: 0 parent interface: igc3
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=29
[17:33 gw01 dvl ~] %

main host:

[17:28 r730-01 dvl ~] % netstat -finet6 -rn
Routing tables

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::/96                             link#7                        URS             lo0
default                           2001:db8:1:7055::             UGS         bridge0
default                           fe80::227c:14ff:fef5:8e53%bridge0 UG      bridge0
::1                               link#7                        UHS             lo0
::ffff:0.0.0.0/96                 link#7                        URS             lo0
2001:db8:1:7055::/64              link#11                       U           bridge0
2001:db8:1:7055:2c42:3ac2:b990:7a0d link#7                      UHS             lo0
2001:db8:1:7055:b6f9:d572:6622:ea2d link#7                      UHS             lo0
2001:db8:1:7055:c348::141         link#7                        UHS             lo0
2001:db8:1:7055:c348:9dc1:0:443   link#7                        UHS             lo0
2001:db8:1:7055:c348:c283:772a:6fed link#7                      UHS             lo0
fe80::%lo0/10                     link#7                        URS             lo0
fe80::%lo0/64                     link#7                        U               lo0
fe80::1%lo0                       link#7                        UHS             lo0
fe80::%lo1/64                     link#8                        U               lo1
fe80::1%lo0                       link#7                        UHS             lo0
fe80::%lo2/64                     link#9                        U               lo2
fe80::1%lo0                       link#7                        UHS             lo0
fe80::%lo3/64                     link#10                       U               lo3
fe80::1%lo0                       link#7                        UHS             lo0
fe80::%bridge0/64                 link#11                       U           bridge0
fe80::5a9c:fcff:fe10:8c57%lo0     link#7                        UHS             lo0
ff02::/16                         link#7                        URS             lo0

other host:

[17:33 r730-03 dvl ~] % netstat -finet6 -rn
Routing tables

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#6                        URS         lo0
default                           fe80::227c:14ff:fef5:8e53%bridge0 UG  bridge0
::1                               link#6                        UHS         lo0
::ffff:0.0.0.0/96                 link#6                        URS         lo0
2001:db8:1:7055::/64              link#7                        U       bridge0
2001:db8:1:7055:31da:7e45:0:53    link#6                        UHS         lo0
2001:db8:1:7055:6006::1           link#6                        UHS         lo0
fe80::%lo0/10                     link#6                        URS         lo0
fe80::%lo0/64                     link#6                        U           lo0
fe80::1%lo0                       link#6                        UHS         lo0
fe80::%bridge0/64                 link#7                        U       bridge0
fe80::21b:21ff:fe41:2ffa%lo0      link#6                        UHS         lo0
fe80::%lo1/64                     link#8                        U           lo1
fe80::1%lo0                       link#6                        UHS         lo0
ff02::/16                         link#6                        URS         lo0
Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Related Posts

Leave a Comment

Scroll to Top