Changing how I use IP address with FreeBSD’s vnet – so IPv6 works

The vnet (network subsystem virtualization infrastructure) on FreeBSD is both a blessing and a curse. For me, it isn’t working well with IPv6. I’ve hit issues with pf (since solved, I’m sure). It’s a useful feature but can be confusing. It helps to have lots of time to think about it and what you’re doing.

I talked about this amongst the FreeBSD jail group and out of that came a blog post: The correct way to configure bridges in FreeBSD for IPv6 (and IPv4) by crest.

In this post:

FreeBSD 13.2-RELEASE-p2
Unless otherwise stated, all configuration show is from /etc/rc.conf

igb0 was my primary NIC. It looked like this before I started.

[18:41 r730-01 dvl /etc] % ifconfig                            
igb0: flags=8963 metric 0 mtu 1500
	options=4a500b9
	ether ec:f4:bb:ef:c9:54
	inet 10.55.0.141 netmask 0xffffff00 broadcast 10.55.0.255
	inet 10.55.0.73 netmask 0xffffffff broadcast 10.55.0.73
	inet 10.55.0.44 netmask 0xffffffff broadcast 10.55.0.44
	inet 10.55.0.151 netmask 0xffffffff broadcast 10.55.0.151
	inet 10.55.0.150 netmask 0xffffffff broadcast 10.55.0.150
	inet 10.55.0.32 netmask 0xffffffff broadcast 10.55.0.32
	inet 10.55.0.34 netmask 0xffffffff broadcast 10.55.0.34
	inet 10.55.0.4 netmask 0xffffffff broadcast 10.55.0.4
	inet 10.55.0.54 netmask 0xffffffff broadcast 10.55.0.54
	inet 10.55.0.112 netmask 0xffffffff broadcast 10.55.0.112
	inet 10.55.0.6 netmask 0xffffffff broadcast 10.55.0.6
	inet 10.55.0.10 netmask 0xffffffff broadcast 10.55.0.10
	inet 10.55.0.33 netmask 0xffffffff broadcast 10.55.0.33
	inet 10.55.0.17 netmask 0xffffffff broadcast 10.55.0.17
	inet 10.55.0.20 netmask 0xffffffff broadcast 10.55.0.20
	inet 10.55.0.49 netmask 0xffffffff broadcast 10.55.0.49
	inet 10.55.0.16 netmask 0xffffffff broadcast 10.55.0.16
	inet 10.55.0.31 netmask 0xffffffff broadcast 10.55.0.31
	inet 10.55.0.24 netmask 0xffffffff broadcast 10.55.0.24
	inet 10.55.0.8 netmask 0xffffffff broadcast 10.55.0.8
	inet 10.55.0.53 netmask 0xffffffff broadcast 10.55.0.53
	inet 10.55.0.131 netmask 0xffffffff broadcast 10.55.0.131
	inet 10.55.0.28 netmask 0xffffffff broadcast 10.55.0.28
	inet 10.55.0.37 netmask 0xffffffff broadcast 10.55.0.37
	inet 10.55.0.40 netmask 0xffffffff broadcast 10.55.0.40
	inet 10.55.0.45 netmask 0xffffffff broadcast 10.55.0.45
	inet 10.55.0.35 netmask 0xffffffff broadcast 10.55.0.35
	inet 10.55.0.22 netmask 0xffffffff broadcast 10.55.0.22
	inet 10.55.0.27 netmask 0xffffffff broadcast 10.55.0.27
	inet 10.55.0.42 netmask 0xffffffff broadcast 10.55.0.42
	inet 10.55.0.46 netmask 0xffffffff broadcast 10.55.0.46
	inet 10.55.0.50 netmask 0xffffffff broadcast 10.55.0.50
	inet 10.55.0.30 netmask 0xffffffff broadcast 10.55.0.30
	inet 10.55.0.39 netmask 0xffffffff broadcast 10.55.0.39
	inet 10.55.0.5 netmask 0xffffffff broadcast 10.55.0.5
	inet 10.55.0.3 netmask 0xffffffff broadcast 10.55.0.3
	inet6 fe80::eef4:bbff:feef:c954%igb0 prefixlen 64 scopeid 0x1
	inet6 2001:0DB8:8abf:7055:c348::141 prefixlen 64
	inet6 2001:0DB8:8abf:7055:c348:9dc1:0:443 prefixlen 128
	inet6 2001:0DB8:8abf:7055:2c42:3ac2:b990:7a0d prefixlen 128
	inet6 2001:0DB8:8abf:7055:c348:c283:772a:6fed prefixlen 128
	inet6 2001:0DB8:8abf:7055:b6f9:d572:6622:ea2d prefixlen 128
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=23

This is my bridge used for my vnet jail, pkg01. Also shown, is the epair device. tap0 is used with my bhyve installation.

igb0bridge: flags=8843 metric 0 mtu 1500
	ether 58:9c:fc:10:8c:57
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: tap0 flags=143
	        ifmaxaddr 0 port 14 priority 128 path cost 2000000
	member: e0a_pkg01 flags=143
	        ifmaxaddr 0 port 16 priority 128 path cost 2000
	member: igb0 flags=143
	        ifmaxaddr 0 port 1 priority 128 path cost 20000
	groups: bridge vm-switch viid-4c918@
	nd6 options=9
e0a_pkg01: flags=8963 metric 0 mtu 1500
	options=8
	ether 02:20:77:ef:c9:54
	hwaddr 02:13:a1:e3:20:0a
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T )
	status: active
	nd6 options=29
tap0: flags=8943 metric 0 mtu 1500
	description: vmnet/hass/0/public
	options=80000
	ether 58:9c:fc:00:77:5a
	groups: tap vm-port
	media: Ethernet autoselect
	status: active
	nd6 options=29
	Opened by PID 75426

Duplicate the MAC address

I will be duplicating the MAC address of my primary NIC (the one with the network connection) onto my bridge.

Why?

I got from talking with crest:

You want a *globally* unique mac address and your physical NIC has one you can take while it’s a bridge port
Otherwise the bridge will generate a randomized mac address with only 24 bits of entropy
Instead of making one up, we use one which we know is unique
You may be using DHCP with your primary NIC: any leases based on the NIC will now work with the bridge<

NOTE: Instead of doing this, add entries to /boot/loader.conf (see below).

This was from the blog post, but I found it did not work for me.

Run this command:

[18:43 r730-01 dvl /etc] % sudo sysctl net.link.bridge.inherit_mac=1

Add the following to /etc/sysctl.conf:

net.link.bridge.inherit_mac=1

What worked for me instead was:

Could not get the same MAC so I did this instead:

[0:18 r730-01 dvl /mnt] % tail -2 /boot/loader.conf
# to get the same MAC on igb0 and bridge0
if_bridge_load=YES

And with this in /etc/sysctl.conf:

net.link.bridge.inherit_mac=1

With that change I got:

[0:29 r730-01 dvl ~] % ifconfig igb0        
igb0: flags=8963 metric 0 mtu 1500
	options=4a100b9
	ether ec:f4:bb:ef:c9:54
	media: Ethernet autoselect (1000baseT )
	status: active
	nd6 options=29
[0:29 r730-01 dvl ~] % ifconfig bridge0
bridge0: flags=8843 metric 0 mtu 1500
	ether ec:f4:bb:ef:c9:54

Modify ifconfig from NIC to bridge

In my /etc/rc.conf file, I changed igb0 references to bridge0:

ifconfig_bridge0="inet 10.55.0.141 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 2001:470:8abf:7055:c348::141 prefixlen 64 accept_rtadv" # self

However, we need to make sure my primary NIC is up. So:

ifconfig_igb0="up -tso -vlanhwtso"

Why? From the how-to guide I am following: “A further complication is that the bridge has to have unmodified access to the Ethernet frames, but most 1Gb/s and faster as well as virtual network interfaces have offloading features like TSO and LRO to rewrite the small (by modern standards) 1500 byte Ethernet frames into “fake” larger frames to reduce the CPU overhead of processing the packet inside each frame. While useful to IP hosts these offloading features have to be disabled to bridge Ethernet or route and filter the IP packets inside.”

We also want to enable a link-local adddress on the bridge, and add it’s first member interface (my primary NIC, igb0).

I specify DHCP just because, and in case that’s what you’re doing. You’ll see where I enable rtsold and configure it to use the bridge interface

create_args_bridge0="inet6 auto_linklocal -ifdisabled addm igb0"
ifconfig_bridge0_ipv6="inet6 accept_rtadv"

rtsold_enable="YES"
rtsold_flags="-i -m bridge0"

Summary

This section shows the resulting lines in /etc/rc.conf:

cloned_interfaces="bridge0"
create_args_bridge0="inet6 auto_linklocal -ifdisabled addm igb0"
ifconfig_bridge0="inet 10.55.0.143 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 2001:470:8abf:7055:6006::1 prefixlen 64 accept_rtadv" # self

# and we need igb0 up still:
ifconfig_igb0="up -tso -vlanhwtso"

# enable rtsold and configure it to use the bridge interface
rtsold_enable="YES"
rtsold_flags="-i -m bridge0"

The reboot

Really, make sure you have console access before you do this reboot.

After reboot:

[19:13 r730-01 dvl ~] % ifconfig | less
igb0: flags=8963 metric 0 mtu 1500
        options=4e503bb
        ether ec:f4:bb:ef:c9:54
        media: Ethernet autoselect (1000baseT )
        status: active

        nd6 options=29

igb1..igb7, ix0, and ix1 have been snipped from the past

lo0: flags=8049 metric 0 mtu 16384
        options=680003
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21
lo1: flags=8049 metric 0 mtu 16384
        options=680003
        inet 127.1.0.201 netmask 0xffffffff
        inet6 fe80::1%lo1 prefixlen 64 scopeid 0xc
        groups: lo
        nd6 options=21
bridge0: flags=8843 metric 0 mtu 1500
        ether 58:9c:fc:10:8c:57
        inet 10.55.0.141 netmask 0xffffff00 broadcast 10.55.0.255
        inet6 fe80::5a9c:fcff:fe10:8c57%bridge0 prefixlen 64 scopeid 0xd
        inet6 2001:470:8abf:7055:c348::141 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb0 flags=143
                ifmaxaddr 0 port 1 priority 128 path cost 2000000
        groups: bridge
        nd6 options=23
pflog0: flags=141 metric 0 mtu 33160
        groups: pflog
tap0: flags=8843 metric 0 mtu 1500
        description: vmnet/hass/0/public
        options=80000
        ether 58:9c:fc:00:77:5a
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29
        Opened by PID 5364

You’ll see the bridge and primary NIC do not have the same MAC. That’s because I hadn’t made the /boot/loader.conf entries yet.

Started one jail, was able to ssh in.

Started all jails:

[19:20 r730-01 dvl /etc/jail.conf.d] % sudo service jail start         
Starting jails: dns1 cliff2 mysql01 pg01 pg02 pg03 cannot start jail  "pkg01": 
ifconfig: BRDGADD bridge0: Invalid argument
jail: pkg01: /usr/local/sbin/jib addm pkg01 bridge0: failed
 dev-ingress01 dev-nginx01 test-ingress01 test-nginx01 stage-ingress01 stage-nginx01 bacula besser certs-rsync certs git svn webserver cannot start jail  "zm": 
jail: "zm" not found
 dev-pgeu mqtt01 bacula-sd-02 jail-testing talos samdrucker nsnotify bacula-sd-03 fileserver cannot start jail  "mydev": 
jail: "mydev" already exists
 jail_within_jail serpico dns-hidden-master unifi01.

That zm jail was removed some time ago, but the jail_list entry in /etc/rc.conf still referenced it. I fixed that.

Things I tried:

In /etc/jail.conf.d/pkg01.conf, I commented out:

#  exec.poststop   += "/usr/local/sbin/jib destroy $name";

#  exec.prestart   += "/usr/local/sbin/jib addm  $name bridge0";

Got:

[19:22 r730-01 dvl /etc/jail.conf.d] % sudo service jail start pkg01
Starting jails: cannot start jail  "pkg01": 
37
ifconfig: interface e0b_pkg01 does not exist
jail: pkg01: /sbin/ifconfig e0b_pkg01 vnet pkg01: failed
.
[19:22 r730-01 dvl /etc/jail.conf.d] %

I’m sure that’s from this line in /etc/jail.conf.d/pkg01.conf:

  vnet.interface   = "e0b_$name";

Tried this instead: https://codeberg.org/pkgbase/website/src/branch/main/howto/jails.md

NOTE: this is not the whole definition for this jail, it’s just the bits pertaining to the networking.

  vnet;

  # from https://codeberg.org/pkgbase/website/src/branch/main/howto/jails.md
  $gw     = "10.55.0.1";
  $gw6    = "2001:470:8abf:3053:0:0:0:0";
  $bridge = "bridge0";
  $id      = "29";
  $jepair  = "epair${id}b";
  $ipaddr  = "10.55.0.${id}/24";
  $ip6addr = "2001:470:8abf:7055:c348:9dc1:0:${id}/64";

  vnet.interface = "$jepair";
  exec.prestart   = "ifconfig epair${id} create up";
  exec.prestart  += "ifconfig epair${id}a up descr vnet-${name}";
  exec.prestart  += "ifconfig $bridge addm epair${id}a up";

  # this ifconfig and routing could be done in the jail in /etc/rc.conf 
  exec.start      = "/sbin/ifconfig lo0 127.0.0.1 up";
  exec.start     += "/sbin/ifconfig epair${id}b ${ipaddr}";
  exec.start     += "/sbin/ifconfig epair${id}b inet6 ${ip6addr}";
  exec.start     += "/sbin/route add default ${gw}";
  exec.start     += "/bin/sh /etc/rc";

  exec.prestop    = "ifconfig epair${id}b -vnet ${name}";

  exec.poststop   = "ifconfig $bridge deletem epair${id}a";
  exec.poststop  += "ifconfig epair${id}a destroy";

In the jail’s /etc/rc.conf, I commented out all ifconfig and defaultrouter lines, because those directives are now here in the configuration file for the jail.

First try got:

[19:51 r730-01 dvl /etc/jail.conf.d] % sudo service jail start pkg01 
Starting jails: cannot start jail  "pkg01": 
epair29a
40
jail: pkg01: /sbin/route add -inet6 default 2001:470:8abf:3053:0:0:0:0: failed
.

I commented out the route add -inet6 line (not shown above) and tried again:

[19:53 r730-01 dvl /etc/jail.conf.d] % sudo service jail start pkg01
Starting jails: cannot start jail  "pkg01": 
ifconfig: interface epair29 already exists
jail: pkg01: ifconfig epair29 create up: failed
.

Yes, I must manually delete it because the start failed. This is usually destroyed when the jail stops (see the exec.poststop commands above).

[19:53 r730-01 dvl /etc/jail.conf.d] % sudo ifconfig epair29a destroy
[19:53 r730-01 dvl /etc/jail.conf.d] % sudo service jail start pkg01 
Starting jails: pkg01.

Success. I was able to ssh into the jail and found:

[16:04 air01 dan ~] % pkg01   
Last login: Sun Aug 13 19:54:50 2023 from air01.startpoint.vpn.unixathome.org
[20:04 pkg01 dan ~] % ifconfig
lo0: flags=8049 metric 0 mtu 16384
	options=680003
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21
pflog0: flags=0<> metric 0 mtu 33160
	groups: pflog
epair29b: flags=8863 metric 0 mtu 1500
	options=8
	ether 02:13:a1:e3:20:0b
	inet 10.55.0.29 netmask 0xffffff00 broadcast 10.55.0.255
	inet6 2001:470:8abf:7055:c348:9dc1:0:29 prefixlen 64
	inet6 fe80::13:a1ff:fee3:200b%epair29b prefixlen 64 scopeid 0x3
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T )
	status: active
	nd6 options=21
[20:04 pkg01 dan ~] %

Success.