Trouble with vnet and pf

In the past, I have tried vnet jails with pf, and hit trouble. I was never able to get pf to allow the vnet traffic when having a default ‘block log all’ rule. More recently, I encountered the same problem when using byhve.

This time, I moved byhve to another host, which is not using pf and I am writing this post to document the issue.

This post is based on two gists I saved:

1. https://gist.github.com/dlangille/0f94e784ba1eb9523147e817eda0199f
2. https://gist.github.com/dlangille/1224e6a67087b019a1311c190e59c26c (which is more comprehensive)

In this post:

1. FreeBSD 12.2-RELEASE-p7

The skips

There are several skips in pf.conf, which I will not produce here in full because the file is 620 lines long.

[dan@slocum:~] $ grep skip /etc/pf.conf
set skip on lo0
set skip on epair
set skip on bridge0
set skip on tap1
set skip on vm-public
set skip on bridge
set skip on tap

Some of ifconfig

This is some of ifconfig, but not all IP addreses are shown:

[dan@slocum:~] $ ifconfig -u
ix2: flags=8943 metric 0 mtu 1500
	options=a538b9
	ether 00:1b:21:39:a9:c5
...
	inet 10.55.0.24 netmask 0xffffffff broadcast 10.55.0.24
...
	media: Ethernet autoselect (10Gbase-SR )
	status: active
	nd6 options=23
lo0: flags=8049 metric 0 mtu 16384
	options=680003
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21
pflog0: flags=141 metric 0 mtu 33160
	groups: pflog
vm-public: flags=8843 metric 0 mtu 1500
	ether 3e:18:c0:b5:b1:c5
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: tap1 flags=143
	        ifmaxaddr 0 port 9 priority 128 path cost 2000000
	member: ix2 flags=143
	        ifmaxaddr 0 port 3 priority 128 path cost 2000
	groups: bridge vm-switch viid-4c918@
	nd6 options=1
tap1: flags=8943 metric 0 mtu 1500
	description: vmnet-myguest-0-public
	options=80000
	ether 58:9c:fc:10:07:29
	groups: tap vm-port
	media: Ethernet autoselect
	status: active
	nd6 options=21
	Opened by PID 52659

The problem

My bhyve Virtual Machine (VM at 10.55.0.123) cannot converse with a webserver (in a jail at 10.55.0.24) on the host (10.55.0.73). The VM can contact other webservers (i.e. webservers which are not in jails on this host).

To be clear: both the VM and the webserver (which the VM cannot contact) are on the same host. That host is running pf.

To repear: the bhyve VM cannot reach my package server at 10.55.0.24 (which is actually an Nginx proxy to the real server). The problem is firewall rules.

The package server is in a jail. That is relevant to the firewall rules.

Demonstration

In the VM, other websites can be fetched:

dvl@testing:~ % fetch http://google.ca
fetch: http://google.ca: size of remote file is not known
google.ca                                               12 kB   48 MBps    00s

Reaching websites, NOT on this host, works.

Reaching a website on this host, does not, despite pings being OK:

dvl@testing:~ % ping fedex.int.unixathome.org
PING fedex.int.unixathome.org (10.55.0.24): 56 data bytes
64 bytes from 10.55.0.24: icmp_seq=0 ttl=64 time=0.222 ms
^C
--- fedex.int.unixathome.org ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.222/0.222/0.222/0.000 ms

dvl@testing:~ % fetch http://fedex.int.unixathome.org
fetch: http://fedex.int.unixathome.org: Operation timed out

tcpdump

While the above fetch is failing, you can see this traffic being blocked on pflog:

[dan@slocum:~] $ sudo tcpdump -n -e -ttt -i pflog0
00:00:01.058011 rule 0/0(match): block out on ix2: 10.55.0.24.80 > 10.55.0.123.13882: Flags [S.], seq 2041794959, ack 
198628267, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3016114281 ecr 3758404640], length 0

Where slocum is the host of both the bhyve VM and the webserver it cannot reach.

That means the website is getting the requests from the VM and is replying back, but that reply is blocked.

Also vital here, is the rule which is blocking that package. block out on ix2

Try as I might, I am unable to create a rule which allows that traffic to pass.

The output of sudo pfctl -sa includes:

pass out quick on ix2 inet proto tcp from 10.55.0.24 port = http to any flags S/SA keep state

There is something missing.