In the past, I have tried vnet jails with pf, and hit trouble. I was never able to get pf to allow the vnet traffic when having a default ‘block log all’ rule. More recently, I encountered the same problem when using byhve.
This time, I moved byhve to another host, which is not using pf and I am writing this post to document the issue.
This post is based on two gists I saved:
- https://gist.github.com/dlangille/1224e6a67087b019a1311c190e59c26c (which is more comprehensive)
In this post:
- FreeBSD 12.2-RELEASE-p7
There are several skips in pf.conf, which I will not produce here in full because the file is 620 lines long.
[dan@slocum:~] $ grep skip /etc/pf.conf set skip on lo0 set skip on epair set skip on bridge0 set skip on tap1 set skip on vm-public set skip on bridge set skip on tap
Some of ifconfig
This is some of ifconfig, but not all IP addreses are shown:
[dan@slocum:~] $ ifconfig -u ix2: flags=8943
metric 0 mtu 1500 options=a538b9 ether 00:1b:21:39:a9:c5 ... inet 10.55.0.24 netmask 0xffffffff broadcast 10.55.0.24 ... media: Ethernet autoselect (10Gbase-SR ) status: active nd6 options=23 lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 pflog0: flags=141 metric 0 mtu 33160 groups: pflog vm-public: flags=8843 metric 0 mtu 1500 ether 3e:18:c0:b5:b1:c5 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap1 flags=143 ifmaxaddr 0 port 9 priority 128 path cost 2000000 member: ix2 flags=143 ifmaxaddr 0 port 3 priority 128 path cost 2000 groups: bridge vm-switch viid-4c918@ nd6 options=1 tap1: flags=8943 metric 0 mtu 1500 description: vmnet-myguest-0-public options=80000 ether 58:9c:fc:10:07:29 groups: tap vm-port media: Ethernet autoselect status: active nd6 options=21 Opened by PID 52659
My bhyve Virtual Machine (VM at 10.55.0.123) cannot converse with a webserver (in a jail at 10.55.0.24) on the host (10.55.0.73). The VM can contact other webservers (i.e. webservers which are not in jails on this host).
To be clear: both the VM and the webserver (which the VM cannot contact) are on the same host. That host is running pf.
To repear: the bhyve VM cannot reach my package server at 10.55.0.24 (which is actually an Nginx proxy to the real server). The problem is firewall rules.
The package server is in a jail. That is relevant to the firewall rules.
In the VM, other websites can be fetched:
dvl@testing:~ % fetch http://google.ca fetch: http://google.ca: size of remote file is not known google.ca 12 kB 48 MBps 00s
Reaching websites, NOT on this host, works.
Reaching a website on this host, does not, despite pings being OK:
dvl@testing:~ % ping fedex.int.unixathome.org PING fedex.int.unixathome.org (10.55.0.24): 56 data bytes 64 bytes from 10.55.0.24: icmp_seq=0 ttl=64 time=0.222 ms ^C --- fedex.int.unixathome.org ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.222/0.222/0.222/0.000 ms dvl@testing:~ % fetch http://fedex.int.unixathome.org fetch: http://fedex.int.unixathome.org: Operation timed out
While the above fetch is failing, you can see this traffic being blocked on pflog:
[dan@slocum:~] $ sudo tcpdump -n -e -ttt -i pflog0 00:00:01.058011 rule 0/0(match): block out on ix2: 10.55.0.24.80 > 10.55.0.123.13882: Flags [S.], seq 2041794959, ack 198628267, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3016114281 ecr 3758404640], length 0
Where slocum is the host of both the bhyve VM and the webserver it cannot reach.
That means the website is getting the requests from the VM and is replying back, but that reply is blocked.
Also vital here, is the rule which is blocking that package. block out on ix2
Try as I might, I am unable to create a rule which allows that traffic to pass.
The output of sudo pfctl -sa includes:
pass out quick on ix2 inet proto tcp from 10.55.0.24 port = http to any flags S/SA keep state
There is something missing.