Debugging aids for pf firewall rules [on FreeBSD]

It is not often that I need to debug pf firewall rules. Yet, when I do, I cannot remember the commands for what I want to do.

What is being blocked

First, I want to see the firewall rule numbers in the tcpdump output. I am logging all blocked packets, via pflog0. I can use that to see what is being blocked and by what rule.

These are found in my /etc/pf.conf:

logopt = "log"
logall = "log"

set block-policy return
set loginterface $ext_if # this is my WAN network interface

block log all

There are other entries in there, I’m just searching for log.

I also enable pflog via this magic:

[15:36 gw01 dvl ~] % cat /etc/rc.conf.d/pflog 
pflog_enable="YES"
[15:36 gw01 dvl ~] % ps auwx | grep pflog                                      
root     2269   0.0  0.0  14488  3092  -  Is   19:14      0:00.00 pflogd: [priv] (pflogd)
_pflogd  2271   0.0  0.0  14488  3204  -  S    19:14      0:15.36 pflogd: [running] -s 116 -i pflog0 -f /var/log/pflog (pflogd)
dvl     33244   0.0  0.0  13836  2388  1  S+   15:37      0:00.00 grep pflog

That log file is not your typical log file:

[15:38 gw01 dvl ~] % sudo file /var/log/pflog
/var/log/pflog: pcap capture file, microsecond ts (little-endian) - version 2.4 (OpenBSD PFLOG, capture length 116)

More on that in the next section.

What is being blocked?

Part one, what is being blocked and by what rule?

sudo tcpdump -n -e -ttt -r /var/log/pflog host 10.55.0.131
reading from file /var/log/pflog, link-type PFLOG (OpenBSD pflog file), snapshot length 116
 00:00:00.000000 rule 22/0(match): block in on vlan2: 10.52.0.39.56395 > 10.55.0.131.8080: Flags [S], seq 3656895980, win 14600, options [mss 1460, [|tcp]
 00:00:00.181786 rule 22/0(match): block in on vlan2: 10.52.0.39.56394 > 10.55.0.131.8080: Flags [S], seq 1415809346, win 14600, options [mss 1460, [|tcp]
 00:00:00.729830 rule 22/0(match): block in on vlan2: 10.52.0.39.56393 > 10.55.0.131.8080: Flags [S], seq 416312006, win 14600, options [mss 1460, [|tcp]
 00:00:00.079803 rule 22/0(match): block in on vlan2: 10.52.0.39.56395 > 10.55.0.131.8080: Flags [S], seq 3656895980, win 14600, options [mss 1460, [|tcp]
...
 00:00:00.559888 rule 22/0(match): block in on vlan2: 10.52.0.39.56635 > 10.55.0.131.8080: Flags [S], seq 3284565237, win 14600, options [mss 1460, [|tcp]
 00:00:00.705310 rule 22/0(match): block in on vlan2: 10.52.0.182.57233 > 10.55.0.131.8080: Flags [S], seq 2550298074, win 14600, options [mss 1460, [|tcp]
 00:00:00.734961 rule 22/0(match): block in on vlan2: 10.52.0.39.56637 > 10.55.0.131.8080: Flags [S], seq 3302656103, win 14600, options [mss 1460, [|tcp]

That’s all my wireless access points talking to the Unifi controller (10.55.0.131.8080) on vlan2. They are being blocked by rule 22.

What is rule 22?

This is the harder part. At first, I found this on a website but they got it wrong:

Filter information as above and prepend rule numbers:

pfctl -vvsr show

[14:27 gw01 dvl ~] % sudo pfctl -vvsr show 


pfctl: unknown command line argument: show ...
usage: pfctl [-AdeghMmNnOPqRrvz] [-a anchor] [-D macro=value] [-F modifier]
	[-f file] [-i interface] [-K host | network]
	[-k host | network | gateway | label | id] [-o level] [-p device]
	[-s modifier] [-t table -T command [address ...]] [-x level]

I tried to contact them, but found no contact information on the website.

I did another search, and found https://bsdwatch.net/articles/pfctl-cheatsheet which said:

pfctl -vvsr : Show filter information as above, and prepend rule numbers

[14:43 gw01 dvl ~] % sudo pfctl -vvsr
No ALTQ support in kernel
ALTQ related functions disabled
@0 scrub from any to  fragment no reassemble
  [ Evaluations: 770000    Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 26094 State Creations: 0     ]
  [ Last Active Time: N/A ]
@1 scrub from  to any fragment no reassemble
  [ Evaluations: 770000    Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 26094 State Creations: 0     ]
  [ Last Active Time: N/A ]
...
@22 block return log all
  [ Evaluations: 140118    Packets: 110317    Bytes: 8068228     States: 0     ]
  [ Inserted: uid 0 pid 26094 State Creations: 0     ]
....
@319 pass out on vlan219 inet from any to 233.252.22.59 flags S/SA keep state
  [ Evaluations: 4947      Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 26094 State Creations: 0     ]
  [ Last Active Time: N/A ]

Success!

But wait! There’s more!

Here, I can monitor, with rule numbers, in real time:

[16:58 gw01 dvl ~] % sudo tcpdump -n -e -ttt -i pflog0 inbound and action block and on vlan2 and port 8080 
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), snapshot length 262144 bytes
 00:00:00.000000 rule 22/0(match): block in on vlan2: 10.52.0.39.58012 > 10.55.0.131.8080: Flags [S], seq 2837144143, win 14600, options [mss 1460,sackOK,TS val 785956344 ecr 0,nop,wscale 4], length 0
 00:00:01.770043 rule 22/0(match): block in on vlan2: 10.52.0.39.58014 > 10.55.0.131.8080: Flags [S], seq 190564475, win 14600, options [mss 1460,sackOK,TS val 785956521 ecr 0,nop,wscale 4], length 0
...

That is taken from Filtering Log Output

Thank you for coming to my TED talk.