kernel: Limiting closed port RST response from x to y packets/sec

For 3 days now, I’ve been seeing these messages. If you search online, it’s usually the result of port scanning.

Aug  7 14:05:15 zuul kernel: Limiting closed port RST response from 212 to 195 packets/sec
Aug  7 14:05:16 zuul kernel: Limiting closed port RST response from 219 to 215 packets/sec
Aug  7 14:05:17 zuul kernel: Limiting closed port RST response from 220 to 193 packets/sec
Aug  7 14:05:18 zuul kernel: Limiting closed port RST response from 218 to 193 packets/sec
Aug  7 14:05:19 zuul kernel: Limiting closed port RST response from 235 to 207 packets/sec
Aug  7 14:05:19 zuul kernel: Limiting closed port RST response from 257 to 197 packets/sec
Aug  7 14:05:21 zuul kernel: Limiting closed port RST response from 215 to 198 packets/sec

Failed attempt to block

Yesterday, I decided to start monitoring who was trying various ports. I ran sudo tcpdump -ni pflog0 to see what was being logged. I had a block log all in directive in my /etc/pf.conf file.

I got some numbers, grep’d out the IP addresses, and banned about 443 addresses.

It made no difference. I gave up and went for a walk on the beach.

To be clear: the blocking worked. However, it did not solve the issue.

Today, I decided to pay attention to monitoring.

Monitoring

In hindsight, there was a lack of the usual emails coming in, from logcheck, Nagios, etc.

My Nagios web page showed me this for the host in question:

• PROCS CRITICAL: 7554 processes
• SWAP WARNING – 69% free (5609 MB out of 8191 MB) [5609 (69%)]

While looking at the list of processes, I saw a huge amount of sendmail. How much sendmail you ask? This much sendmail:

[14:49 zuul dan ~] % ps auwwx | grep -c  sendmail
7852

That’s a log of sendmail for this host.

As I type this, it’s up to 8092 processes: 1 running, 8091 sleeping

I checked one jail:

[14:50 www dvl ~] % mailq | wc -l
   14891

That jail should not have 15,000 queued emails.

I restarted that jail, just to kill all the sendmail processes.

I noticed the host is not overly loaded:

[14:51 zuul dan ~] % w
 2:51PM  up 35 days,  3:12, 4 users, load averages: 1.83, 1.68, 1.18

After restarting that one jail, have only 4369 processes:2 running, 4367 sleeping.

From the host, I saw lost of these under ps auwwx:

logcheck 99300  0.0  0.0   18380   2152  -  SNsJ Sat18       0:00.41 sendmail -oi dan@example.org (dma)
www      99344  0.0  0.0   18380   1264  -  SsJ  01:30       0:00.05 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t (dma)

I suspect it’s dma. A few days ago, my blog past was Replacing postfix with dma + auth.

This is from within the jail I just restarted:

[14:53 www dvl ~] % ps auwwx | grep send
www      4893  0.0  0.0 18388  6192  -  SsJ  14:52   0:00.00 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t (dma)
logcheck 5369  0.0  0.0 18388  6316  -  SNsJ 14:52   0:00.00 sendmail -oi dan@example.org (dma)
dvl      5418  0.0  0.0 12808  2020  3  S+J  14:53   0:00.00 grep send

One fix attempt, within the jail

Let’s try this fix:

[14:53 www dvl ~] % grep sendmail /etc/rc.conf
# Prevent sendmail to try to connect to localhost
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
[14:53 www dvl ~] % sudo sysrc sendmail_enable="NONE"
sendmail_enable: NO -> NONE
[14:54 www dvl ~] % 

None, is better than no, in this case. We’re using just dma.

After restarting the jail, still see:

[14:55 www dvl ~] % ps auwwx | grep send      
logcheck 8117  1.4  0.0 18388  6176  -  SNsJ 14:55   0:00.00 sendmail -oi dan@example.org (dma)
dvl      8139  0.0  0.0 12808  2016  3  S+J  14:55   0:00.00 grep send
[14:55 www dvl ~] % date
Wed Aug  7 14:55:44 UTC 2024

[14:56 www dvl ~] % cat /etc/mail/mailer.conf
# mailer.conf for use with dma(8)
# If sendmail is configured, an example of mailer.conf that uses sendmail
# instead can be found in /usr/share/examples/sendmail.

sendmail	/usr/libexec/dma
mailq		/usr/libexec/dma
newaliases	/usr/libexec/dma
[14:56 www dvl ~] % 

Looks right.

[14:56 www dvl ~] % mailq | wc -l
   14911

Wow, that’s a lot of queued mail. I don’t want that delivered. Here, I delete that:

[14:57 www dvl ~] % sudo su -l
root@www:~ # cd /var/spool/dma
root@www:/var/spool/dma # ls | head
M1061.35f5aa848000
M1064.2808fa048000
M1067.3d6f8e848000
M106a.547b3c048000
M106d.3f7acd448000
M1070.13a1aa648000
M1073.484bc0a48000
M1077.3dbcf6648000
M1080.4b45a1a48000
M1084.4467f6048000
root@www:/var/spool/dma # rm M*
root@www:/var/spool/dma # rm Q*
root@www:/var/spool/dma # ls
flush
root@www:/var/spool/dma # 

Looking at the jail’s logs:

[14:58 www dvl ~] % sudo tail -F /var/log/maillog
Aug  7 14:56:00 www dma[528e8.590eb4c48000][8228]:  trying delivery
Aug  7 14:56:00 www dma[528e8.590eb4c48000][8228]: using smarthost (zuul.example.org:25)
Aug  7 14:56:00 www dma[528e8.590eb4c48000][8228]: trying remote delivery to zuul.example.org [203.0.113.66] pref 0
Aug  7 14:56:00 www dma[528e8.590eb4c48000][8228]: connect to zuul.example.org [203.0.113.66] failed: Connection refused
Aug  7 14:58:01 www dma[5297d][8538]: new mail from user=www uid=80 envelope_from=
Aug  7 14:58:01 www dma[5297d][8538]: mail to= queued as 5297d.3b264dc48000
Aug  7 14:58:01 www dma[5297d.3b264dc48000][8540]:  trying delivery
Aug  7 14:58:01 www dma[5297d.3b264dc48000][8540]: using smarthost (zuul.example.org:25)
Aug  7 14:58:01 www dma[5297d.3b264dc48000][8540]: trying remote delivery to zuul.example.org [203.0.113.66] pref 0
Aug  7 14:58:01 www dma[5297d.3b264dc48000][8540]: connect to zuul.example.org [203.0.113.66] failed: Connection refused

Fixing the host

Hmm, let’s swap back to postfix on the jail host.

Those steps are not shown here. However, in short it was:

1. sudo service postfix enable
2. correct the contents of /etc/mail/mailer.conf to use postfix, not dma
3. sudo service postfix start

Here is /var/log/messages afterwards:

Aug  7 15:00:16 zuul kernel: Limiting closed port RST response from 229 to 187 packets/sec
Aug  7 15:00:20 zuul kernel: Limiting closed port RST response from 194 to 215 packets/sec
Aug  7 15:00:35 zuul kernel: Limiting closed port RST response from 216 to 201 packets/sec
Aug  7 15:05:09 zuul kernel: sonewconn: pcb 0xfffff803be721000 (203.0.113.66:25 (proto 6)): Listen queue overflow: 151 already in queue awaiting acceptance (1 occurrences), euid 0, rgid 0, jail 0
Aug  7 15:00:35 zuul kernel: Limiting closed port RST response from 216 to 201 packets/sec
Aug  7 15:05:09 zuul kernel: sonewconn: pcb 0xfffff803be721000 (203.0.113.66:25 (proto 6)): Listen queue overflow: 151 already in queue awaiting acceptance (1 occurrences), euid 0, rgid 0, jail 0

Conclusion

The phone calls are coming from inside the house.

I was doing this to myself.

Every jail (all 15 of them) is configured to relay to the jail host. There was nothing listening on port 25 there… Starting postfix on the expected post allowed the mail to flow, the sendmail processes to terminate, and the closed port log entries the stop.

The port in question is port 25. Those attempted port connections are not logged. Only the blocked connections show up in the pflog0 logs.

Now I wait while 8100 queues emails make their way trough the system. Many will be deleted and remain unread.

Yeah, I should have known dma on the hosts was wrong.

The next day

It’s Thursday 8 August, 2024. Tropical storm Debby is about 116 miles away. It rained all night. There was a tornado watch at 8 PM last night. 4 adults, two toddlers, and three dogs scrunched into the laundry room for 30 minutes. More rain coming.

Today, most of the email has filtered. This affected multiple hosts, since the same work was done on each. Today, zuul has about 304 emails in the queue. Checking the mail logs on that host:

5DF40C0F6      1380 Wed Aug  7 15:17:52  MAILER-DAEMON
(host smtp.fastmail.com[103.168.172.60] said: 552 5.7.1 : Data command rejected: Already reached per-day limit for messages sent from "dvl@example.com" of 8000, try again later (in reply to DATA command))
                                         dan@example.com

I’ll just have to be patient to receive and delete the rest of that email later.

Additionally: this affects me sending my personal emails – limited by that 8000. Oops.