This is a long post. There’s a lot of stuff in here. There’s no quick and dirty how-to. It’s a diagnostic record. Hope it helps.
This morning I saw log entries I’ve never noticed before. They seem to have started 9 hours ago. First, this email arrived.
cliff2 is one one two hosts behind cliff:
[7:25 pro05 dvl ~] % host cliff cliff.int.unixathome.org has address 10.55.0.44 cliff.int.unixathome.org has address 10.55.0.14
In this post:
- FreeBSD 15.0
- Jan 18 18:34:48 cliff2 pkg[70008]: postfix upgraded: 3.10.3,1 -> 3.10.6,1
- the host is r730-01 (that post was created before this host was upodated to FreeBSD 15.0)
Of the 16 emails received, they were all about cliff2, never cliff1.
Date: Wed, 18 Feb 2026 12:22:30 +0000 (UTC) From: Mail Delivery System <MAILER-DAEMON@cliff2.int.unixathome.org> To: Postmaster <postmaster@cliff2.int.unixathome.org> Subject: Postfix SMTP server: errors from webserver.int.unixathome.org[10.55.0.3] Message-Id: <20260218122230.767072C399@cliff2.int.unixathome.org> Transcript of session follows. Out: 220 cliff2.int.unixathome.org ESMTP Postfix In: EHLO webserver.int.unixathome.org Out: 250-cliff2.int.unixathome.org Out: 250-PIPELINING Out: 250-SIZE 10485760000 Out: 250-ETRN Out: 250-STARTTLS Out: 250-ENHANCEDSTATUSCODES Out: 250-8BITMIME Out: 250-DSN Out: 250-SMTPUTF8 Out: 250 CHUNKING In: STARTTLS Out: 220 2.0.0 Ready to start TLS In: EHLO webserver.int.unixathome.org Out: 250-cliff2.int.unixathome.org Out: 250-PIPELINING Out: 250-SIZE 10485760000 Out: 250-ETRN Out: 250-ENHANCEDSTATUSCODES Out: 250-8BITMIME Out: 250-DSN Out: 250-SMTPUTF8 Out: 250 CHUNKING In: MAIL FROM:<nagios@webserver.int.unixathome.org> Out: 452 4.3.1 Insufficient system storage Session aborted, reason: lost connection For other details, see the local mail logfile
There were also a few email from MAIL FROM:
Log entries
The log entries for the sending host are:
Feb 18 12:22:30 webserver dma[15199]: new mail from user=nagios uid=181 envelope_from=<nagios@webserver.int.unixathome.org> Feb 18 12:22:30 webserver dma[15199]: mail to=<dan@langille.org> queued as 15199.3f8617a420a0 Feb 18 12:22:30 webserver dma[15199.3f8617a420a0][98813]: <dan@langille.org> trying delivery Feb 18 12:22:30 webserver dma[15199.3f8617a420a0][98813]: using smarthost (cliff.int.unixathome.org:25) Feb 18 12:22:30 webserver dma[15199.3f8617a420a0][98813]: trying remote delivery to cliff.int.unixathome.org [10.55.0.44] pref 0 Feb 18 12:22:30 webserver dma[15199.3f8617a420a0][98813]: remote delivery deferred: cliff.int.unixathome.org [10.55.0.44] failed after MAIL FROM: 452 4.3.1 Insufficient system storage Feb 18 12:22:30 webserver dma[15199.3f8617a420a0][98813]: trying remote delivery to cliff.int.unixathome.org [10.55.0.14] pref 0 Feb 18 12:22:30 webserver dma[15199.3f8617a420a0][98813]: <dan@langille.org> delivery successful
On the receiving host (cliff2), I found:
Feb 18 12:22:02 cliff2 postfix/smtpd[95569]: disconnect from webserver.int.unixathome.org[10.55.0.3] helo=1 quit=1 commands=2 Feb 18 12:22:30 cliff2 postfix/smtpd[95569]: connect from webserver.int.unixathome.org[10.55.0.3] Feb 18 12:22:30 cliff2 postfix/smtpd[95569]: Anonymous TLS connection established from webserver.int.unixathome.org[10.55.0.3]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 Feb 18 12:22:30 cliff2 postfix/smtpd[95569]: NOQUEUE: reject: MAIL from webserver.int.unixathome.org[10.55.0.3]: 452 4.3.1 Insufficient system storage; proto=ESMTP helo=<webserver.int.unixathome.org> Feb 18 12:22:30 cliff2 postfix/smtpd[95569]: warning: not enough free space in mail queue: 15472168960 bytes < 1.5*message size limit Feb 18 12:22:30 cliff2 postfix/smtpd[95569]: NOQUEUE: lost connection after MAIL from webserver.int.unixathome.org[10.55.0.3] Feb 18 12:22:30 cliff2 postfix/cleanup[98873]: 767072C399: message-id=<20260218122230.767072C399@cliff2.int.unixathome.org> Feb 18 12:22:30 cliff2 postfix/smtpd[95569]: disconnect from webserver.int.unixathome.org[10.55.0.3] ehlo=2 starttls=1 mail=0/1 commands=3/4 Feb 18 12:22:30 cliff2 postfix/qmgr[7445]: 767072C399: from=<double-bounce@cliff2.int.unixathome.org>, size=1277, nrcpt=1 (queue active) Feb 18 12:22:30 cliff2 postfix/cleanup[98873]: 79B732C911: message-id=<20260218122230.767072C399@cliff2.int.unixathome.org> Feb 18 12:22:30 cliff2 postfix/local[98877]: 767072C399: to=<postmaster@cliff2.int.unixathome.org>, orig_to=<postmaster>, relay=local, delay=0.02, delays=0.01/0.01/0/0, dsn=2.0.0, status=sent (forwarded as 79B732C911) Feb 18 12:22:30 cliff2 postfix/qmgr[7445]: 79B732C911: from=<double-bounce@cliff2.int.unixathome.org>, size=1434, nrcpt=1 (queue active) Feb 18 12:22:30 cliff2 postfix/qmgr[7445]: 767072C399: removed Feb 18 12:22:30 cliff2 postfix/smtp[98880]: Untrusted TLS connection established to smtp.fastmail.com[103.168.172.60]:587: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 Feb 18 12:22:30 cliff2 postfix/smtp[98880]: 79B732C911: to=<dan@langille.org>, orig_to=<postmaster>, relay=smtp.fastmail.com[103.168.172.60]:587, delay=0.36, delays=0/0.01/0.11/0.23, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as CF51A800C2 ti_phl-compute-08_3001043_1771417350_11 via phl-compute-08) Feb 18 12:22:30 cliff2 postfix/qmgr[7445]: 79B732C911: removed
The mail queues on both hosts are empty.
Is it really a space issue?
No, it’s a mail configuration issue. See line 5, highlighted above.
That message repeats:
[12:35 cliff2 dvl ~] % sudo grep 'warning: not enough free space in mail queue' /var/log/maillog Feb 18 04:46:54 cliff2 postfix/smtpd[11796]: warning: not enough free space in mail queue: 15723220992 bytes < 1.5*message size limit Feb 18 04:47:35 cliff2 postfix/smtpd[11796]: warning: not enough free space in mail queue: 15720923136 bytes < 1.5*message size limit Feb 18 05:47:10 cliff2 postfix/smtpd[84061]: warning: not enough free space in mail queue: 15652507648 bytes < 1.5*message size limit Feb 18 06:31:40 cliff2 postfix/smtpd[3824]: warning: not enough free space in mail queue: 15704023040 bytes < 1.5*message size limit Feb 18 06:31:50 cliff2 postfix/smtpd[3824]: warning: not enough free space in mail queue: 15688192000 bytes < 1.5*message size limit Feb 18 06:32:20 cliff2 postfix/smtpd[3824]: warning: not enough free space in mail queue: 15684222976 bytes < 1.5*message size limit Feb 18 07:35:50 cliff2 postfix/smtpd[90728]: warning: not enough free space in mail queue: 15561146368 bytes < 1.5*message size limit Feb 18 07:40:51 cliff2 postfix/smtpd[10376]: warning: not enough free space in mail queue: 15549181952 bytes < 1.5*message size limit Feb 18 08:27:51 cliff2 postfix/smtpd[32105]: warning: not enough free space in mail queue: 15599861760 bytes < 1.5*message size limit Feb 18 08:27:51 cliff2 postfix/smtpd[38446]: warning: not enough free space in mail queue: 15599861760 bytes < 1.5*message size limit Feb 18 08:31:10 cliff2 postfix/smtpd[49465]: warning: not enough free space in mail queue: 15538733056 bytes < 1.5*message size limit Feb 18 08:31:41 cliff2 postfix/smtpd[49465]: warning: not enough free space in mail queue: 15529779200 bytes < 1.5*message size limit Feb 18 10:22:30 cliff2 postfix/smtpd[2402]: warning: not enough free space in mail queue: 15584747520 bytes < 1.5*message size limit Feb 18 11:22:30 cliff2 postfix/smtpd[6039]: warning: not enough free space in mail queue: 15494496256 bytes < 1.5*message size limit Feb 18 12:02:02 cliff2 postfix/smtpd[6761]: warning: not enough free space in mail queue: 15571550208 bytes < 1.5*message size limit Feb 18 12:22:30 cliff2 postfix/smtpd[95569]: warning: not enough free space in mail queue: 15472168960 bytes < 1.5*message size limit [12:35 cliff2 dvl ~] %
It seems someone wants to send a bigger message. 15GB email? I don’t think so. You’re not sending that. At first, I thought, I’ll just boost the limit. In this case, no. Don’t send me that email.
I suspect the email is from Nagios. I suspect the email is generated by a notification and for some reason, it has exploded.
Nagios restart
I’ve discovered that Nagios restart reproduces the problem.
Let’s deliver that email locally by commenting out this line from /etc/mail/aliases:
#root: dan@langille.org
… and restarting Nagios did not generate the issue.
What is also interesting, I’m seeing similar reports such as Postfix SMTP server: errors from unifi01.int.unixathome.org[10.55.0.131] – yet the mail log on unifi01 is empty. Logging is working:
[16:38 unifi01 dvl ~] % sudo cat /var/log/maillog Feb 18 00:00:00 unifi01 newsyslog[98880]: logfile turned over Feb 18 16:38:45 unifi01 dma[4d136][84888]: new mail from user=dvl uid=1002 envelope_from=<dvl@unifi01.int.unixathome.org> Feb 18 16:38:45 unifi01 dma[4d136][84888]: mail to=<dan@langille.org> queued as 4d136.372c11442000 Feb 18 16:38:45 unifi01 dma[4d136.372c11442000][84890]: <dan@langille.org> trying delivery Feb 18 16:38:45 unifi01 dma[4d136.372c11442000][84890]: using smarthost (cliff.int.unixathome.org:25) Feb 18 16:38:45 unifi01 dma[4d136.372c11442000][84890]: trying remote delivery to cliff.int.unixathome.org [10.55.0.14] pref 0 Feb 18 16:38:45 unifi01 dma[4d136.372c11442000][84890]: <dan@langille.org> delivery successful
Yet, cliff2 says mail came from there:
Feb 18 16:23:41 cliff2 postfix/smtpd[28059]: connect from unifi01.int.unixathome.org[10.55.0.131] Feb 18 16:23:41 cliff2 postfix/smtpd[28059]: Anonymous TLS connection established from unifi01.int.unixathome.org[10.55.0.131]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (25 6/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 Feb 18 16:23:41 cliff2 postfix/smtpd[28059]: NOQUEUE: reject: MAIL from unifi01.int.unixathome.org[10.55.0.131]: 452 4.3.1 Insufficient system storage; proto=ESMTP helo=<unif i01.int.unixathome.org> Feb 18 16:23:41 cliff2 postfix/smtpd[28059]: warning: not enough free space in mail queue: 15225958400 bytes < 1.5*message size limit Feb 18 16:23:41 cliff2 postfix/cleanup[28068]: EF82B2C5A0: message-id=<20260218162341.EF82B2C5A0@cliff2.int.unixathome.org> Feb 18 16:23:41 cliff2 postfix/smtpd[28059]: disconnect from unifi01.int.unixathome.org[10.55.0.131] ehlo=2 starttls=1 mail=0/1 rset=1 quit=1 commands=5/6 Feb 18 16:23:41 cliff2 postfix/qmgr[7445]: EF82B2C5A0: from=<double-bounce@cliff2.int.unixathome.org>, size=1278, nrcpt=1 (queue active) Feb 18 16:23:41 cliff2 postfix/cleanup[28068]: F2A122BB68: message-id=<20260218162341.EF82B2C5A0@cliff2.int.unixathome.org> Feb 18 16:23:41 cliff2 postfix/local[28069]: EF82B2C5A0: to=<postmaster@cliff2.int.unixathome.org>, orig_to=<postmaster>, relay=local, delay=0.02, delays=0.01/0.01/0/0, dsn=2.0.0, status=sent (forwarded as F2A122BB68) Feb 18 16:23:41 cliff2 postfix/qmgr[7445]: F2A122BB68: from=<double-bounce@cliff2.int.unixathome.org>, size=1435, nrcpt=1 (queue active) Feb 18 16:23:41 cliff2 postfix/qmgr[7445]: EF82B2C5A0: removed Feb 18 16:23:42 cliff2 postfix/smtp[28070]: Untrusted TLS connection established to smtp.fastmail.com[103.168.172.60]:587: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 Feb 18 16:23:42 cliff2 postfix/smtp[28070]: F2A122BB68: to=<dan@langille.org>, orig_to=<postmaster>, relay=smtp.fastmail.com[103.168.172.60]:587, delay=0.65, delays=0/0.01/0.21/0.42, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 8FF3E80104 ti_phl-compute-08_3096974_1771431822_1 via phl-compute-08) Feb 18 16:23:42 cliff2 postfix/qmgr[7445]: F2A122BB68: removed
Now I’m wonderng if this is a Postfix configuration issue?
I did a postconf -n difference between the two hosts:
[12:00 pro05 dvl ~/tmp] % diff -ruN cliff1 cliff2 --- cliff1 2026-02-18 12:00:00 +++ cliff2 2026-02-18 11:59:20 @@ -1,4 +1,4 @@ -[16:59 cliff1 dvl ~] % postconf -n +[16:58 cliff2 dvl ~] % postconf -n alias_maps = hash:/etc/mail/aliases command_directory = /usr/local/sbin compatibility_level = 3.6 @@ -17,7 +17,7 @@ message_size_limit = 10485760000 meta_directory = /usr/local/libexec/postfix mydestination = $myhostname, localhost.$mydomain, localhost -myhostname = cliff1.int.unixathome.org +myhostname = cliff2.int.unixathome.org mynetworks_style = class newaliases_path = /usr/local/bin/newaliases queue_directory = /var/spool/postfix [12:00 pro05 dvl ~/tmp] %
Nothing.
Another source
Feb 18 18:02:02 dns-hidden-master dma[3ad8b][35058]: new mail from user=logcheck uid=915 envelope_from=<logcheck@dns-hidden-master.int.unixathome.org> Feb 18 18:02:02 dns-hidden-master dma[3ad8b][35058]: mail to=<dan@langille.org> queued as 3ad8b.51b887042000 Feb 18 18:02:02 dns-hidden-master dma[3ad8b.51b887042000][35110]: <dan@langille.org> trying delivery Feb 18 18:02:02 dns-hidden-master dma[3ad8b.51b887042000][35110]: using smarthost (cliff.int.unixathome.org:25) Feb 18 18:02:02 dns-hidden-master dma[3ad8b.51b887042000][35110]: trying remote delivery to cliff.int.unixathome.org [10.55.0.44] pref 0 Feb 18 18:02:02 dns-hidden-master dma[3ad8b.51b887042000][35110]: remote delivery deferred: cliff.int.unixathome.org [10.55.0.44] failed after MAIL FROM: 452 4.3.1 Insufficient system storage Feb 18 18:02:02 dns-hidden-master dma[3ad8b.51b887042000][35110]: trying remote delivery to cliff.int.unixathome.org [10.55.0.14] pref 0
Then I tried manually from that same source:
[18:03 dns-hidden-master dvl ~] % echo testing | mail dan@langille.org
The log entries:
Feb 18 18:03:51 dns-hidden-master dma[39652]: new mail from user=dvl uid=1002 envelope_from=<dvl@dns-hidden-master.int.unixathome.org> Feb 18 18:03:51 dns-hidden-master dma[39652]: mail to=<dan@langille.org> queued as 39652.3ee88ac42000 Feb 18 18:03:51 dns-hidden-master dma[39652.3ee88ac42000][45462]: <dan@langille.org> trying delivery Feb 18 18:03:51 dns-hidden-master dma[39652.3ee88ac42000][45462]: using smarthost (cliff.int.unixathome.org:25) Feb 18 18:03:51 dns-hidden-master dma[39652.3ee88ac42000][45462]: trying remote delivery to cliff.int.unixathome.org [10.55.0.44] pref 0 Feb 18 18:03:51 dns-hidden-master dma[39652.3ee88ac42000][45462]: remote delivery deferred: cliff.int.unixathome.org [10.55.0.44] failed after MAIL FROM: 452 4.3.1 Insufficient system storage Feb 18 18:03:51 dns-hidden-master dma[39652.3ee88ac42000][45462]: trying remote delivery to cliff.int.unixathome.org [10.55.0.14] pref 0 Feb 18 18:03:51 dns-hidden-master dma[39652.3ee88ac42000][45462]: <dan@langille.org> delivery successful
So, a very small message is triggering the event. I tried again. It is repeatable.
Simple telnet
telnet is a time honored tool. The following highlighted lines are the ones I typed.
[18:24 mydev dvl ~] % telnet cliff2 25 Trying 10.55.0.44... Connected to cliff2.int.unixathome.org. Escape character is '^]'. 220 cliff2.int.unixathome.org ESMTP Postfix EHLO mydev 250-cliff2.int.unixathome.org 250-PIPELINING 250-SIZE 10485760000 250-ETRN 250-STARTTLS 250-ENHANCEDSTATUSCODES 250-8BITMIME 250-DSN 250-SMTPUTF8 250 CHUNKING MAIL FROM:dan@langille.org 452 4.3.1 Insufficient system storage
So, that’s the same as the stuff at the top of this post, apart from STARTTLS which I was not prepared to from within telnet.
I tried this with cliff1 – flawless. All good there.
Restarting postfix, then the jail
I’m sure it’s not space:
[16:54 r730-01 dvl ~] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data01 5.81T 6.36G 5.81T - - 2% 0% 1.00x ONLINE - data02 928G 705G 223G - - 67% 75% 1.00x ONLINE - data03 7.25T 1.30T 5.95T - - 49% 17% 1.00x ONLINE - data04 29.1T 6.11T 23.0T - - 0% 21% 1.00x ONLINE - zroot 107G 47.9G 59.1G - - 54% 44% 1.00x ONLINE - [18:30 r730-01 dvl ~] % zfs list | grep cliff2 data02/jails/cliff2 4.00G 13.6G 2.46G /jails/cliff2
It’s all just one filesystem in there:
[18:36 cliff2 dvl ~] % zfs list no datasets available [18:36 cliff2 dvl ~] % df -h Filesystem Size Used Avail Capacity Mounted on data02/jails/cliff2 16G 2.5G 14G 15% /
Let’s try:
[18:30 cliff2 dvl ~] % sudo service postfix restart postfix/postfix-script: stopping the Postfix mail system postfix/postfix-script: starting the Postfix mail system [18:30 cliff2 dvl ~] %
That didn’t help.
I tried a jail restart, did not help:
[18:31 r730-01 dvl ~] % sudo service jail restart cliff2 Stopping jails: cliff2. Starting jails: cliff2.
Let’s try snapshots:
[18:37 r730-01 dvl ~] % zfs list -r -t snapshot data02/jails/cliff2 | grep -v @autosnap NAME USED AVAIL REFER MOUNTPOINT data02/jails/cliff2@mkjail-202509051453 725M - 2.24G - data02/jails/cliff2@mkjail-202602101242 716M - 2.38G -
Let’s destroy those two:
[18:37 r730-01 dvl ~] % sudo zfs destroy data02/jails/cliff2@mkjail-202509051453 [18:37 r730-01 dvl ~] % sudo zfs destroy data02/jails/cliff2@mkjail-202602101242
Umm, that seems to have fixed it:
[18:39 mydev dvl ~] % telnet cliff2 25 Trying 10.55.0.44... Connected to cliff2.int.unixathome.org. Escape character is '^]'. 220 cliff2.int.unixathome.org ESMTP Postfix EHLO mydev 250-cliff2.int.unixathome.org 250-PIPELINING 250-SIZE 10485760000 250-ETRN 250-STARTTLS 250-ENHANCEDSTATUSCODES 250-8BITMIME 250-DSN 250-SMTPUTF8 250 CHUNKING MAIL FROM:dan@langille.org 250 2.1.0 Ok
WTF?
[18:39 r730-01 dvl ~] % zfs list data02/jails/cliff2 NAME USED AVAIL REFER MOUNTPOINT data02/jails/cliff2 2.53G 15.0G 2.46G /jails/cliff2 [18:39 r730-01 dvl ~] % zpool list data02 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data02 928G 704G 224G - - 66% 75% 1.00x ONLINE - [18:40 r730-01 dvl ~] %
No quota or reservation
The pool had 223G free at the start of this.
Let’s look around…
[18:40 r730-01 dvl ~] % zfs get all data02/jails/cliff2 | grep quota data02/jails/cliff2 quota none default data02/jails/cliff2 refquota none default data02/jails/cliff2 defaultuserquota 0 - data02/jails/cliff2 defaultgroupquota 0 - data02/jails/cliff2 defaultprojectquota 0 - data02/jails/cliff2 defaultuserobjquota 0 - data02/jails/cliff2 defaultgroupobjquota 0 - data02/jails/cliff2 defaultprojectobjquota 0 - [18:41 r730-01 dvl ~] % zfs get all data02/jails | grep quota data02/jails quota none default data02/jails refquota none default data02/jails defaultuserquota 0 - data02/jails defaultgroupquota 0 - data02/jails defaultprojectquota 0 - data02/jails defaultuserobjquota 0 - data02/jails defaultgroupobjquota 0 - data02/jails defaultprojectobjquota 0 - [18:41 r730-01 dvl ~] % zfs get all data02 | grep quota data02 quota none default data02 refquota none default data02 defaultuserquota 0 - data02 defaultgroupquota 0 - data02 defaultprojectquota 0 - data02 defaultuserobjquota 0 - data02 defaultgroupobjquota 0 - data02 defaultprojectobjquota 0 - [18:41 r730-01 dvl ~] % zfs get all data02 | grep reserve [18:41 r730-01 dvl ~] % zfs get all data02 | grep res data02 compressratio 1.79x - data02 reservation none default data02 compression zstd received data02 aclinherit restricted default data02 sharesmb off default data02 refreservation none default data02 usedbyrefreservation 0B - data02 refcompressratio 1.00x - [18:41 r730-01 dvl ~] % zfs get all data02/jails/cliff2 | grep reservation data02/jails/cliff2 reservation none default data02/jails/cliff2 refreservation none default data02/jails/cliff2 usedbyrefreservation 0B - [18:42 r730-01 dvl ~] % zfs get all data02/jails | grep reservation data02/jails reservation none default data02/jails refreservation none default data02/jails usedbyrefreservation 0B - [18:42 r730-01 dvl ~] % zfs get all data02 | grep reservation data02 reservation none default data02 refreservation none default data02 usedbyrefreservation 0B - [18:42 r730-01 dvl ~] %
So, what was blocking this?
zpool status
As an afterthought:
[18:42 r730-01 dvl ~] % zpool status pool: data01 state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:00:06 with 0 errors on Thu Feb 12 03:53:00 2026 config: NAME STATE READ WRITE CKSUM data01 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/Y7P0A022TEVE ONLINE 0 0 0 gpt/Y7P0A02ATEVE ONLINE 0 0 0 gpt/Y7P0A02DTEVE ONLINE 0 0 0 gpt/Y7P0A02GTEVE ONLINE 0 0 0 gpt/Y7P0A02LTEVE ONLINE 0 0 0 gpt/Y7P0A02MTEVE ONLINE 0 0 0 gpt/Y7P0A02QTEVE ONLINE 0 0 0 gpt/Y7P0A033TEVE ONLINE 0 0 0 errors: No known data errors pool: data02 state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:08:44 with 0 errors on Wed Feb 18 04:03:38 2026 config: NAME STATE READ WRITE CKSUM data02 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/S6WSNJ0T208743F ONLINE 0 0 0 gpt/S6WSNJ0T207774T ONLINE 0 0 0 errors: No known data errors pool: data03 state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:49:32 with 0 errors on Thu Feb 12 04:42:48 2026 config: NAME STATE READ WRITE CKSUM data03 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/WD_22492H800867 ONLINE 0 0 0 gpt/WD_230151801284 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/WD_230151801478 ONLINE 0 0 0 gpt/WD_230151800473 ONLINE 0 0 0 errors: No known data errors pool: data04 state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:48:58 with 0 errors on Wed Feb 18 04:44:04 2026 config: NAME STATE READ WRITE CKSUM data04 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/S7KGNU0Y722875X ONLINE 0 0 0 gpt/S7KGNU0Y915666E ONLINE 0 0 0 gpt/S7KGNU0Y912937J ONLINE 0 0 0 gpt/S7KGNU0Y912955D ONLINE 0 0 0 gpt/S7U8NJ0Y716854P ONLINE 0 0 0 gpt/S7U8NJ0Y716801F ONLINE 0 0 0 gpt/S757NS0Y700758M ONLINE 0 0 0 gpt/S757NS0Y700760R ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:02:22 with 0 errors on Tue Feb 17 04:03:50 2026 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/zfs0_20170718AA0000185556 ONLINE 0 0 0 gpt/zfs1_20170719AA1178164201 ONLINE 0 0 0 errors: No known data errors
Cause found
The cause has been found. It was another dataset, with a reservation. It was discussed during the ZFS production users call. They asked me a few questions, I tried a few times, etc.
First, write some data.
[22:25 cliff2 dvl ~] % sudo dd if=/dev/random of=/tmp/delete=me bs=1M count=20000 status=progress dd: /tmp/delete=me: No space left on devicerred 76.001s, 220 MB/s 16172+0 records in 16171+1 records out 16956915712 bytes transferred in 79.099811 secs (214373656 bytes/sec)
OK, it filled up. Next, we looked at this.
[22:31 r730-01 dvl ~] % zpool status data02 pool: data02 state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:08:44 with 0 errors on Wed Feb 18 04:03:38 2026 config: NAME STATE READ WRITE CKSUM data02 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/S6WSNJ0T208743F ONLINE 0 0 0 gpt/S6WSNJ0T207774T ONLINE 0 0 0 errors: No known data errors [22:31 r730-01 dvl ~] % zfs list data02/jails NAME USED AVAIL REFER MOUNTPOINT data02/jails 363G 0B 9.54G /jails [22:32 r730-01 dvl ~] % zfs list -r data02 NAME USED AVAIL REFER MOUNTPOINT data02 900G 0B 96K none data02/freshports 294G 0B 88K none data02/freshports/dev-ingress01 228G 0B 88K none data02/freshports/dev-ingress01/dvl-src 197G 0B 197G /jails/dev-ingress01/usr/home/dvl/src data02/freshports/dev-ingress01/freshports 22.5G 0B 2.06G /jails/dev-ingress01/var/db/freshports data02/freshports/dev-ingress01/freshports/cache 2.11M 0B 132K /jails/dev-ingress01/var/db/freshports/cache data02/freshports/dev-ingress01/freshports/cache/html 1.88M 0B 1.88M /jails/dev-ingress01/var/db/freshports/cache/html data02/freshports/dev-ingress01/freshports/cache/spooling 104K 0B 104K /jails/dev-ingress01/var/db/freshports/cache/spooling data02/freshports/dev-ingress01/freshports/message-queues 20.5G 0B 11.7M /jails/dev-ingress01/var/db/freshports/message-queues data02/freshports/dev-ingress01/freshports/message-queues/archive 20.4G 0B 11.3G /jails/dev-ingress01/var/db/freshports/message-queues/archive data02/freshports/dev-ingress01/ingress 5.46G 0B 132K /jails/dev-ingress01/var/db/ingress data02/freshports/dev-ingress01/ingress/latest_commits 528K 0B 108K /jails/dev-ingress01/var/db/ingress/latest_commits data02/freshports/dev-ingress01/ingress/message-queues 1.44M 0B 628K /jails/dev-ingress01/var/db/ingress/message-queues data02/freshports/dev-ingress01/ingress/repos 5.45G 0B 120K /jails/dev-ingress01/var/db/ingress/repos data02/freshports/dev-ingress01/ingress/repos/doc 525M 0B 522M /jails/dev-ingress01/var/db/ingress/repos/doc data02/freshports/dev-ingress01/ingress/repos/ports 2.28G 0B 2.27G /jails/dev-ingress01/var/db/ingress/repos/ports data02/freshports/dev-ingress01/ingress/repos/src 2.66G 0B 2.65G /jails/dev-ingress01/var/db/ingress/repos/src data02/freshports/dev-ingress01/jails 3.00G 0B 104K /jails/dev-ingress01/jails data02/freshports/dev-ingress01/jails/freshports 3.00G 0B 405M /jails/dev-ingress01/jails/freshports data02/freshports/dev-ingress01/jails/freshports/ports 2.61G 0B 2.61G /jails/dev-ingress01/jails/freshports/usr/ports data02/freshports/dev-ingress01/modules 4.38M 0B 4.38M /jails/dev-ingress01/usr/local/lib/perl5/site_perl/FreshPorts data02/freshports/dev-ingress01/scripts 3.30M 0B 3.30M /jails/dev-ingress01/usr/local/libexec/freshports data02/freshports/dev-nginx01 54.6M 0B 96K none data02/freshports/dev-nginx01/www 54.5M 0B 96K /jails/dev-nginx01/usr/local/www data02/freshports/dev-nginx01/www/freshports 51.7M 0B 51.7M /jails/dev-nginx01/usr/local/www/freshports data02/freshports/dev-nginx01/www/freshsource 2.71M 0B 2.71M /jails/dev-nginx01/usr/local/www/freshsource data02/freshports/dvl-ingress01 18.9G 0B 96K none data02/freshports/dvl-ingress01/dvl-src 80.3M 0B 80.3M /jails/dvl-ingress01/usr/home/dvl/src data02/freshports/dvl-ingress01/freshports 4.38G 0B 96K /jails/dvl-ingress01/var/db/freshports data02/freshports/dvl-ingress01/freshports/cache 2.31M 0B 96K /jails/dvl-ingress01/var/db/freshports/cache data02/freshports/dvl-ingress01/freshports/cache/html 2.01M 0B 1.93M /jails/dvl-ingress01/var/db/freshports/cache/html data02/freshports/dvl-ingress01/freshports/cache/spooling 208K 0B 208K /jails/dvl-ingress01/var/db/freshports/cache/spooling data02/freshports/dvl-ingress01/freshports/message-queues 4.38G 0B 16.8M /jails/dvl-ingress01/var/db/freshports/message-queues data02/freshports/dvl-ingress01/freshports/message-queues/archive 4.37G 0B 4.37G /jails/dvl-ingress01/var/db/freshports/message-queues/archive data02/freshports/dvl-ingress01/ingress 8.60G 0B 140K /jails/dvl-ingress01/var/db/ingress data02/freshports/dvl-ingress01/ingress/latest_commits 100K 0B 100K /jails/dvl-ingress01/var/db/ingress/latest_commits data02/freshports/dvl-ingress01/ingress/message-queues 160K 0B 160K /jails/dvl-ingress01/var/db/ingress/message-queues data02/freshports/dvl-ingress01/ingress/repos 8.60G 0B 112K /jails/dvl-ingress01/var/db/ingress/repos data02/freshports/dvl-ingress01/ingress/repos/doc 954M 0B 520M /jails/dvl-ingress01/var/db/ingress/repos/doc data02/freshports/dvl-ingress01/ingress/repos/ports 3.43G 0B 2.22G /jails/dvl-ingress01/var/db/ingress/repos/ports data02/freshports/dvl-ingress01/ingress/repos/src 4.24G 0B 2.56G /jails/dvl-ingress01/var/db/ingress/repos/src data02/freshports/dvl-ingress01/jails 5.83G 0B 104K /jails/dvl-ingress01/jails data02/freshports/dvl-ingress01/jails/freshports 5.83G 0B 404M /jails/dvl-ingress01/jails/freshports data02/freshports/dvl-ingress01/jails/freshports/ports 5.43G 0B 2.64G /jails/dvl-ingress01/jails/freshports/usr/ports data02/freshports/dvl-ingress01/modules 2.67M 0B 2.67M /jails/dvl-ingress01/usr/local/lib/perl5/site_perl/FreshPorts data02/freshports/dvl-ingress01/scripts 2.34M 0B 2.34M /jails/dvl-ingress01/usr/local/libexec/freshports data02/freshports/dvl-nginx01 22.2M 0B 96K none data02/freshports/dvl-nginx01/www 22.1M 0B 96K none data02/freshports/dvl-nginx01/www/freshports 20.2M 0B 20.2M /jails/dvl-nginx01/usr/local/www/freshports data02/freshports/dvl-nginx01/www/freshsource 1.78M 0B 1.78M /jails/dvl-nginx01/usr/local/www/freshsource data02/freshports/jailed 3.97G 0B 96K none data02/freshports/jailed/dev-ingress01 96K 0B 96K none data02/freshports/jailed/dev-nginx01 1.36G 0B 96K none data02/freshports/jailed/dev-nginx01/cache 1.36G 0B 96K /var/db/freshports/cache data02/freshports/jailed/dev-nginx01/cache/categories 1.28M 0B 1.20M /var/db/freshports/cache/categories data02/freshports/jailed/dev-nginx01/cache/commits 96K 0B 96K /var/db/freshports/cache/commits data02/freshports/jailed/dev-nginx01/cache/daily 12.0M 0B 11.9M /var/db/freshports/cache/daily data02/freshports/jailed/dev-nginx01/cache/general 4.38M 0B 4.30M /var/db/freshports/cache/general data02/freshports/jailed/dev-nginx01/cache/news 184K 0B 96K /var/db/freshports/cache/news data02/freshports/jailed/dev-nginx01/cache/packages 5.19M 0B 5.10M /var/db/freshports/cache/packages data02/freshports/jailed/dev-nginx01/cache/pages 96K 0B 96K /var/db/freshports/cache/pages data02/freshports/jailed/dev-nginx01/cache/ports 1.34G 0B 1.34G /var/db/freshports/cache/ports data02/freshports/jailed/dev-nginx01/cache/spooling 224K 0B 120K /var/db/freshports/cache/spooling data02/freshports/jailed/dvl-ingress01 192K 0B 96K none data02/freshports/jailed/dvl-ingress01/distfiles 96K 0B 96K none data02/freshports/jailed/dvl-nginx01 1.56M 0B 96K none data02/freshports/jailed/dvl-nginx01/cache 1.37M 0B 148K /var/db/freshports/cache data02/freshports/jailed/dvl-nginx01/cache/categories 96K 0B 96K /var/db/freshports/cache/categories data02/freshports/jailed/dvl-nginx01/cache/commits 96K 0B 96K /var/db/freshports/cache/commits data02/freshports/jailed/dvl-nginx01/cache/daily 96K 0B 96K /var/db/freshports/cache/daily data02/freshports/jailed/dvl-nginx01/cache/general 96K 0B 96K /var/db/freshports/cache/general data02/freshports/jailed/dvl-nginx01/cache/news 176K 0B 96K /var/db/freshports/cache/news data02/freshports/jailed/dvl-nginx01/cache/packages 96K 0B 96K /var/db/freshports/cache/packages data02/freshports/jailed/dvl-nginx01/cache/pages 96K 0B 96K /var/db/freshports/cache/pages data02/freshports/jailed/dvl-nginx01/cache/ports 208K 0B 128K /var/db/freshports/cache/ports data02/freshports/jailed/dvl-nginx01/cache/spooling 200K 0B 120K /var/db/freshports/cache/spooling data02/freshports/jailed/dvl-nginx01/freshports 96K 0B 96K none data02/freshports/jailed/stage-ingress01 192K 0B 96K none data02/freshports/jailed/stage-ingress01/data 96K 0B 96K none data02/freshports/jailed/stage-nginx01 1.62G 0B 96K none data02/freshports/jailed/stage-nginx01/cache 1.62G 0B 248K /var/db/freshports/cache data02/freshports/jailed/stage-nginx01/cache/categories 2.53M 0B 2.44M /var/db/freshports/cache/categories data02/freshports/jailed/stage-nginx01/cache/commits 96K 0B 96K /var/db/freshports/cache/commits data02/freshports/jailed/stage-nginx01/cache/daily 8.57M 0B 8.48M /var/db/freshports/cache/daily data02/freshports/jailed/stage-nginx01/cache/general 5.54M 0B 5.45M /var/db/freshports/cache/general data02/freshports/jailed/stage-nginx01/cache/news 184K 0B 96K /var/db/freshports/cache/news data02/freshports/jailed/stage-nginx01/cache/packages 13.4M 0B 13.3M /var/db/freshports/cache/packages data02/freshports/jailed/stage-nginx01/cache/pages 96K 0B 96K /var/db/freshports/cache/pages data02/freshports/jailed/stage-nginx01/cache/ports 1.59G 0B 1.59G /var/db/freshports/cache/ports data02/freshports/jailed/stage-nginx01/cache/spooling 232K 0B 120K /var/db/freshports/cache/spooling data02/freshports/jailed/test-ingress01 192K 0B 96K none data02/freshports/jailed/test-ingress01/data 96K 0B 96K none data02/freshports/jailed/test-nginx01 1008M 0B 96K none data02/freshports/jailed/test-nginx01/cache 1008M 0B 236K /var/db/freshports/cache data02/freshports/jailed/test-nginx01/cache/categories 2.32M 0B 2.24M /var/db/freshports/cache/categories data02/freshports/jailed/test-nginx01/cache/commits 96K 0B 96K /var/db/freshports/cache/commits data02/freshports/jailed/test-nginx01/cache/daily 12.4M 0B 12.3M /var/db/freshports/cache/daily data02/freshports/jailed/test-nginx01/cache/general 3.47M 0B 3.36M /var/db/freshports/cache/general data02/freshports/jailed/test-nginx01/cache/news 184K 0B 96K /var/db/freshports/cache/news data02/freshports/jailed/test-nginx01/cache/packages 4.51M 0B 4.43M /var/db/freshports/cache/packages data02/freshports/jailed/test-nginx01/cache/pages 96K 0B 96K /var/db/freshports/cache/pages data02/freshports/jailed/test-nginx01/cache/ports 985M 0B 985M /var/db/freshports/cache/ports data02/freshports/jailed/test-nginx01/cache/spooling 232K 0B 120K /var/db/freshports/cache/spooling data02/freshports/stage-ingress01 19.1G 0B 96K none data02/freshports/stage-ingress01/cache 2.14M 0B 96K /jails/stage-ingress01/var/db/freshports/cache data02/freshports/stage-ingress01/cache/html 1.94M 0B 1.86M /jails/stage-ingress01/var/db/freshports/cache/html data02/freshports/stage-ingress01/cache/spooling 104K 0B 104K /jails/stage-ingress01/var/db/freshports/cache/spooling data02/freshports/stage-ingress01/freshports 10.7G 0B 96K none data02/freshports/stage-ingress01/freshports/archive 10.7G 0B 10.7G /jails/stage-ingress01/var/db/freshports/message-queues/archive data02/freshports/stage-ingress01/freshports/message-queues 9.96M 0B 7.89M /jails/stage-ingress01/var/db/freshports/message-queues data02/freshports/stage-ingress01/ingress 5.33G 0B 96K /jails/stage-ingress01/var/db/ingress data02/freshports/stage-ingress01/ingress/latest_commits 404K 0B 100K /jails/stage-ingress01/var/db/ingress/latest_commits data02/freshports/stage-ingress01/ingress/message-queues 1012K 0B 180K /jails/stage-ingress01/var/db/ingress/message-queues data02/freshports/stage-ingress01/ingress/repos 5.33G 0B 5.31G /jails/stage-ingress01/var/db/ingress/repos data02/freshports/stage-ingress01/jails 405M 0B 104K /jails/stage-ingress01/jails data02/freshports/stage-ingress01/jails/freshports 404M 0B 404M /jails/stage-ingress01/jails/freshports data02/freshports/stage-ingress01/ports 2.63G 0B 2.63G /jails/stage-ingress01/jails/freshports/usr/ports data02/freshports/test-ingress01 23.9G 0B 96K none data02/freshports/test-ingress01/freshports 12.8G 0B 2.05G /jails/test-ingress01/var/db/freshports data02/freshports/test-ingress01/freshports/cache 2.09M 0B 96K /jails/test-ingress01/var/db/freshports/cache data02/freshports/test-ingress01/freshports/cache/html 1.89M 0B 1.89M /jails/test-ingress01/var/db/freshports/cache/html data02/freshports/test-ingress01/freshports/cache/spooling 104K 0B 104K /jails/test-ingress01/var/db/freshports/cache/spooling data02/freshports/test-ingress01/freshports/message-queues 10.8G 0B 9.07M /jails/test-ingress01/var/db/freshports/message-queues data02/freshports/test-ingress01/freshports/message-queues/archive 10.8G 0B 10.8G /jails/test-ingress01/var/db/freshports/message-queues/archive data02/freshports/test-ingress01/ingress 8.10G 0B 128K /jails/test-ingress01/var/db/ingress data02/freshports/test-ingress01/ingress/latest_commits 344K 0B 100K /jails/test-ingress01/var/db/ingress/latest_commits data02/freshports/test-ingress01/ingress/message-queues 932K 0B 164K /jails/test-ingress01/var/db/ingress/message-queues data02/freshports/test-ingress01/ingress/repos 8.10G 0B 5.33G /jails/test-ingress01/var/db/ingress/repos data02/freshports/test-ingress01/jails 3.00G 0B 96K /jails/test-ingress01/jails data02/freshports/test-ingress01/jails/freshports 3.00G 0B 405M /jails/test-ingress01/jails/freshports data02/freshports/test-ingress01/jails/freshports/ports 2.60G 0B 2.60G /jails/test-ingress01/jails/freshports/usr/ports data02/jails 363G 0B 9.54G /jails data02/jails/bacula 17.8G 0B 16.3G /jails/bacula data02/jails/bacula-sd-02 4.49G 0B 2.91G /jails/bacula-sd-02 data02/jails/bacula-sd-03 5.62G 0B 4.11G /jails/bacula-sd-03 data02/jails/besser 9.29G 0B 5.72G /jails/besser data02/jails/certs 3.93G 0B 2.41G /jails/certs data02/jails/certs-rsync 3.91G 0B 2.42G /jails/certs-rsync data02/jails/cliff2 18.3G 0B 18.3G /jails/cliff2 data02/jails/dev-ingress01 6.07G 0B 3.97G /jails/dev-ingress01 data02/jails/dev-nginx01 5.02G 0B 3.36G /jails/dev-nginx01 data02/jails/dns-hidden-master 4.36G 0B 2.64G /jails/dns-hidden-master data02/jails/dns1 12.0G 0B 4.90G /jails/dns1 data02/jails/dvl-ingress01 10.1G 0B 6.29G /jails/dvl-ingress01 data02/jails/dvl-nginx01 2.89G 0B 1.28G /jails/dvl-nginx01 data02/jails/git 5.94G 0B 4.26G /jails/git data02/jails/jail_within_jail 1.44G 0B 585M /jails/jail_within_jail data02/jails/mqtt01 4.68G 0B 3.03G /jails/mqtt01 data02/jails/mydev 26.7G 0B 19.8G /jails/mydev data02/jails/mysql01 14.7G 0B 5.40G /jails/mysql01 data02/jails/mysql02 5.75G 0B 7.79G /jails/mysql02 data02/jails/nsnotify 4.98G 0B 2.60G /jails/nsnotify data02/jails/pg01 50.9G 0B 10.8G /jails/pg01 data02/jails/pg02 13.7G 0B 11.0G /jails/pg02 data02/jails/pg03 14.5G 0B 10.8G /jails/pg03 data02/jails/pkg01 17.9G 0B 13.7G /jails/pkg01 data02/jails/samdrucker 5.74G 0B 4.16G /jails/samdrucker data02/jails/serpico 4.06G 0B 2.50G /jails/serpico data02/jails/stage-ingress01 7.25G 0B 3.63G /jails/stage-ingress01 data02/jails/stage-nginx01 3.02G 0B 1.44G /jails/stage-nginx01 data02/jails/svn 11.6G 0B 9.77G /jails/svn data02/jails/talos 3.82G 0B 2.38G /jails/talos data02/jails/test-ingress01 3.89G 0B 1.89G /jails/test-ingress01 data02/jails/test-nginx01 2.99G 0B 1.38G /jails/test-nginx01 data02/jails/unifi01 32.4G 0B 12.5G /jails/unifi01 data02/jails/webserver 13.3G 0B 11.3G /jails/webserver data02/reserved 180G 180G 96K none data02/vm 62.0G 0B 6.41G /usr/local/vm data02/vm/freebsd-test 701M 0B 112K /usr/local/vm/freebsd-test data02/vm/freebsd-test/disk0 700M 0B 700M - data02/vm/hass 52.0G 0B 13.4G /usr/local/vm/hass data02/vm/home-assistant 351M 0B 351M /usr/local/vm/home-assistant data02/vm/myguest 2.55G 0B 2.55G /usr/local/vm/myguest
After putting that into a gist and pastiing it to the channle, I noticed the highlighted line 180 above.
Then I remembered. Some time ago I’d create this dataset to stop the zpool from filling up. I had read about this approach, and tried it.
I freed up space by deleting that /tmp/delete=me file (I didn’t intend that to be an =).
The next day, I ran this command to clear the reservation and free up that “free space”.
[13:23 r730-01 dvl ~] % sudo zfs set refreservation=0 data02/reserved [13:24 r730-01 dvl ~] %
Monitoring
What has failed me is my monitoring. The zpool was getting full, but not in this sense:
[14:20 r730-01 dvl ~] % zpool list data02 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data02 928G 705G 223G - - 67% 75% 1.00x ONLINE -
I’m sure the default monitoring on this zpool is checking the CAP (capacity) – I may look at adjusting the monitoring.
Cleaning up old snapshots
I need to move most of data02 into data04.
[14:24 r730-01 dvl ~] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data01 5.81T 6.36G 5.81T - - 2% 0% 1.00x ONLINE - data02 928G 618G 310G - - 56% 66% 1.00x ONLINE - data03 7.25T 1.27T 5.98T - - 48% 17% 1.00x ONLINE - data04 29.1T 6.11T 23.0T - - 0% 21% 1.00x ONLINE - zroot 107G 48.2G 58.8G - - 53% 45% 1.00x ONLINE -
In the short term, I’ll buy some time by deleting old snapshots. Look at all this old stuff:
[14:31 r730-01 dvl ~] % zfs list -o name -r -t snapshot data02 | grep @mkjail data02/jails/bacula@mkjail-202509051453 data02/jails/bacula@mkjail-202602101338 data02/jails/bacula-sd-02@mkjail-202509051453 data02/jails/bacula-sd-02@mkjail-202602101338 data02/jails/bacula-sd-03@mkjail-202509051453 data02/jails/bacula-sd-03@mkjail-202602101338 data02/jails/besser@mkjail-202602101619 data02/jails/certs@mkjail-202509051453 data02/jails/certs@mkjail-202602101338 data02/jails/certs-rsync@mkjail-202509051453 data02/jails/certs-rsync@mkjail-202602101338 data02/jails/dev-ingress01@mkjail-202509051453 data02/jails/dev-ingress01@mkjail-202602101338 data02/jails/dev-nginx01@mkjail-202509051453 data02/jails/dev-nginx01@mkjail-202602101338 data02/jails/dns-hidden-master@mkjail-202509051453 data02/jails/dns-hidden-master@mkjail-202602101513 data02/jails/dns1@mkjail-202509051453 data02/jails/dns1@mkjail-202602101242 data02/jails/dvl-ingress01@mkjail-202509051453 data02/jails/dvl-ingress01@mkjail-202602101338 data02/jails/dvl-nginx01@mkjail-202509051453 data02/jails/dvl-nginx01@mkjail-202602101338 data02/jails/git@mkjail-202509051453 data02/jails/git@mkjail-202602101338 data02/jails/jail_within_jail@mkjail-202509051453 data02/jails/jail_within_jail@mkjail-202602101609 data02/jails/mqtt01@mkjail-202509051453 data02/jails/mqtt01@mkjail-202602101338 data02/jails/mydev@mkjail-202509051453 data02/jails/mydev@mkjail-202602101529 data02/jails/mysql01@mkjail-202509051453 data02/jails/mysql01@mkjail-202602101242 data02/jails/mysql02@mkjail-202602102333 data02/jails/nsnotify@mkjail-202509051453 data02/jails/nsnotify@mkjail-202602101551 data02/jails/nsnotify@mkjail-202602101558 data02/jails/pg01@mkjail-202509051453 data02/jails/pg01@mkjail-202602101338 data02/jails/pg02@mkjail-202509051453 data02/jails/pg02@mkjail-202602101338 data02/jails/pg03@mkjail-202509051453 data02/jails/pg03@mkjail-202602101338 data02/jails/pkg01@mkjail-202509051453 data02/jails/pkg01@mkjail-202602082233 data02/jails/samdrucker@mkjail-202509051453 data02/jails/samdrucker@mkjail-202602101338 data02/jails/serpico@mkjail-202509051453 data02/jails/serpico@mkjail-202602101536 data02/jails/stage-ingress01@mkjail-202509051453 data02/jails/stage-ingress01@mkjail-202602101338 data02/jails/stage-nginx01@mkjail-202509051453 data02/jails/stage-nginx01@mkjail-202602101338 data02/jails/svn@mkjail-202509051453 data02/jails/svn@mkjail-202602101338 data02/jails/talos@mkjail-202509051453 data02/jails/talos@mkjail-202602101338 data02/jails/test-ingress01@mkjail-202509051453 data02/jails/test-ingress01@mkjail-202602101338 data02/jails/test-nginx01@mkjail-202509051453 data02/jails/test-nginx01@mkjail-202602101338 data02/jails/unifi01@mkjail-202509051453 data02/jails/unifi01@mkjail-202602101605 data02/jails/webserver@mkjail-202509051453 data02/jails/webserver@mkjail-202602101338
Quickly scripted:
[14:32 r730-01 dvl ~] % zfs list -o name -r -t snapshot data02 | grep @mkjail | xargs -n 1 sudo zfs destroy [14:32 r730-01 dvl ~] % zpool list data02 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data02 928G 618G 310G - - 56% 66% 1.00x ONLINE -
But there’s more, that was just mkjail related.
[14:42 r730-01 dvl ~] % zfs list -o name -r -t snapshot data02 | grep -v @autosnap | grep -v @empty NAME data02/freshports/dvl-ingress01/ingress/repos/doc@for-dvl-ingress01 data02/freshports/dvl-ingress01/ingress/repos/ports@for-dvl-ingress01 data02/freshports/dvl-ingress01/ingress/repos/src@for-dvl-ingress01 data02/jails/besser@before.26.1.0 data02/jails/besser@before.26.1.1 data02/jails/besser@before.26.1.1_1 data02/jails/besser@before.26.1.1_2 data02/jails/besser@before.26.2.0 data02/jails/dns1@before.bind918 data02/jails/dns1@bind.9.18 data02/jails/mysql01@MySQL-8.0 data02/jails/mysql01@mysql80-part2 data02/jails/mysql01@mysql80-part3 data02/jails/pkg01@before15.0 data02/jails/unifi01@before.mongodb60-6.0.24
@autosnap is sanoid-related. @empty is on some special FreshPorts filesystems. The reset look good to go.
Let’s try a dry run, and notice we don’t need sudo for that.
[14:49 r730-01 dvl ~] % zfs list -o name -r -t snapshot data02 | grep -v @autosnap | grep -v @empty | xargs -n 1 zfs destroy -n cannot open 'NAME': dataset does not exist cannot destroy 'data02/jails/mysql01@mysql80-part3': snapshot has dependent clones use '-R' to destroy the following datasets: data02/jails/mysql02@autosnap_2026-02-13_00:00:01_daily data02/jails/mysql02@autosnap_2026-02-14_00:00:11_daily data02/jails/mysql02@autosnap_2026-02-15_00:00:01_daily data02/jails/mysql02@autosnap_2026-02-16_00:00:02_daily data02/jails/mysql02@autosnap_2026-02-17_00:00:00_daily data02/jails/mysql02@autosnap_2026-02-18_00:00:06_daily data02/jails/mysql02@autosnap_2026-02-19_00:00:03_daily data02/jails/mysql02@autosnap_2026-02-19_08:00:04_hourly data02/jails/mysql02@autosnap_2026-02-19_09:00:01_hourly data02/jails/mysql02@autosnap_2026-02-19_10:00:02_hourly data02/jails/mysql02@autosnap_2026-02-19_10:00:02_frequently data02/jails/mysql02@autosnap_2026-02-19_10:15:09_frequently data02/jails/mysql02@autosnap_2026-02-19_10:30:02_frequently data02/jails/mysql02@autosnap_2026-02-19_10:45:09_frequently data02/jails/mysql02@autosnap_2026-02-19_11:00:01_hourly data02/jails/mysql02@autosnap_2026-02-19_11:00:01_frequently data02/jails/mysql02@autosnap_2026-02-19_11:15:08_frequently data02/jails/mysql02@autosnap_2026-02-19_11:30:00_frequently data02/jails/mysql02@autosnap_2026-02-19_11:45:08_frequently data02/jails/mysql02@autosnap_2026-02-19_12:00:01_hourly data02/jails/mysql02@autosnap_2026-02-19_12:00:01_frequently data02/jails/mysql02@autosnap_2026-02-19_12:15:08_frequently data02/jails/mysql02@autosnap_2026-02-19_12:30:01_frequently data02/jails/mysql02@autosnap_2026-02-19_12:45:09_frequently data02/jails/mysql02@autosnap_2026-02-19_13:00:02_hourly data02/jails/mysql02@autosnap_2026-02-19_13:00:02_frequently data02/jails/mysql02@autosnap_2026-02-19_13:15:09_frequently data02/jails/mysql02@autosnap_2026-02-19_13:30:01_frequently data02/jails/mysql02@autosnap_2026-02-19_13:45:08_frequently data02/jails/mysql02@autosnap_2026-02-19_14:00:03_hourly data02/jails/mysql02@autosnap_2026-02-19_14:00:03_frequently data02/jails/mysql02@autosnap_2026-02-19_14:15:09_frequently data02/jails/mysql02@autosnap_2026-02-19_14:30:00_frequently data02/jails/mysql02@autosnap_2026-02-19_14:45:10_frequently data02/jails/mysql02
Two things:
- Add -H to remove that NAME header
- don’t touch data02/jails/mysql01@mysql80-part3 yet
[14:50 r730-01 dvl ~] % zfs list -Ho name -r -t snapshot data02 | grep -v @autosnap | grep -v @empty | grep -v mysql80-part3 | xargs -n 1 zfs destroy -n [14:51 r730-01 dvl ~] %
That looks good. This time, for real (with sudo and without -n)
[14:51 r730-01 dvl ~] % zfs list -Ho name -r -t snapshot data02 | grep -v @autosnap | grep -v @empty | grep -v mysql80-part3 | xargs -n 1 sudo zfs destroy [14:54 r730-01 dvl ~] % zpool list data02 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data02 928G 591G 337G - - 53% 63% 1.00x ONLINE -
So what?
Yeah, this was my fault. As most things computing-related are. It’s never the system’s fault.
I am grateful that although some things stopped working, nothing seems corrupted. I don’t mean from a bitrot/checksum point of view (with respect to ZFS). I mean the MySQL database is not corrupted, the PostgreSQL databases are fine (as expected), none of the FreshPorts nodes were affected at all (they kept processing incoming commits).
The Postfix issue seems to have been the canary. I suspect it won’t accept an incoming message unless it can allocate a certain amount of space. It could not. It coughed. If I had not had a reservation on data02/reserved, usage would have grown to 80% when monitoring would have triggered.
I think my reservation has to be less than 20% of the pool size, let’s save 15% .15 * 928 = 139.2
Let’s this this idea.
Get the pool to 80%
Let’s trigger the monitoring alert.
[17:58 r730-01 dvl ~] % zpool list data02 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data02 928G 591G 337G - - 53% 63% 1.00x ONLINE -
How full is that? .8 * 928 = 742.4 which is 151.4 more than the 591 now in use (that’s all in GB).
Let’s write to a file and take up some space.
[18:17 cliff2 dvl ~] % sudo dd if=/dev/random of=/tmp/delete-me bs=1M count=151400 status=progress 158554128384 bytes (159 GB, 148 GiB) transferred 622.025s, 255 MB/s 151400+0 records in 151400+0 records out 158754406400 bytes transferred in 622.731688 secs (254932276 bytes/sec)
Close, oh, so close!
[18:25 r730-01 dvl ~] % zpool list data02 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data02 928G 739G 189G - - 67% 79% 1.00x ONLINE -
Let’s do another 5G to another file:
[18:27 cliff2 dvl ~] % sudo dd if=/dev/random of=/tmp/delete-me-2 bs=1M count=5000 status=progress 5109710848 bytes (5110 MB, 4873 MiB) transferred 17.001s, 301 MB/s 5000+0 records in 5000+0 records out 5242880000 bytes transferred in 17.443215 secs (300568443 bytes/sec) [18:31 r730-01 dvl ~] % zpool list data02 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data02 928G 743G 185G - - 68% 80% 1.00x ONLINE -
There we go! Target acquired.
Good, and the monitoring is triggered (after I manually scheduled a check): ZFS POOL ALARM: POOL data02 usage is WARNING (80%)
So let’s go with a reservation of 130G, and take note of the AVAIL value before and after.
[18:35 r730-01 dvl ~] % zfs list data02 NAME USED AVAIL REFER MOUNTPOINT data02 744G 155G 96K none [18:35 r730-01 dvl ~] % sudo zfs set refreservation=130G data02/reserved [18:35 r730-01 dvl ~] % zfs list data02 NAME USED AVAIL REFER MOUNTPOINT data02 874G 25.1G 96K none [18:36 r730-01 dvl ~] %
Let’s delete my test files:
[18:34 r730-01 dvl ~] % zfs list data02 NAME USED AVAIL REFER MOUNTPOINT data02 744G 155G 96K none
It says 155G available.
[18:37 cliff2 dvl ~] % sudo rm -rf /tmp/delete-me* [18:37 cliff2 dvl ~] %
And now we have:
[18:37 r730-01 dvl ~] % zfs list data02 NAME USED AVAIL REFER MOUNTPOINT data02 869G 30.0G 96K none
Pretty much the same. Guess why?
Snapshots.
Or in this case, one snapshot:
[18:39 r730-01 dvl ~] % zfs list -r -t snapshot data02/jails/cliff2 | tail -4 data02/jails/cliff2@autosnap_2026-02-19_18:00:06_hourly 0B - 2.46G - data02/jails/cliff2@autosnap_2026-02-19_18:00:06_frequently 0B - 2.46G - data02/jails/cliff2@autosnap_2026-02-19_18:15:30_frequently 376K - 2.46G - data02/jails/cliff2@autosnap_2026-02-19_18:30:02_frequently 148G - 150G -
Let’s sacrifice that one:
[18:39 r730-01 dvl ~] % sudo zfs destroy data02/jails/cliff2@autosnap_2026-02-19_18:30:02_frequently [18:40 r730-01 dvl ~] % [18:40 r730-01 dvl ~] % zfs list data02 NAME USED AVAIL REFER MOUNTPOINT data02 721G 178G 96K none [18:40 r730-01 dvl ~] % zpool list data02 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data02 928G 591G 337G - - 52% 63% 1.00x ONLINE -
There, that’s better.
Overview
My monitoring didn’t account for the reservation. My zpool would (and did) fill up before triggering any monitoring.
I adjusted my reservation size based on existing monitoring.
If I recall correctly, the post I read about reservation was based on: if the zpool fills up, this saves you some space, so you notice the problem, you solve the problem, thereby allowing the system to continue running. The theory being that the reservation is just stopping some rogue process/jail from completely filling up the zpool – it’s easier to manipulate with a mostly full zpool than a 100% zpool.
Yes, you could set a quota to restrict certain jails from exploding, but that seems like more like a user-based tool (don’t let user foo use more than 100GB in their home dir).
Reading elsewhere, I think the primary use of reservations is to reserve future space not yet used. Or “A ZFS reservation is an allocation of disk space from the pool that is guaranteed to be available to a dataset”. Or in other words, always ensure that user foo has 150GB available from the pool).
In short: I’m not convinced reservation or quota is useful to me in this situation. However, I’ll keep the reservation on data02/reserved for now.
Thanks.
