I’m going to implement zfstools on all my ZFS-based hosts today. I first started using this tool in July 2019.
In this post:
- FreeBSD 12.0 and 12.1
- zfstools 0.3.6_1
- sanoid-2.0.1_2
Local snapshots only
I will be using zfstool only for creating local snapshots. If I wanted snapshots for sending to other hosts, I would probably use a tool such as sysutils/sanoid, which is policy-driven solution for snapshot management and replication.
For now, there is no replication in my plans.
But let’s look at sanoid first
Hmm, as I type this, I want to check out sanoid.
Let’s build it and install it on a server, and see if it does the one thing I’m afraid I’ll miss from zfstools: deletion of zero-size snapshots.
The test sandoid configuration
My test configuration looks like this:
[system/usr/home]
use_template = production
[template_production]
frequent_period = 1
frequently = 5
hourly = 36
daily = 30
monthly = 3
yearly = 0
autosnap = yes
autoprune = yes
This is just for testing. That’s 5 frequent snapshot, every minute or so.
When setting values, it is useful to know the defaults.
Let’s run it.
Some test runs
For these tests, I ran this from the command line:
$ sudo /usr/local/bin/sanoid --cron $ zfs list -r -t snapshot system/usr/home NAME USED AVAIL REFER MOUNTPOINT system/usr/home@autosnap_2019-11-11_18:20:17_daily 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:20:17_monthly 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:20:17_frequently 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:20:17_hourly 0 - 22.3G -
That seems to make sense. One of everything.
Then I decided to delete all of them:
$ sudo zfs destroy system/usr/home@autosnap_2019-11-11_18:20:17_daily $ sudo zfs destroy system/usr/home@autosnap_2019-11-11_18:20:17_monthly $ sudo zfs destroy system/usr/home@autosnap_2019-11-11_18:20:17_frequently $ sudo zfs destroy system/usr/home@autosnap_2019-11-11_18:20:17_hourly
Run it again:
$ sudo /usr/local/bin/sanoid --cron $ zfs list -r -t snapshot system/usr/home NAME USED AVAIL REFER MOUNTPOINT system/usr/home@autosnap_2019-11-11_18:25:12_frequently 0 - 22.3G -
OK, just one. Why?
One more time.
$ sudo /usr/local/bin/sanoid --cron $ zfs list -r -t snapshot system/usr/home NAME USED AVAIL REFER MOUNTPOINT system/usr/home@autosnap_2019-11-11_18:25:12_frequently 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_daily 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_hourly 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_frequently 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_monthly 0 - 22.3G -
That seems a little better. Let’s try another.
$ sudo /usr/local/bin/sanoid --cron $ zfs list -r -t snapshot system/usr/home NAME USED AVAIL REFER MOUNTPOINT system/usr/home@autosnap_2019-11-11_18:25:12_frequently 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_daily 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_hourly 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_frequently 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_monthly 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:31:18_frequently 0 - 22.3G -
Well, one more frequently, of zero size.
Enough testing
After lunch, a bit of tv, and some more test runs. I have this:
$ sudo /usr/local/bin/sanoid --cron $ zfs list -r -t snapshot system/usr/home NAME USED AVAIL REFER MOUNTPOINT system/usr/home@autosnap_2019-11-11_18:27:32_daily 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_hourly 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_frequently 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:27:32_monthly 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:31:18_frequently 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:32:41_frequently 0 - 22.3G - system/usr/home@autosnap_2019-11-11_18:50:38_frequently 0 - 22.3G - system/usr/home@autosnap_2019-11-11_19:00:40_hourly 0 - 22.3G - system/usr/home@autosnap_2019-11-11_19:00:40_frequently 0 - 22.3G - system/usr/home@autosnap_2019-11-11_19:22:22_frequently 0 - 22.3G -
This seems to work as advertised.
The running configuration
This is the configuration I am using now:
[dan@tallboy:/usr/home] $ cat /usr/local/etc/sanoid/sanoid.conf [system] use_template = production recursive = yes [template_production] frequently = 5 hourly = 48 daily = 90 monthly = 6 yearly = 2 autosnap = yes autoprune = yes [template_ignore] autoprune = no autosnap = no monitor = no [dan@tallboy:/usr/home] $
Monitoring
sanoid is the first tool I have enountered with some with built-in monitoring for Nagios.
$ /usr/local/bin/sanoid --monitor-capacity
OK ZPOOL system : 30%
$ /usr/local/bin/sanoid --monitor-health
OK ZPOOL system : ONLINE {Size:452G Free:315G Cap:30%}
$ /usr/local/bin/sanoid --monitor-snapshots
OK: all monitored datasets (system, system/bootenv, system/bootenv/11.1-RELEASE, system/bootenv/default, system/iocage, system/iocage/.defaults, system/iocage/base, system/iocage/base/10.2-RELEASE, system/iocage/base/10.2-RELEASE/root, system/iocage/base/10.2-RELEASE/root/bin, system/iocage/base/10.2-RELEASE/root/boot, system/iocage/base/10.2-RELEASE/root/lib, system/iocage/base/10.2-RELEASE/root/libexec, system/iocage/base/10.2-RELEASE/root/rescue, system/iocage/base/10.2-RELEASE/root/sbin, system/iocage/base/10.2-RELEASE/root/usr, system/iocage/base/10.2-RELEASE/root/usr/bin, system/iocage/base/10.2-RELEASE/root/usr/include, system/iocage/base/10.2-RELEASE/root/usr/lib, system/iocage/base/10.2-RELEASE/root/usr/lib32, system/iocage/base/10.2-RELEASE/root/usr/libdata, system/iocage/base/10.2-RELEASE/root/usr/libexec, system/iocage/base/10.2-RELEASE/root/usr/sbin, system/iocage/base/10.2-RELEASE/root/usr/share, system/iocage/base/10.2-RELEASE/root/usr/src, system/iocage/base/10.3-RELEASE, system/iocage/base/10.3-RELEASE/root, system/iocage/base/10.3-RELEASE/root/bin, system/iocage/base/10.3-RELEASE/root/boot, system/iocage/base/10.3-RELEASE/root/lib, system/iocage/base/10.3-RELEASE/root/libexec, system/iocage/base/10.3-RELEASE/root/rescue, system/iocage/base/10.3-RELEASE/root/sbin, system/iocage/base/10.3-RELEASE/root/usr, system/iocage/base/10.3-RELEASE/root/usr/bin, system/iocage/base/10.3-RELEASE/root/usr/include, system/iocage/base/10.3-RELEASE/root/usr/lib, system/iocage/base/10.3-RELEASE/root/usr/lib32, system/iocage/base/10.3-RELEASE/root/usr/libdata, system/iocage/base/10.3-RELEASE/root/usr/libexec, system/iocage/base/10.3-RELEASE/root/usr/sbin, system/iocage/base/10.3-RELEASE/root/usr/share, system/iocage/base/10.3-RELEASE/root/usr/src, system/iocage/download, system/iocage/download/10.1-RELEASE, system/iocage/download/10.2-RELEASE, system/iocage/download/10.3-RELEASE, system/iocage/download/11.1-RELEASE, system/iocage/images, system/iocage/jails, system/iocage/jails/clavin2, system/iocage/jails/clavin2/root, system/iocage/jails/mailjail2, system/iocage/jails/mailjail2/root, system/iocage/jails/mx-ingress02.freshports.org, system/iocage/jails/mx-ingress02.freshports.org/root, system/iocage/jails/ns1, system/iocage/jails/ns1/root, system/iocage/jails/ns2.unixathome.org, system/iocage/jails/ns2.unixathome.org/root, system/iocage/jails/tallboy-mqtt, system/iocage/jails/tallboy-mqtt/root, system/iocage/jails/tallboy-mqtt/root.cloned.DELETE.ME.after.2020.01.01, system/iocage/jails/wikis, system/iocage/jails/wikis/root, system/iocage/log, system/iocage/releases, system/iocage/releases/10.1-RELEASE, system/iocage/releases/10.1-RELEASE/root, system/iocage/releases/10.2-RELEASE, system/iocage/releases/10.2-RELEASE/root, system/iocage/releases/10.3-RELEASE, system/iocage/releases/10.3-RELEASE/root, system/iocage/releases/11.1-RELEASE, system/iocage/templates, system/jails.DELETEME.AFTER.2020.01.01, system/jails.DELETEME.AFTER.2020.01.01/releases, system/tmp, system/usr, system/usr/home, system/usr/local, system/usr/obj, system/usr/ports, system/usr/src, system/var, system/var/crash, system/var/db, system/var/db/pkg, system/var/empty, system/var/log, system/var/mail, system/var/run, system/var/tmp) have fresh snapshots
[dan@tallboy:/usr/home] $
All tools should be following this example.
Running from cron
If I was going to implement sanoid, I’d follow on from the above with this crontab entry at /usr/local/etc/cron.d/sanoid:
# mail any output to `dan', no matter whose crontab this is MAILTO=dan@example.org PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin */5 * * * * root /usr/local/bin/sanoid --cron
NOTE 1:
- After doing the above and running into some concurrency issues (see https://github.com/jimsalterjrs/sanoid/issues/459). I moved to this approach:
*/5 * * * * root /usr/bin/lockf -t 0 /tmp/.sanoid-cron /usr/local/bin/sanoid --cron
See man lockf for details.
NOTE 2:
- After getting emails from the cron job saying lockf: /tmp/.sanoid-cron: already locked, I started with this approach:
*/5 * * * * root /usr/bin/lockf -t 0 /tmp/.sanoid-cron-snapshot /usr/local/bin/sanoid --take-snapshots */5 * * * * root /usr/bin/lockf -t 0 /tmp/.sanoid-cron-prune /usr/local/bin/sanoid --prune-snapshots
See https://github.com/jimsalterjrs/sanoid/issues/496 for my query on this approach. It turns out to be the recommended strategy.
Why into /usr/local/etc/cron.d/sanoid?
- Simplicity.
- When upgrading the host, I don’t have to merge anything with respect to /etc/crontab.
- man 8 cron says that directory is for third-party package provided crontab files.
zfstools
I used zfstools on r710-01. Ironically, that host is powered off now. I pulled up the crontab for that host via Bacula. It contained this:
[dan@r720-01:~] $ sudo cat /tmp/bacula-restores/usr/local/etc/cron.d/zfstools PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin 15,30,45 * * * * root /usr/local/sbin/zfs-auto-snapshot frequent 4 0 * * * * root /usr/local/sbin/zfs-auto-snapshot hourly 24 7 0 * * * root /usr/local/sbin/zfs-auto-snapshot daily 7 14 0 * * 7 root /usr/local/sbin/zfs-auto-snapshot weekly 4 28 0 1 * * root /usr/local/sbin/zfs-auto-snapshot monthly 12 [dan@r720-01:~] $
As I said earlier, one feature of zfstools I would like to see on other applications is the auto-deletion of zero-size snapshots. For example, in the above hourly snapshots, the 24 snapshots might be spread over a longer than 24 hour period if data is not written in every hour.
For example, in the following, you can see that my 24 hourly snapshots start on Aug 8 and run through to Sep 26. This is a dataset which does not update on a frequent basis. I argue that this approach is more useful to me than 24 zero-sized snapshots.
tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-08-31-06h00 13.8M - 9.80G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-01-13h00 1.91M - 9.79G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-01-14h00 1.93M - 9.74G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-02-18h00 53.3M - 9.78G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-02-19h00 53.3M - 9.78G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-03-05h00 63.3M - 9.79G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-05-14h00 2.08M - 9.69G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-05-19h00 1.94M - 9.58G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-05-20h00 2.58M - 9.58G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-05-21h00 1.94M - 9.58G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-06-15h00 1.96M - 9.58G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-06-16h00 468K - 9.65G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-06-18h00 468K - 9.65G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-07-04h00 4.18M - 9.50G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-08-04h00 53.4M - 9.50G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-15-05h00 3.28M - 9.52G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-17-05h00 42.2M - 9.81G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-19-05h00 648K - 9.70G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-19-11h00 296K - 9.70G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-19-13h00 2.05M - 9.70G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-20-11h00 2.08M - 9.70G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-20-13h00 2.06M - 9.70G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-22-11h00 240K - 9.73G - tank_fast/poudriere/data/packages@zfs-auto-snap_hourly-2019-09-26-14h00 0 - 9.73G -
This is very impressive feature and I’d like to see it as an option in all such tools.
