using zfs rollback for cache clearing

2 Comments / iocage, Jails, ZFS / By

I’m in the final stages of the FreshPorts packages project. One of the last tasks is clearing the packages cache from disk when new package information is loaded into the database.

See also:

Several of the configuration items have been learned from putting my poudriere instance into a jail.

In this post:

  1. FreeBSD 12.1
  2. py37-iocage-1.2_5

If I was to do this cache clearing by hand, it would be:

rm -rf /var/db/freshports/cache/packages/*

After a twitter thread with great technical information, Allan Jude suggested using zfs rollback to clear the cache.

This post will document how I implemented that great idea. This really isn’t worthy of a blog post, except that it deals with jailing a ZFS filesystem and delegating permissions to a non-root user.

All for one, one for all

I created three ZFS filesystems.

NOTE: if using iocage, this must be in the same zpool as the iocage jails.

[dan@slocum:~] $ sudo zfs create -o canmount=off -o mountpoint=none system/data/freshports-cache-packages
[dan@slocum:~] $ sudo zfs create system/data/freshports-cache-packages/dev-nginx01
[dan@slocum:~] $ sudo zfs create system/data/freshports-cache-packages/test-nginx01
[dan@slocum:~] $ sudo zfs create system/data/freshports-cache-packages/stage-nginx01
[dan@slocum:~] $ zfs list -r system/data/freshports-cache-packagese-packages
NAME                                                  USED  AVAIL  REFER  MOUNTPOINT
system/data/freshports-cache-packages                 703K  13.5T   176K  none
system/data/freshports-cache-packages/dev-nginx01     176K  13.5T   176K  none
system/data/freshports-cache-packages/stage-nginx01   176K  13.5T   176K  none
system/data/freshports-cache-packages/test-nginx01    176K  13.5T   176K  none
[dan@slocum:~] $

That’s one for each of the dev, test, and stage webservers for FreshPorts. For the rest of this post, I’ll talk only about system/data/freshports-cache-packages/dev-nginx01 , unless otherwise stated.

If you don’t use the same zpool

If you don’t use the same zpool, you’ll get this error:

[dan@slocum:~] $ sudo iocage do iocage start dev-nginx01
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_start.py", line 314, in __start_jail__
    stdout=su.PIPE, stderr=su.PIPE)
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['zfs', 'get', '-H', 'creation', 'system/data/freshports-cache-packages/dev-nginx01']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/iocage_cli/start.py", line 60, in cli
    ioc.IOCage(jail=jail, rc=rc).start(ignore_exception=ignore)
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/iocage.py", line 1907, in start
    used_ports=used_ports,
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_start.py", line 87, in __init__
    raise e
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_start.py", line 84, in __init__
    self.__start_jail__()
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_start.py", line 321, in __start_jail__
    stderr=su.STDOUT)
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_common.py", line 612, in checkoutput
    out = su.check_output(*args, **kwargs)
  File "/usr/local/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/usr/local/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['zfs', 'create', '-o', 'compression=lz4', '-o', 'mountpoint=none', 'system/data/freshports-cache-packages/dev-nginx01']' returned non-zero exit status 1.

Jail that filesystem

The first step was:

sudo zfs set jailed=on system/data/freshports-cache-packages/dev-nginx01

This setting allows the filesystem to be controlled from within a jail.

Associate that filesystem with the jail

Tell iocage to mount this filesystem within the jail. The jail must be stopped to do this.

NOTE: the jail_zfs_dataset parameter is NOT THE FULL filesystem name. You drop the zpool name at the start.

See below for a shorter command which should also work.

sudo iocage set jail_zfs_dataset=data/freshports-cache-packages/dev-nginx01 jail_zfs=1 enforce_statfs=1 allow_mount_zfs=true dev-nginx01

I will outline those various settings. They are all iocage-specific but have similar corresponding FreeBSD jail configuration settings.

  1. jail_zfs_dataset: Set the dataset to be jailed and fully handed over to a jail
  2. jail_zfs: Enable automatic ZFS jailing inside the jail (see man iocage)
  3. enforce_statfs: sets sysctl security.jail.enforce_statfs=1 within the jail
  4. allow_mount_zfs: sets sysctl security.jail.mount_zfs_allowed=1 within the jail

OF NOTE: jail_zfs=1 will set enforce_statfs=1 and allow_mount_zfs=1 – see man iocage.

Thus, the above command could be shortened to:

sudo iocage set jail_zfs=1 jail_zfs_dataset=data/freshports-cache-packages/dev-nginx01 dev-nginx01

I moved jail_zfs=1 to the front of the command only because jail_zfs_dataset requires it and that placement appeals to my sense of order.

Start the jail

One you start the jail, you’ll see this from within the jail:

[dan@dev-nginx01:~] $ zfs list
NAME                                                USED  AVAIL  REFER  MOUNTPOINT
system                                             4.06T  13.5T   288K  /
system/data                                         465G  13.5T   895K  /data
system/data/freshports-cache-packages               703K  13.5T   176K  none
system/data/freshports-cache-packages/dev-nginx01   176K  13.5T   176K  none
[dan@dev-nginx01:~] $

Once I got the jail running, these are the parameters in use. it has been reformatted to fit in 80 columns. Some data is redacted.

[dan@slocum:~] $ jls -n -j 34
12345678901234567890123456789012345678901234567890123456789012345678901234567890
devfs_ruleset=1018 nodying enforce_statfs=1 host=new ip4=disable ip6=disable
jid=34 name=ioc-dev-nginx01 osreldate=1201000 osrelease=12.1-RELEASE-p3 
parent=0 path=/iocage/jails/dev-nginx01/root persist securelevel=2 sysvmsg=new 
sysvsem=new sysvshm=new vnet=inherit allow.nochflags allow.nomlock allow.mount 
allow.mount.nodevfs allow.mount.nofdescfs allow.mount.nonullfs
allow.mount.noprocfs allow.mount.zfs allow.noquotas allow.noraw_sockets 
allow.noread_msgbuf allow.reserved_ports allow.set_hostname allow.nosocket_af 
allow.nosysvipc children.cur=0 children.max=0 cpuset.id=33 host.domainname=none 
host.hostid=0 host.hostname=dev-nginx01.[redacted] host.hostuuid=dev-nginx01 
ip4.addr=[redacted] ip4.saddrsel ip6.addr= ip6.saddrsel
[dan@slocum:~] $

That may be useful in tracking down the problem you are having.

Set the mountpoint within the jail

Within the jail, I set the mountpoint to a temporary location, near the intended mountpoint.

sudo zfs set mountpoint=/var/db/freshports/cache/packages.new system/data/freshports-cache-packages/dev-nginx01

I also had to set owner, group, and permissions. See the next section.

Permissions and the snapshot

You want to do this chown and chmod before you do the snapshot, otherwise they will be undone when you do your first rollback. Don’t be like me and have to redo it later.

$ sudo chown www:freshports packages
$ sudo chmod 0775 packages
$ ls -ld packages 
drwxrwxr-x  2 www  freshports  2 Apr 29 14:38 packages

This snapshot is how we rollback and clear the cache.

sudo zfs snapshot system/data/freshports-cache-packages/dev-nginx01@empty

Copy over existing data

For fun and games, I wanted to copy over the existing data. This isn’t the best approach in a busy production environment, but it’s good enough for this place.


[dan@dev-nginx01:/var/db/freshports/cache] $ sudo mv packages/* packages.new/
[dan@dev-nginx01:/var/db/freshports/cache] $ sudo rm -rf packages 
[dan@dev-nginx01:/var/db/freshports/cache] $ sudo zfs set mountpoint=/var/db/freshports/cache/packages system/data/freshports-cache-packages/dev-nginx01
[dan@dev-nginx01:/var/db/freshports/cache] $ sudo rmdir packages.new 
[dan@dev-nginx01:/var/db/freshports/cache] $

Test the rollback

Let’s try a rollback.

[dan@dev-nginx01:/var/db/freshports/cache] $ ls packages/
astro       databases   devel       emulators   graphics    math        multimedia  net-mgmt    x11         x11-servers
chinese     deskutils   dns         games       lang        misc        net         sysutils    x11-fonts   x11-wm
[dan@dev-nginx01:/var/db/freshports/cache] $ sudo zfs rollback system/data/freshports-cache-packages/dev-nginx01@empty
[dan@dev-nginx01:/var/db/freshports/cache] $ ls packages/
[dan@dev-nginx01:/var/db/freshports/cache] $

Empty!

Delegating to non-root

I want the freshports user to be able to rollback this filesystem. I use the zfs allow command

$ sudo zfs allow freshports rollback system/data/freshports-cache-packages/dev-nginx01

Testing the delegation

This failed. I had thought about this issue before writing this post, but forgot to deal with it first:

$ echo zfs rollback system/data/freshports-cache-packages/dev-nginx01@empty | sudo su -fm freshports
cannot rollback to 'system/data/freshports-cache-packages/dev-nginx01@empty': more recent snapshots or bookmarks exist
use '-r' to force deletion of the following snapshots and bookmarks:
system/data/freshports-cache-packages/dev-nginx01@autosnap_2020-04-29_15:20:09_frequently
system/data/freshports-cache-packages/dev-nginx01@autosnap_2020-04-29_15:00:09_frequently
system/data/freshports-cache-packages/dev-nginx01@autosnap_2020-04-29_15:00:09_hourly
[dan@dev-nginx01:/var/db/freshports/cache] $ ls -l packages
total 1
drwxrwxr--  2 www  freshports  3 Apr 29 15:21 editors
[dan@dev-nginx01:/var/db/freshports/cache] $

My sanoid configuration is triggering snapshots on my dataset. If you have automatic generation of snapshots, you’d want to disable them if you want to rollback with a -r. Since this is a cache, there is no reason for snapshots.

My sanoid.conf configuration change consisted of adding this:

[system/data/freshports-cache-packages]
    use_template = ignore
    recursive    = yes

Where ignore was:

[template_ignore]
    autoprune = no
    autosnap  = no
    monitor   = no

In the meantime, I manually destroyed those snapshots and tried again:

$ echo zfs rollback system/data/freshports-cache-packages/dev-nginx01@empty | sudo su -fm freshports
$ ls -l packages
total 0
$

Success!

Items for discussion

I would prefer the filesystem name to be the same in each host but I see how that can’t happen. The way it is implemented now, I could use `hostname -s` and get the correct name:

$ zfs list system/data/freshports-cache-packages/`hostname -s`
NAME                                                USED  AVAIL  REFER  MOUNTPOINT
system/data/freshports-cache-packages/dev-nginx01   176K  13.5T   176K  /var/db/freshports/cache/packages

Thanks Allan.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

2 thoughts on “using zfs rollback for cache clearing”

Leave a Comment

Scroll to Top