ZFS is very flexible about mountpoints, and there are many features available to provide great flexibility.
When you create your second zpool this is what it might look like:
$$ zfs list -r main_tank NAME USED AVAIL REFER MOUNTPOINT main_tank 893G 3.52T 96K /main_tank main_tank/data 786G 3.52T 88K /main_tank/data main_tank/data/dvl 755G 3.52T 755G /main_tank/data/dvl main_tank/data/freshports 31.4G 3.52T 88K /main_tank/data/freshports main_tank/data/freshports/backend 3.11G 3.52T 88K /main_tank/data/freshports/backend
This is a pool I created long ago, but it will be a decent example. When you create zpool main_tank, the default mountpoint is /main_tank.
You might be happy with that, but you don’t have to be content. You can do magical things.
Skip to the bottom to see more.
Some highlights are:
- mount point can be inherited
- not all filesystems in a zpool need to be mounted
- each filesystem (directory) can have different ZFS characteristics
In my case, let’s look at this new zpool I created earlier today and I will show you some very simple alternatives. This zpool use NVMe devices which should be faster than SSDs especially when used with multiple concurrent writes. This is my plan: run all the Bacula regression tests concurrently.
My use case: storage area for bacula-regression tests in a one jail. There are 7 such jails.
These are the dictories within the mysql57 jail which get the most use during regression testing:
- /var/db/mysql
- /usr/home/dan/src/bacula
My goal: have as few mount directives as possible, use inheritance, and make it just mount into the right places in the jail.
How much space is this?
[dan@mysql57:~/src/bacula] $ sudo du -d1 -ch /var/db/mysql /usr/home/dan/src/bacula 1.6M /var/db/mysql/mysql 901K /var/db/mysql/performance_schema 560K /var/db/mysql/regress 1.0M /var/db/mysql/sys 579M /var/db/mysql 147M /usr/home/dan/src/bacula/regress 33M /usr/home/dan/src/bacula/gui 21M /usr/home/dan/src/bacula/bacula 81M /usr/home/dan/src/bacula/.git 283M /usr/home/dan/src/bacula 862M total [dan@mysql57:~/src/bacula] $
Less than 1GB, good. Let’s proceed.
My new zpool, already created is named nvd, and it mounted at /nvd.
[dan@knew:~] $ ls -ld /nvd drwxr-xr-x 2 root wheel 2 Apr 21 13:26 /nvd
Let’s not mount that. This could have been specified during zpool create via a -o canmount=off parameter.
[dan@knew:~] $ sudo zfs set canmount=off nvd [dan@knew:~] $ ls -ld /nvd ls: /nvd: No such file or directory [dan@knew:~] $
Let’s turn compression off. Again, this could have been a parameter.
[dan@knew:~] $ sudo zfs set compression=lz4 nvd [dan@knew:~] $
These are the various plalces I will mounting datasystems from this zpool:
[dan@knew:~] $ zfs list | egrep -e '(mysql|pg)' | grep -v root system/iocage/jails/mysql56 9.30G 29.0T 210K /iocage/jails/mysql56 system/iocage/jails/mysql57 9.26G 29.0T 210K /iocage/jails/mysql57 system/iocage/jails/mysql80 2.53G 29.0T 210K /iocage/jails/mysql80 system/iocage/jails/pg10 8.03G 29.0T 210K /iocage/jails/pg10 system/iocage/jails/pg11 8.03G 29.0T 210K /iocage/jails/pg11 system/iocage/jails/pg94 7.52G 29.0T 210K /iocage/jails/pg94 system/iocage/jails/pg95 6.78G 29.0T 210K /iocage/jails/pg95 system/iocage/jails/pg96 6.51G 29.0T 210K /iocage/jails/pg96
Let’s start by setting the mountpoint, but remember it will not be mounted because we specified canmount=off:
[dan@knew:~] $ sudo zfs set mountpoint=/iocage/jails nvd [dan@knew:~] $
We are going to be messing about with data, and while you might be able to do this while the jail is running, I am not.
[dan@knew:~] $ sudo iocage stop mysql56
* Stopping mysql56
+ Executing prestop OK
+ Stopping services OK
+ Removing devfs_ruleset: 11 OK
+ Removing jail process OK
+ Executing poststop OK
[dan@knew:~] $
What data do we have in the MySQL directory? Let’s save that away into a temp directory:
$ cd /iocage/jails/mysql56/root/var/db/ $ sudo mkdir mysql-tmp $ sudo sudo zfs create -p nvd/mysql56/root/var/db/ cannot create 'nvd/mysql56/root/var/db/': trailing slash in name
Now let’s create the directory structure we need:
$ sudo sudo zfs create -p nvd/mysql56/root/var/db $ sudo sudo zfs create -p nvd/mysql56/root/var/db/mysql $ sudo sudo zfs create -p nvd/mysql56/root/var/db/mysql_secure $ sudo sudo zfs create -p nvd/mysql56/root/var/db/mysql_tmpdir
$ sudo zfs set canmount=on nvd/mysql56/root/var/db/mysql $ sudo zfs set canmount=on nvd/mysql56/root/var/db/mysql_secure $ sudo zfs set canmount=on nvd/mysql56/root/var/db/mysql_tmpdir
What do we see?
$ ls -l total 17706 drwx------ 2 operator operator 10 Aug 23 2015 entropy drwxr-xr-x 3 root wheel 4 Mar 18 21:05 etcupdate drwx------ 4 root wheel 11 Mar 18 21:05 freebsd-update drwx------ 2 root wheel 2 Jun 22 2018 hyperv drwx------ 2 root wheel 2 Dec 4 2012 ipf -r--r--r-- 1 nobody wheel 247659 Sep 2 2017 locate.database drwxr-xr-x 2 logcheck logcheck 5 Aug 18 2015 logcheck -rw------- 1 root wheel 28808 Aug 5 2018 mergemaster.mtree drwxr-xr-x 5 root wheel 5 Apr 21 20:30 mysql-tmp drwx------ 2 root wheel 2 Jun 22 2018 ntp -rw-r--r-- 1 root wheel 10663 Oct 27 12:10 ntpd.leap-seconds.list drwxr-xr-x 2 root wheel 10 Apr 21 04:13 pkg drwxr-xr-x 2 root wheel 2 Dec 4 2012 ports drwxr-xr-x 2 root wheel 2 Dec 4 2012 portsnap drwx------ 2 postfix wheel 3 May 16 2016 postfix -rw-r--r-- 1 root wheel 2097920 Mar 18 21:05 services.db drwx--x--x 3 root wheel 3 Mar 18 2016 sudo drwxr-xr-x 3 root wheel 3 Jun 22 2018 zfsd
Nothing. Where is the stuff we created?
[dan@knew:/iocage/jails/mysql56/root/var/db] $ zfs get -r canmount nvd NAME PROPERTY VALUE SOURCE nvd canmount off local nvd/mysql56 canmount on default nvd/mysql56/root canmount on default nvd/mysql56/root/var canmount on default nvd/mysql56/root/var/db canmount on default nvd/mysql56/root/var/db/mysql canmount on local nvd/mysql56/root/var/db/mysql_secure canmount on local nvd/mysql56/root/var/db/mysql_tmpdir canmount on local
Look at all that canmount. Let’s fix that.
$ sudo zfs set canmount=off nvd/mysql56 cannot unmount '/iocage/jails/mysql56/root/var': Device busy
Oh, it’s mounted:
sudo zfs umount nvd/mysql56/root/var/db/mysql_tmpdir sudo zfs umount nvd/mysql56/root/var/db sudo zfs umount nvd/mysql56/root/var sudo zfs umount nvd/mysql56/root sudo zfs umount nvd/mysql56
Turn that stuff off:
$ sudo zfs set canmount=off nvd/mysql56 $ sudo zfs set canmount=off nvd/mysql56/root $ sudo zfs set canmount=off nvd/mysql56/root/var $ sudo zfs set canmount=off nvd/mysql56/root/var/db
There, this looks better:
$ zfs get -r canmount nvd NAME PROPERTY VALUE SOURCE nvd canmount off local nvd/mysql56 canmount off local nvd/mysql56/root canmount off local nvd/mysql56/root/var canmount off local nvd/mysql56/root/var/db canmount off local nvd/mysql56/root/var/db/mysql canmount on local nvd/mysql56/root/var/db/mysql_secure canmount on local nvd/mysql56/root/var/db/mysql_tmpdir canmount on local
Now let’s mount and see what we have:
$ sudo zfs mount nvd/mysql56/root/var/db/mysql $ sudo zfs mount nvd/mysql56/root/var/db/mysql_secure $ sudo zfs mount nvd/mysql56/root/var/db/mysql_tmpdir $ ls -l total 17708 drwx------ 2 operator operator 10 Aug 23 2015 entropy drwxr-xr-x 3 root wheel 4 Mar 18 21:05 etcupdate drwx------ 4 root wheel 11 Mar 18 21:05 freebsd-update drwx------ 2 root wheel 2 Jun 22 2018 hyperv drwx------ 2 root wheel 2 Dec 4 2012 ipf -r--r--r-- 1 nobody wheel 247659 Sep 2 2017 locate.database drwxr-xr-x 2 logcheck logcheck 5 Aug 18 2015 logcheck -rw------- 1 root wheel 28808 Aug 5 2018 mergemaster.mtree drwxr-xr-x 2 root wheel 2 Apr 21 20:32 mysql drwxr-xr-x 5 root wheel 5 Apr 21 20:30 mysql-tmp drwxr-xr-x 2 root wheel 2 Apr 21 20:32 mysql_secure drwxr-xr-x 2 root wheel 2 Apr 21 20:32 mysql_tmpdir drwx------ 2 root wheel 2 Jun 22 2018 ntp -rw-r--r-- 1 root wheel 10663 Oct 27 12:10 ntpd.leap-seconds.list drwxr-xr-x 2 root wheel 10 Apr 21 04:13 pkg drwxr-xr-x 2 root wheel 2 Dec 4 2012 ports drwxr-xr-x 2 root wheel 2 Dec 4 2012 portsnap drwx------ 2 postfix wheel 3 May 16 2016 postfix -rw-r--r-- 1 root wheel 2097920 Mar 18 21:05 services.db drwx--x--x 3 root wheel 3 Mar 18 2016 sudo drwxr-xr-x 3 root wheel 3 Jun 22 2018 zfsd [dan@knew:/iocage/jails/mysql56/root/var/db] $
There, we have the directores (mysql, mysql-tmp, and mysql_secure) appearing.
Those permissions need fixing though:
$ sudo sudo chown -R mysql:mysql mysql mysql_secure mysql_tmpdir $ ls -ld mysql mysql_secure mysql_tmpdir drwxr-xr-x 2 mysql mysql 2 Apr 21 20:32 mysql drwxr-xr-x 2 mysql mysql 2 Apr 21 20:32 mysql_secure drwxr-xr-x 2 mysql mysql 2 Apr 21 20:32 mysql_tmpdir
Oh.. atime I forgot about that.
$ zfs get -r atime nvd NAME PROPERTY VALUE SOURCE nvd atime on default nvd/mysql56 atime on default nvd/mysql56/root atime on default nvd/mysql56/root/var atime on default nvd/mysql56/root/var/db atime on default nvd/mysql56/root/var/db/mysql atime on default nvd/mysql56/root/var/db/mysql_secure atime on default nvd/mysql56/root/var/db/mysql_tmpdir atime on default
If we turn off atime we reduce the number of writes because we are not updating the atime of the files with every read.
$ sudo zfs set atime=off nvd $ zfs get -r atime nvd NAME PROPERTY VALUE SOURCE nvd atime off local nvd/mysql56 atime off inherited from nvd nvd/mysql56/root atime off inherited from nvd nvd/mysql56/root/var atime off inherited from nvd nvd/mysql56/root/var/db atime off inherited from nvd nvd/mysql56/root/var/db/mysql atime off inherited from nvd nvd/mysql56/root/var/db/mysql_secure atime off inherited from nvd nvd/mysql56/root/var/db/mysql_tmpdir atime off inherited from nvd $
Now I copy over the data:
$ sudo cp -a mysql-tmp/mysql/* mysql/ $ sudo cp -a mysql-tmp/mysql_secure/* mysql_secure/ cp: mysql-tmp/mysql_secure/*: No such file or directory
Nothing to copy? are you sure?
$ sudo find mysql-tmp/mysql_secure mysql-tmp/mysql_secure
Yes, I’m sure.
$ sudo cp -a mysql-tmp/mysql_tmpdir/* mysql_tmpdir/ cp: mysql-tmp/mysql_tmpdir/*: No such file or directory $ sudo find mysql-tmp/mysql_tmpdir/ mysql-tmp/mysql_tmpdir/
$ zpool list nvd NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT nvd 222G 1.34G 221G - - 0% 0% 1.00x ONLINE -
So far, this has used about 1.34G
Now let’s set up the bacula regression code directory.
$ sudo sudo zfs create -p -o canmount=off nvd/mysql56/root/usr/home/dan/src/bacula $ zfs get -r canmount nvd NAME PROPERTY VALUE SOURCE nvd canmount off local nvd/mysql56 canmount off local nvd/mysql56/root canmount off local nvd/mysql56/root/usr canmount on default nvd/mysql56/root/usr/home canmount on default nvd/mysql56/root/usr/home/dan canmount on default nvd/mysql56/root/usr/home/dan/src canmount on default nvd/mysql56/root/usr/home/dan/src/bacula canmount off local nvd/mysql56/root/var canmount off local nvd/mysql56/root/var/db canmount off local nvd/mysql56/root/var/db/mysql canmount on local nvd/mysql56/root/var/db/mysql_secure canmount on local nvd/mysql56/root/var/db/mysql_tmpdir canmount on local $
I failed. I should have supplied more -o parameters.
$ sudo zfs set canmount=off nvd/mysql56/root/usr/home/dan/src $ sudo zfs set canmount=off nvd/mysql56/root/usr/home/dan $ sudo zfs set canmount=off nvd/mysql56/root/usr/home $ sudo zfs set canmount=off nvd/mysql56/root/usr $ zfs get -r canmount nvd NAME PROPERTY VALUE SOURCE nvd canmount off local nvd/mysql56 canmount off local nvd/mysql56/root canmount off local nvd/mysql56/root/usr canmount off local nvd/mysql56/root/usr/home canmount off local nvd/mysql56/root/usr/home/dan canmount off local nvd/mysql56/root/usr/home/dan/src canmount off local nvd/mysql56/root/usr/home/dan/src/bacula canmount off local nvd/mysql56/root/var canmount off local nvd/mysql56/root/var/db canmount off local nvd/mysql56/root/var/db/mysql canmount on local nvd/mysql56/root/var/db/mysql_secure canmount on local nvd/mysql56/root/var/db/mysql_tmpdir canmount on local
That is better, but see below for more changes.
Let’s move the data out of the way.
$ sudo mv bacula baulca.origin
Note my typo above.
Let’s fix bacula and copy over the data.
$ sudo zfs set canmount=yes nvd/mysql56/root/usr/home/dan/src/bacula cannot set property for 'nvd/mysql56/root/usr/home/dan/src/bacula': 'canmount' must be one of 'on | off | noauto' $ sudo zfs set canmount=on nvd/mysql56/root/usr/home/dan/src/bacula $ ls -l total 19 drwxr-xr-x 6 dltstats dltstats 7 Jan 14 23:33 baulca.origin $ sudo zfs mount nvd/mysql56/root/usr/home/dan/src/bacula $ ls -l total 20 drwxr-xr-x 2 root wheel 2 Apr 21 20:48 bacula drwxr-xr-x 6 dltstats dltstats 7 Jan 14 23:33 baulca.origin $
set permissions:
$ sudo chown dan:dan bacula
Copy over the data
$ sudo cp -a baulca.origin/* bacula/ cp: baulca.origin/regress/weird-files/sock-file is a socket (not copied). $
Sockets can be ignored.
One more check to see what is mounted:
$ zfs get -r mounted nvd NAME PROPERTY VALUE SOURCE nvd mounted no - nvd/mysql56 mounted no - nvd/mysql56/root mounted no - nvd/mysql56/root/usr mounted no - nvd/mysql56/root/usr/home mounted no - nvd/mysql56/root/usr/home/dan mounted no - nvd/mysql56/root/usr/home/dan/src mounted no - nvd/mysql56/root/usr/home/dan/src/bacula mounted yes - nvd/mysql56/root/var mounted no - nvd/mysql56/root/var/db mounted no - nvd/mysql56/root/var/db/mysql mounted yes - nvd/mysql56/root/var/db/mysql_secure mounted yes - nvd/mysql56/root/var/db/mysql_tmpdir mounted yes -
Start the jail:
$ sudo iocage start mysql56 * Starting mysql56 + Started OK + Using devfs_ruleset: 11 + Starting services OK + Executing poststart OK
Log into the jail:
[dan@mysql56:~] $ cd src [dan@mysql56:~/src] $ ls -l total 21 drwxr-xr-x 6 dan dan 7 Apr 21 20:56 bacula drwxr-xr-x 6 dan dan 7 Jan 14 23:33 baulca.origin [dan@mysql56:~/src] $
How does MySQL look?
[dan@mysql56:~/src] $ cd /var/db [dan@mysql56:/var/db] $ ls -l total 17709 drwx------ 2 operator operator 10 Aug 23 2015 entropy drwxr-xr-x 3 root wheel 4 Mar 18 21:05 etcupdate drwx------ 4 root wheel 11 Mar 18 21:05 freebsd-update drwx------ 2 root wheel 2 Jun 22 2018 hyperv drwx------ 2 root wheel 2 Dec 4 2012 ipf -r--r--r-- 1 nobody wheel 247659 Sep 2 2017 locate.database drwxr-xr-x 2 logcheck logcheck 5 Aug 18 2015 logcheck -rw------- 1 root wheel 28808 Aug 5 2018 mergemaster.mtree drwxr-xr-x 6 mysql mysql 24 Apr 21 20:56 mysql drwxr-xr-x 5 root wheel 5 Apr 21 20:30 mysql-tmp drwxr-xr-x 2 mysql mysql 2 Apr 21 20:32 mysql_secure drwxr-xr-x 2 mysql mysql 2 Apr 21 22:24 mysql_tmpdir drwx------ 2 root wheel 2 Jun 22 2018 ntp -rw-r--r-- 1 root wheel 10663 Oct 27 12:10 ntpd.leap-seconds.list drwxr-xr-x 2 root wheel 10 Apr 21 04:13 pkg drwxr-xr-x 2 root wheel 2 Dec 4 2012 ports drwxr-xr-x 2 root wheel 2 Dec 4 2012 portsnap drwx------ 2 postfix wheel 3 May 16 2016 postfix -rw-r--r-- 1 root wheel 2097920 Mar 18 21:05 services.db drwx--x--x 3 root wheel 3 Mar 18 2016 sudo drwxr-xr-x 3 root wheel 3 Jun 22 2018 zfsd [dan@mysql56:/var/db] $
Everything looks good.
For testing, I started regression tests in mysql57, first, then in mysql56. It didn’t take long for mysql56 to catch up and overtake.
The mountpoints
Here are the mountpoint I created:
[dan@knew:~] $ zfs get -r mountpoint nvd NAME PROPERTY VALUE SOURCE nvd mountpoint /iocage/jails local nvd/mysql56 mountpoint /iocage/jails/mysql56 inherited from nvd nvd/mysql56/root mountpoint /iocage/jails/mysql56/root inherited from nvd nvd/mysql56/root/usr mountpoint /iocage/jails/mysql56/root/usr inherited from nvd nvd/mysql56/root/usr/home mountpoint /iocage/jails/mysql56/root/usr/home inherited from nvd nvd/mysql56/root/usr/home/dan mountpoint /iocage/jails/mysql56/root/usr/home/dan inherited from nvd nvd/mysql56/root/usr/home/dan/src mountpoint /iocage/jails/mysql56/root/usr/home/dan/src inherited from nvd nvd/mysql56/root/usr/home/dan/src/bacula mountpoint /iocage/jails/mysql56/root/usr/home/dan/src/bacula inherited from nvd nvd/mysql56/root/var mountpoint /iocage/jails/mysql56/root/var inherited from nvd nvd/mysql56/root/var/db mountpoint /iocage/jails/mysql56/root/var/db local nvd/mysql56/root/var/db/mysql mountpoint /iocage/jails/mysql56/root/var/db/mysql inherited from nvd/mysql56/root/var/db nvd/mysql56/root/var/db/mysql_secure mountpoint /iocage/jails/mysql56/root/var/db/mysql_secure inherited from nvd/mysql56/root/var/db nvd/mysql56/root/var/db/mysql_tmpdir mountpoint /iocage/jails/mysql56/root/var/db/mysql_tmpdir inherited from nvd/mysql56/root/var/db [dan@knew:~] $
A you can see there are two mount points which have SOURCE specified explicitly:
- nvd at
/iocage/jails
- nvd/mysql56/root/var/db at iocage/jails/mysql56/root/var/db
We do not need that second local and we can make do with just one. Let me fix that:
[dan@knew:~] $ sudo zfs inherit mountpoint nvd/mysql56/root/var/db cannot unmount '/iocage/jails/mysql56/root/var/db/mysql_tmpdir': Device busy [dan@knew:~] $ sudo iocage stop mysql56 * Stopping mysql56 + Executing prestop OK + Stopping services OK + Removing devfs_ruleset: 11 OK + Removing jail process OK + Executing poststop OK [dan@knew:~] $ sudo zfs inherit mountpoint nvd/mysql56/root/var/db [dan@knew:~] $ zfs get -r mountpoint nvd NAME PROPERTY VALUE SOURCE nvd mountpoint /iocage/jails local nvd/mysql56 mountpoint /iocage/jails/mysql56 inherited from nvd nvd/mysql56/root mountpoint /iocage/jails/mysql56/root inherited from nvd nvd/mysql56/root/usr mountpoint /iocage/jails/mysql56/root/usr inherited from nvd nvd/mysql56/root/usr/home mountpoint /iocage/jails/mysql56/root/usr/home inherited from nvd nvd/mysql56/root/usr/home/dan mountpoint /iocage/jails/mysql56/root/usr/home/dan inherited from nvd nvd/mysql56/root/usr/home/dan/src mountpoint /iocage/jails/mysql56/root/usr/home/dan/src inherited from nvd nvd/mysql56/root/usr/home/dan/src/bacula mountpoint /iocage/jails/mysql56/root/usr/home/dan/src/bacula inherited from nvd nvd/mysql56/root/var mountpoint /iocage/jails/mysql56/root/var inherited from nvd nvd/mysql56/root/var/db mountpoint /iocage/jails/mysql56/root/var/db inherited from nvd nvd/mysql56/root/var/db/mysql mountpoint /iocage/jails/mysql56/root/var/db/mysql inherited from nvd nvd/mysql56/root/var/db/mysql_secure mountpoint /iocage/jails/mysql56/root/var/db/mysql_secure inherited from nvd nvd/mysql56/root/var/db/mysql_tmpdir mountpoint /iocage/jails/mysql56/root/var/db/mysql_tmpdir inherited from nvd [dan@knew:~] $ sudo iocage start mysql56 * Starting mysql56 + Started OK + Using devfs_ruleset: 11 + Starting services OK + Executing poststart OK [dan@knew:~] $
Much better. The goal is to specify as little as possible.
Following this example, I will create more data set, much like the above, for the following jails:
- mysql57
- mysql80
- pg94
- pg95
- pg96
- pg10
- pg11
Hopefully I will get better at getting the specifications correct the first time.