Jun 112020
 

I wrote this post late one night while pondering how to solve this problem. I may have solved it with much less effort.

I want to copy a zroot from old drives to new drives. The new drives are in a test box of mine. Once the new drives are configured, I will replace the existing mirror with them.

Not shown here, I have already installed FreeBSD 12.1 on the new drives.

I think the better solution is a handful of zfs snapshot and zfs send.

The new plan

This is the new plan. This is what I started tonight.

This is the existing system (the old drives):

$ zfs list -r zroot
NAME                              USED  AVAIL  REFER  MOUNTPOINT
zroot                            25.8G   180G    23K  /zroot
zroot/bootenv                    20.9G   180G    34K  legacy
zroot/bootenv/12.1                  1K   180G   482M  /
zroot/bootenv/12S                   1K   180G   600M  /
zroot/bootenv/default            20.9G   180G   622M  /
zroot/bootenv/default/tmp        72.5M   180G   944K  /tmp
zroot/bootenv/default/usr        9.91G   180G  2.76G  /usr
zroot/bootenv/default/usr/local  3.88G   180G  1.06G  /usr/local
zroot/bootenv/default/var        9.44G   180G  6.24G  /var
zroot/bootenv/default/var/audit    75K   180G    23K  /var/audit
zroot/bootenv/default/var/empty    23K   180G    23K  /var/empty
zroot/bootenv/default/var/log     214M   180G  17.6M  /var/log
zroot/bootenv/default/var/tmp    1.84M   180G  1.23M  /var/tmp

This is how I copied that over, with renaming to be done later:

[root@slocum:~] # zfs snapshot zroot/bootenv/default/var@ns1
[root@slocum:~] # zfs send zroot/bootenv/default/var@ns1 | ssh  -T root@10.55.0.247 'zfs recv zroot/var1'
Password for root@newslocum.int.unixathome.org:
[root@slocum:~] # 
[root@slocum:~] # zfs send zroot/bootenv/default/usr@ns1 | ssh  -T root@10.55.0.247 'zfs recv zroot/usr1'
Password for root@newslocum.int.unixathome.org:

[root@slocum:~] # zfs send zroot/bootenv/default/usr/local@ns1 | ssh  -T root@10.55.0.247 'zfs recv zroot/usr/local'
Password for root@newslocum.int.unixathome.org:

[root@slocum:~] # zfs send  | ssh  -T root@10.55.0.247 'zfs recv zroot/usr/local'
[root@slocum:~] # zfs send zroot/bootenv/default@ns1 | ssh  -T root@10.55.0.247 'zfs recv zroot/ROOT/default1'
Password for root@newslocum.int.unixathome.org:
[root@slocum:~] # 

[root@slocum:~] # zfs snapshot zroot/bootenv/default/var/tmp@ns1
[root@slocum:~] # zfs send zroot/bootenv/default/var/tmp@ns1 | ssh  -T root@10.55.0.247 'zfs recv zroot/var/log1@ns1'
Password for root@newslocum.int.unixathome.org:
[root@slocum:~] # 

These are the new drives after the above copy:

root@newslocum:~ # zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
zroot                8.62G   197G    88K  /zroot
zroot/ROOT            755M   197G    88K  none
zroot/ROOT/default    477M   197G   477M  /
zroot/ROOT/default1   278M   197G   275M  /zroot/default1
zroot/tmp              88K   197G    88K  /tmp
zroot/usr             625M   197G    88K  /usr
zroot/usr/home        124K   197G   124K  /usr/home
zroot/usr/local       625M   197G   625M  /usr/local
zroot/usr1           1.26G   197G  1.26G  /zroot/usr1
zroot/var            3.04M   197G    88K  /var
zroot/var/audit        88K   197G    88K  /var/audit
zroot/var/crash        88K   197G    88K  /var/crash
zroot/var/log         140K   197G   140K  /var/log
zroot/var/log1       2.48M   197G  2.48M  /var/log1
zroot/var/mail         88K   197G    88K  /var/mail
zroot/var/tmp          88K   197G    88K  /var/tmp
zroot/var1           6.01G   197G  6.01G  /zroot/var1
root@newslocum:~ # 

I think that’s it.

Copy again!

I’ll do another set of snapshots and incremental sends before the replacement occurs.

When the time comes…

I’ll boot the test system from mfsBSD and renamed all those new datasets. (e.g. zfs rename zroot/ROOT/default zroot/ROOT/defaultOLD). Then I’ll do a test reboot to make sure it still boot.

There’s the matter of a zpool export on the old drives and new drives and a zpool import. I thought I might have to do sudo zpool set bootfs=zroot/ROOT/default zroot but I think that’s set already, during the install of FreeBSD onto the new drives.

I’m going to post this now and review it later. There is no rush for this.

The original plan

WARNING: I wrote most of this late at night and I know I rambled on. I think it helped me work through the idea in my head.

I’m posting for entertainment purposes only.


In this post I will describe how I replace a zroot (the bootable base-OS of my FreeBSD server) with a new zroot.

In this post:

  1. FreeBSD 12.1 – but don’t let that stop you.
  2. mfsBSD 12.1 – I think, that’s the plan

Why replace the zroot?

The existing zroot pools seems wrong. Compare this zroot:

[dan@slocum:~] $ zfs list -r zroot
NAME                              USED  AVAIL  REFER  MOUNTPOINT
zroot                            24.2G   181G  4.88G  /zroot
zroot/bootenv                    19.3G   181G    34K  legacy
zroot/bootenv/12.1                  1K   181G   482M  /
zroot/bootenv/default            19.3G   181G   604M  /
zroot/bootenv/default/tmp        79.0M   181G   955K  /tmp
zroot/bootenv/default/usr        9.27G   181G  2.38G  /usr
zroot/bootenv/default/usr/local  3.57G   181G  1.06G  /usr/local
zroot/bootenv/default/usr/obj    2.34M   181G  2.23M  /usr/obj
zroot/bootenv/default/usr/src    1.01G   181G  1.01G  /usr/src
zroot/bootenv/default/var        8.94G   181G  5.83G  /var
zroot/bootenv/default/var/audit    36K   181G    23K  /var/audit
zroot/bootenv/default/var/empty    23K   181G    23K  /var/empty
zroot/bootenv/default/var/log     201M   181G  19.5M  /var/log
zroot/bootenv/default/var/tmp    1.58M   181G  1.24M  /var/tmp
[dan@slocum:~] $ 

With this one:

[dan@r720-01:~] $ zfs list -r zroot
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot               13.5G  56.7G    88K  /zroot
zroot/ROOT          5.24G  56.7G    88K  none
zroot/ROOT/default  5.24G  56.7G  2.90G  /
zroot/tmp           9.32M  56.7G   788K  /tmp
zroot/usr           8.17G  56.7G    88K  /usr
zroot/usr/home      3.67M  56.7G  2.40M  /usr/home
zroot/usr/obj       6.81G  56.7G  4.87G  /usr/obj
zroot/usr/ports       88K  56.7G    88K  /usr/ports
zroot/usr/src       1.36G  56.7G  1.36G  /usr/src
zroot/var           18.8M  56.7G    88K  /var
zroot/var/audit       88K  56.7G    88K  /var/audit
zroot/var/crash      128K  56.7G    88K  /var/crash
zroot/var/log       17.5M  56.7G  1.39M  /var/log
zroot/var/mail       416K  56.7G   112K  /var/mail
zroot/var/tmp        672K  56.7G   296K  /var/tmp
[dan@r720-01:~] $ 

slocum has been around since early 2013 long before the existing standard layout arose. It wasn’t until 2018 that a zroot was added to this system. I think that is where we can trace this back.

However, this is not about layout. It is about function.

I’ve been having a lingering issue with pf and vnet: If I invoked: pfctl -f /etc/pf.conf, I must restart all vnet jails. If I do not, simple things like contacting the mail server from the vnet jail do not work. This is related to epair groups, I am sure.

Kristof Provost had a work-in-progress patch which I was eager to try. This required building a new kernel.

After starting the buildworld, error after error started to appear.

The existing zroot, I discovered today, is missing some essential items. Case in point: there is no /usr/bin/cc on this system. Intriguing.

Next, ldd was missing.

A bunch of includes were not there either.

It got to the point where I decided to build on another host, then install over NFS.

That’s why.

Creating the new zroot

I have ordered two Samsung SSD 860 EVO 250GB 2.5 Inch SATA III Internal SSD drives. They will be used to create a ZFS mirror. I would rather create a new one and leave the exist zroot untouched.

I will use another system to partition the new SSDs and install FreeBSD.

Which datasets do I need?

Which datasets from the old zroot will I need to copy somewhere else?

[dan@slocum:~] $ zfs list -r zroot
NAME                              USED  AVAIL  REFER  MOUNTPOINT
zroot                            24.2G   181G  4.88G  /zroot
zroot/bootenv                    19.3G   181G    34K  legacy
zroot/bootenv/12.1                  1K   181G   482M  /
zroot/bootenv/default            19.3G   181G   604M  /
zroot/bootenv/default/tmp        79.0M   181G   955K  /tmp
zroot/bootenv/default/usr        9.27G   181G  2.38G  /usr
zroot/bootenv/default/usr/local  3.57G   181G  1.06G  /usr/local
zroot/bootenv/default/usr/obj    2.34M   181G  2.23M  /usr/obj
zroot/bootenv/default/usr/src    1.01G   181G  1.01G  /usr/src
zroot/bootenv/default/var        8.94G   181G  5.83G  /var
zroot/bootenv/default/var/audit    36K   181G    23K  /var/audit
zroot/bootenv/default/var/empty    23K   181G    23K  /var/empty
zroot/bootenv/default/var/log     201M   181G  19.5M  /var/log
zroot/bootenv/default/var/tmp    1.58M   181G  1.24M  /var/tmp
[dan@slocum:~] $ 

I do not need /tmp because that directory is cleared upon every boot:

$ grep tmp /etc/rc.conf
clear_tmp_enable="YES"

What about /usr?

Lines 8 and 9 refer to /usr and /usr/local so let’s look in there:

[dan@slocum:/usr] $ ls
bin     home    include lib     lib32   libdata libexec local   obj     ports   sbin    share   src     tests
[dan@slocum:/usr] $ 

I will talk about home below, but I don’t need to move that.

Ports install to /usr/local, so I’m going to need to copy that over.

The new zroot layout will probably look like this:

zroot/usr           8.17G  56.7G    88K  /usr
zroot/usr/home      3.67M  56.7G  2.40M  /usr/home
zroot/usr/obj       6.81G  56.7G  4.87G  /usr/obj
zroot/usr/ports       88K  56.7G    88K  /usr/ports
zroot/usr/src       1.36G  56.7G  1.36G  /usr/src

home, obj, ports, and src will not be required. I will zfs destroy them on the new zroot after creation.

/var/audit is empty on the existing host. Nothing to do here.


[dan@slocum:/usr] $ sudo ls -l /var/audit
total 2
drwxrwx— 2 auditdistd audit 2 Nov 8 2013 dist
drwx—— 2 auditdistd wheel 2 Nov 8 2013 remote
[dan@slocum:/usr] $ sudo ls -l /var/audit/dist
total 0
[dan@slocum:/usr] $ sudo ls -l /var/audit/remote
total 0
[dan@slocum:/usr] $

I don’t need /var/empty because it’s… empty.

I want:

/var/backups
/var/cache
/var/cron
/var/db
/var/lib
/var/mail

… I think I should just copy over all of var

I need to copy over /var/log.

Looking in /var/tmp, I don’t need any of that either.

Configuration files

I need all of /etc/ and /usr/local/etc/

There must be a better way

This is getting complex.

Why don’t I go back to a simple tar | tar solution? I used this back in 1999.

Why not just copy all of zroot over using tar? That is all completely independent of ZFS datasets and mountpoints.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive