I am replacing one zroot with another because of missing binaries and suspect layout. The existing zroot is an old system predating current zroot layouts.

This work is being performed on a test system (that link is not safe for young eyes).

In this post:

FreeBSD 12.1
mfsBSD 12.1

The failed attempt

I tried once, and failed, with with old zroot_old and new zroot_new mounted. zroot_new became unusable and I started again. This post documents that restart. I am sure I could repeat this now.

That zpool import failure was:

root@mfsbsd:~ # zpool import -f -R /dst zroot_dst
cannot import 'zroot_dst': I/O error
	Destroy and re-create the pool from
	a backup source.

/var/log/messages had:

Jun 20 11:40:19 mfsbsd kernel: ZFS filesystem version: 5
Jun 20 11:40:19 mfsbsd kernel: ZFS storage pool version: features support (5000)
Jun 20 11:40:22 mfsbsd ZFS[2371]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$14936720131378251566
Jun 20 11:40:22 mfsbsd ZFS[2372]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$4713809697672316496
Jun 20 11:40:22 mfsbsd ZFS[2373]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$14936720131378251566
Jun 20 11:40:22 mfsbsd ZFS[2374]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$4713809697672316496
Jun 20 11:40:22 mfsbsd kernel: Solaris: WARNING: 
Jun 20 11:40:22 mfsbsd kernel: can't open objset 95, error 5
Jun 20 11:40:22 mfsbsd ZFS[2375]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$14936720131378251566
Jun 20 11:40:22 mfsbsd ZFS[2376]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$4713809697672316496
Jun 20 11:40:22 mfsbsd ZFS[2377]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$14936720131378251566
Jun 20 11:40:22 mfsbsd ZFS[2378]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$4713809697672316496
Jun 20 11:40:22 mfsbsd kernel: Solaris: WARNING: 
Jun 20 11:40:22 mfsbsd kernel: can't open objset 95, error 5
Jun 20 11:40:22 mfsbsd kernel: GEOM_LABEL
Jun 20 11:40:22 mfsbsd ZFS[2379]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$14936720131378251566
Jun 20 11:40:22 mfsbsd kernel: : Cannot create slice 8a57407f-ab1b-11ea-8a52-94de80aad24a.
Jun 20 11:40:22 mfsbsd ZFS[2380]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$4713809697672316496
Jun 20 11:40:22 mfsbsd ZFS[2381]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$14936720131378251566
Jun 20 11:40:22 mfsbsd ZFS[2382]: vdev state changed, pool_guid=$10443359089269485823 vdev_guid=$4713809697672316496
Jun 20 11:40:22 mfsbsd kernel: Solaris: WARNING: 
Jun 20 11:40:22 mfsbsd kernel: can't open objset 95, error 5

Allan Jude mentioned trying zdb -lu /dev/ada0p3 but I had already moved on and created a new zroot.

Install a new zroot

I took the two SSD drives, attached them to the test system, and installed FreeBSD. It looked something like this. I have copy/pasted/edited the existing zroot, so the sizes are wrong for a newly-installed FreeBSD zroot.

[dan@dent:~] $ slocum zfs list -r zroot
NAME                  USED  AVAIL  REFER  MOUNTPOINT
zroot                11.8G   194G    88K  /zroot
zroot/ROOT            754M   194G    88K  none
zroot/ROOT/default    277M   194G   275M  /
zroot/tmp            1.77M   194G    88K  /tmp
zroot/usr            1.87G   194G  1.26G  /usr
zroot/usr/local       629M   194G   625M  /usr/local
zroot/var            9.17G   194G  6.06G  /var
zroot/var/audit       200K   194G    88K  /var/audit
zroot/var/empty        88K   194G    88K  /var/empty
zroot/var/log         243M   194G  24.5M  /var/log
zroot/var/tmp        4.08M   194G  2.51M  /var/tmp

Just in case, I did a zfs snapshot -r zroot@pristine – that reminds me, I just deleted that: zfs destroy -r zroot@pristine.

Renaming the zroot

Now that I’ve done this a few times over the past few days, I had an idea: rename the new zroot before copying over the data.

I booted the test system again, this time, with mfsBSD (see the next section for details on that).

I did this:

# zfs import -R /mnt zroot zroot_new
# zfs export zroot_new
# zfs import -R /mnt zroot_new
# zfs export zroot_new

What does this do?

imports zroot and renames it to zroot_new, mounting it at /mnt
exports zroot_new
imports zroot_new, mounting it at /mnt
exports zroot_new

Why two exports? I wanted to be sure it imported and exported with the new name.

The next step: copy data from the existing zroot.

About this thumb drive

I wrote about burning a thumb drive image to a USB drive.

I’ve booted the test system from that thumbdrive.

Next, I will manipulate those new drives and copy over data from the existing zroot.

Logging into mfsBSD

One of the lovely features of mfsBSD is it allows me to ssh as root for this task. This is not something one should normally do, but this is not a normal circumstance.

$ ssh root@10.0.0.247
The authenticity of host '10.0.0.247 (10.55.0.247)' can't be established.
ECDSA key fingerprint is SHA256:8ysqgLdw3eU29srXdNlyze6QqF5ztkKq4HwaucLO/Z8.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.0.0.247' (ECDSA) to the list of known hosts.
Password for root@mfsbsd:

The password is: mfsroot.

An initial send | receive

While I had the new zroot still in the test server, I did an initial copy over while the production system was still in use.

[root@slocum:~] # zfs send zroot/bootenv/default@ns3           | ssh -T root@10.55.0.247 'zfs receive zroot_new/ROOT/default_new'
[root@slocum:~] # zfs send zroot/bootenv/default/usr@ns3       | ssh -T root@10.55.0.247 'zfs receive zroot_new/usr_new'
[root@slocum:~] # zfs send zroot/bootenv/default/usr/local@ns3 | ssh -T root@10.55.0.247 'zfs receive zroot_new/usr_new/local_new'
[root@slocum:~] # zfs send -R zroot/bootenv/default/var@ns3    | ssh -T root@10.55.0.247 'zfs receive zroot_new/var_new'

I’ve already create the ns3 snapshot on slocum. There are only four things I am interested in. The above commands are white-space adjusted to make it easier to see the pattern in the destination.

In each case, I am copying the data which I need on the new server:

/
/usr
/usr/local
/var

Over on the new server, we now have this:

root@slocumnew:~ # zfs list
NAME                          USED  AVAIL  REFER  MOUNTPOINT
zroot_new                    11.6G   194G    88K  /zroot
zroot_new/ROOT                477M   194G    88K  none
zroot_new/ROOT/default        477M   194G   477M  /
zroot_new/ROOT/default_new    128K   194G   128K  none
zroot_new/tmp                  88K   194G    88K  /tmp
zroot_new/usr                 388K   194G    88K  /usr
zroot_new/usr/home            124K   194G   124K  /usr/home
zroot_new/usr/ports            88K   194G    88K  /usr/ports
zroot_new/usr/src              88K   194G    88K  /usr/src
zroot_new/usr_new            1.87G   194G  1.26G  /zroot/usr_new
zroot_new/usr_new/local_new   625M   194G   625M  /zroot/usr_new/local_new
zroot_new/var                 660K   194G    88K  /var
zroot_new/var/audit            88K   194G    88K  /var/audit
zroot_new/var/crash            88K   194G    88K  /var/crash
zroot_new/var/log             220K   194G   140K  /var/log
zroot_new/var/mail             88K   194G    88K  /var/mail
zroot_new/var/tmp              88K   194G    88K  /var/tmp
zroot_new/var_new            9.20G   194G  6.06G  /zroot/var_new
zroot_new/var_new/audit       256K   194G    88K  /zroot/var_new/audit
zroot_new/var_new/empty        88K   194G    88K  /zroot/var_new/empty
zroot_new/var_new/log         240M   194G  24.8M  /zroot/var_new/log
zroot_new/var_new/tmp        3.60M   194G  2.48M  /zroot/var_new/tmp
root@slocumnew:~ #

Next: rename stuff to put it into the correct location.

Reboot new zroot into mfsBSD

After rebooting the test server into mfsBSD, I started moving stuff around.

root@mfsbsd:~ # zfs rename zroot_new/ROOT/default zroot_new/ROOT/def_old
root@mfsbsd:~ # zfs rename zroot_new/ROOT/default_new zroot_new/ROOT/default

Then I compared the properties and set them the same for both of the above datasets. I created an ssh session to the test host and the production host. This was in two tabs of the same window. A simple ctrl-TAB let me compare the properties with ease.

zfs get all zroot_new/ROOT/def_old
zfs get all zroot_new/ROOT/default

The dataset names are the same length to make comparison easier.

I found the differences, and set the new one accordingly:

root@mfsbsd:~ # zfs set mountpoint=/    zroot_new/ROOT/default
root@mfsbsd:~ # zfs set canmount=noauto zroot_new/ROOT/default

Adjust the bootfs:

root@mfsbsd:~ # zpool set bootfs=zroot_new/ROOT/default zroot_new

That identifies the default bootable dataset for the root pool. I am not sure it was set wrong, but this is the right thing to do.

Let’s rename usr

root@mfsbsd:~ # zfs rename zroot_new/usr           zroot_new/usr_old
root@mfsbsd:~ # zfs rename zroot_new/usr_new       zroot_new/usr
root@mfsbsd:~ # zfs rename zroot_new/usr/local_new zroot_new/usr/local

I adjust the mountpoints:

root@mfsbsd:~ # zfs set mountpoint=/usr zroot_new/usr
root@mfsbsd:~ # zfs set mountpoint=none zroot_new/usr_old

Do similar for var

root@mfsbsd:~ # zfs rename zroot_new/var zroot_new/var_old
root@mfsbsd:~ # zfs set mountpoint=none  zroot_new/var_old

root@mfsbsd:~ # zfs rename zroot_new/var_new zroot_new/var

Now we have:

root@mfsbsd:~ # zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
zroot_new                11.6G   194G    88K  /mnt/zroot
zroot_new/ROOT            478M   194G    88K  none
zroot_new/ROOT/def_old    477M   194G   477M  /mnt
zroot_new/ROOT/default    224K   194G   144K  /mnt
zroot_new/tmp             144K   194G    88K  /mnt/tmp
zroot_new/usr            1.87G   194G  1.26G  /mnt/usr
zroot_new/usr/local       625M   194G   625M  /mnt/usr/local
zroot_new/usr_old         388K   194G    88K  none
zroot_new/usr_old/home    124K   194G   124K  none
zroot_new/usr_old/ports    88K   194G    88K  none
zroot_new/usr_old/src      88K   194G    88K  none
zroot_new/var            9.20G   194G  6.06G  /mnt/var
zroot_new/var/audit       256K   194G    88K  /mnt/var/audit
zroot_new/var/empty        88K   194G    88K  /mnt/var/empty
zroot_new/var/log         240M   194G  24.8M  /mnt/var/log
zroot_new/var/tmp        3.60M   194G  2.48M  /mnt/var/tmp
zroot_new/var_old         680K   194G    88K  none
zroot_new/var_old/audit    88K   194G    88K  none
zroot_new/var_old/crash    88K   194G    88K  none
zroot_new/var_old/log     232K   194G   144K  none
zroot_new/var_old/mail     88K   194G    88K  none
zroot_new/var_old/tmp      96K   194G    88K  none
root@mfsbsd:~ #

Importing and renaming the zpools

The new zroot drives has been installed into the new server. The server has been booted from an mfsBSD thumbdrive.

This imports the existing zroot, renaming it to zroot_old.

Then it imports the new zroot from above, renaming it to zroot.

root@mfsbsd:~ # zpool import -f -R /old zroot zroot_old
root@mfsbsd:~ # zpool import -f -R /new zroot_new zroot

Copying over the data

Remember, the production server is still running, the filesystem is still live, data is still being updated.

We need to cater for those updates.

First, I snapshot the original data. I’ve already used ns1 through ns3:

root@mfsbsd:~ # zfs snapshot -r zroot_old@ns4

What is ns? New Slocum

Let’s see what we have.

This is the ns3 snapshots of the original zpool:

root@mfsbsd:~ # zfs list -r -t snapshot zroot_old | grep ns3
zroot_old@ns3                                                                    0      -    23K  -
zroot_old/bootenv@ns3                                                            0      -    34K  -
zroot_old/bootenv/12.1@ns3                                                       0      -   482M  -
zroot_old/bootenv/12S@ns3                                                        0      -   600M  -
zroot_old/bootenv/default@ns3                                                 192K      -   618M  -
zroot_old/bootenv/default/tmp@ns3                                             772K      -   943K  -
zroot_old/bootenv/default/usr@ns3                                             240K      -  2.76G  -
zroot_old/bootenv/default/usr/local@ns3                                      1.35M      -  1.06G  -
zroot_old/bootenv/default/var@ns3                                            2.03M      -  6.27G  -
zroot_old/bootenv/default/var/audit@ns3                                          0      -    23K  -
zroot_old/bootenv/default/var/empty@ns3                                          0      -    23K  -
zroot_old/bootenv/default/var/log@ns3                                         205K      -  17.5M  -
zroot_old/bootenv/default/var/tmp@ns3                                           1K      -  1.23M  -

This is what we have on the new zpool:

root@mfsbsd:~ # zfs list -r -t snapshot zroot | grep ns3
zroot/ROOT/default@ns3                                    232K      -   275M  -
zroot/usr@ns3                                              96K      -  1.26G  -
zroot/usr/local@ns3                                          0      -   625M  -
zroot/var@ns3                                             904K      -  6.06G  -
zroot/var/audit@ns3                                          0      -    88K  -
zroot/var/empty@ns3                                          0      -    88K  -
zroot/var/log@ns3                                         164K      -  24.8M  -
zroot/var/tmp@ns3                                            0      -  2.48M  -

The final copy of the ns4 snapshot:

root@mfsbsd:~ # zfs send -i zroot_old/bootenv/default@ns3 zroot_old/bootenv/default@ns4 | zfs receive -F zroot/ROOT/default
root@mfsbsd:~ # zfs send -R -i zroot_old/bootenv/default/usr@ns3 zroot_old/bootenv/default/usr@ns4 | zfs receive -F zroot/usr
root@mfsbsd:~ # zfs send -R -i zroot_old/bootenv/default/var@ns3 zroot_old/bootenv/default/var@ns4 | zfs receive -F zroot/var

I didn’t take notes for this part, but I am sure I did this:

root@mfsbsd:~ # zfs export zroot_old
root@mfsbsd:~ # zfs export zroot

Disconnect the original zroot!

This is what I did different this time: I disconnected the data and power cables for the old zroot.

The system booted up just fine, no worries.

For fun and games

You might find this Twitter thread interesting. That was my failed attempt. The screen shots may be educational. The import-by-guid is also a good tool