May 222021
 

I am going to upgrade a FreeBSD 12.2 host to FreeBSD 13.0 with one reboot. Actually, here, because of the zpool update, there is another reboot. This is significantly fewer than the standard upgrade process (yeah, critics claim 1 is not significantly less than 2; sigh). The instructions are taken from vermaden’s blog post on Upgrade FreeBSD with ZFS Boot Environments. I have seen a couple of references to this post lately and this upgrade was the perfect chance to try it.

This procedure assumes a standard layout for the main directories of FreeBSD.

In this post:

  1. All my boot partitions are 512K
  2. FreeBSD 12.2
  3. Updates via Boot Environment (BE)

Details about the host ZFS

The zpools:

[dvl@r720-02:~] $ zpool list
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
data01  1.81T  1.99G  1.81T        -         -     1%     0%  1.00x  ONLINE  -
zroot    109G  18.0G  91.0G        -         -    57%    16%  1.00x  ONLINE  -

The status:

[dvl@r720-02:~] $ zpool status
  pool: data01
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:03 with 0 errors on Thu May 20 03:30:41 2021
config:

	NAME                         STATE     READ WRITE CKSUM
	data01                       ONLINE       0     0     0
	  mirror-0                   ONLINE       0     0     0
	    gpt/S59VNS0N809087J_S00  ONLINE       0     0     0
	    gpt/S59VNJ0N631973D_S01  ONLINE       0     0     0
	  mirror-1                   ONLINE       0     0     0
	    gpt/S5B3NDFN807383E_S02  ONLINE       0     0     0
	    gpt/S5B3NDFN807386P_S03  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:01:35 with 0 errors on Thu May 20 03:32:17 2021
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    ada0p3  ONLINE       0     0     0
	    ada1p3  ONLINE       0     0     0

errors: No known data errors
[dvl@r720-02:~] $ 

The upgrade begins

[dvl@r720-02:~] $ sudo bectl create -r 13
Created successfully


[dvl@r720-02:~] $ sudo bectl mount 13 /var/tmp/BE-13
Mounted successfully on '/var/tmp/BE-13'

[dvl@r720-02:~] $ sudo chroot /var/tmp/BE-13
root@r720-02:/ # bash
[root@r720-02:/] # mount -t devfs devfs /dev

If any of the above steps fail, stop and find out why. For one of my servers did not have a standard ZFS filesystem layout. It had been manually configured into a BE-compatible layout. Mistakes were made. By me.

I removed the old freebsd-update files. This was where I discovered that the filesystem layout on this host did not conform to standard FreeBSD practices. First, check to verify this directory exists and has content.

[root@r720-02:/] # rm -rf /var/db/freebsd-update

NOTE: if the above directory empties quickly, you may have a problem. I bet the directory does not exist. It should. if it does not, I bet your zroot/var is not properly set up for a BE. Check the directory first. Make sure it exists in this BE.

The first of several freebsd-update commands:

[root@r720-02:/] # mkdir /var/db/freebsd-update
[root@r720-02:/] # freebsd-update upgrade -r 13.0-RELEASE
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching public key from update2.freebsd.org... done.
Fetching metadata signature for 12.2-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata files... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/generic world/base world/doc world/lib32

The following components of FreeBSD do not seem to be installed:
kernel/generic-dbg world/base-dbg world/lib32-dbg

Does this look reasonable (y/n)? y

Fetching metadata signature for 13.0-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.


....
/bin/ln
/bin/ls
/bin/mkdir
/bin/mv
To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".

The next:

[root@r720-02:/] # freebsd-update install
src component not installed, skipped
Installing updates...
Kernel updates have been installed.  Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.

NOTE: I did *NOT* reboot, despite the instructions contained above.

[root@r720-02:/] # freebsd-update install
src component not installed, skipped
Installing updates...sysctl: unknown oid 'user.localbase'
Scanning //usr/share/certs/blacklisted for certificates...
Scanning //etc/ssl/blacklisted for certificates...
Scanning //usr/share/certs/trusted for certificates...
Scanning //etc/ssl/certs for certificates...

Completing this upgrade requires removing old shared object files.
Please rebuild all installed 3rd party software (e.g., programs
installed from the ports tree) and then run "/usr/sbin/freebsd-update install"
again to finish installing updates.

This is where I deviate from the instructions I was following.

I update from my own repo. Therefore, I updated /usr/local/etc/pkg/repos/local.conf to make this change:

-   url: "pkg+http://fedex.int.unixathome.org/packages/122amd64-default-master-list/"
+   url: "pkg+http://fedex.int.unixathome.org/packages/13amd64-default-master-list/"

You’ll have to decide if that is something you need to do. Generally, if you are using the FreeBSD repos, you do not have to make this change.

[root@r720-02:/] # pkg bootstrap -f
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/latest, please wait...
Installing pkg-1.16.3...
package pkg is already installed, forced install
Extracting pkg-1.16.3: 100%

Then I returned to the script:

[root@r720-02:/] # pkg upgrade -f
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB   6.5MB/s    00:01    
Processing entries: 100%
FreeBSD repository update completed. 30513 packages processed.
All repositories are up to date.
Updating database digests format: 100%
Checking for upgrades (128 candidates): 100%
Processing candidates (128 candidates): 100%
The following 138 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	mpdecimal: 2.5.1 [FreeBSD]
	p5-Authen-SASL: 2.16_1 [FreeBSD]
	p5-Digest-HMAC: 1.03_1 [FreeBSD]
	p5-GSSAPI: 0.28_1 [FreeBSD]
	p5-IO-Socket-INET6: 2.72_1 [FreeBSD]
	p5-IO-Socket-SSL: 2.070 [FreeBSD]
	p5-Mozilla-CA: 20200520 [FreeBSD]
	p5-Net-SSLeay: 1.88 [FreeBSD]
...

Next, I check to see if everything upgraded. This is based upon my upgrade to new ports repo recipe.

Remove any unneeded packages:

[root@knew:/] # pkg autoremove

Did everything get upgraded?

[root@knew:/] # pkg query -a '%n %t' | awk -v t=$(date -v -6H +%s) '$2 < t { print $1 }'
beadm
p5-Crypt-CBC

I removed both of those packages, because I don't use either.

[root@r720-02:/usr/local/etc] # freebsd-update install
src component not installed, skipped
Installing updates... done.
[root@r720-02:/usr/local/etc] # 

[root@r720-02:/usr/local/etc] # exit
exit
root@r720-02:/ # exit
exit
[dvl@r720-02:~] $ sudo umount /var/tmp/BE-13/dev

[dvl@r720-02:~] $ sudo bectl activate 13
Successfully activated boot environment 13
[dvl@r720-02:~] $ mount | grep /var/tmp
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
[dvl@r720-02:~] $ 

There, that looks better. Checking:

[dvl@r720-02:~] $ bectl list
BE      Active Mountpoint Space Created
13      R      -          10.5G 2021-05-21 16:23
default N      /          936K  2017-04-12 21:36
[dvl@r720-02:~] $ 

Let's make that a temporary boot.

NOTE: I now believe this is not the way I want to proceed in future. After the upcoming zpool upgrade the old BE will be unable to boot. I used the temp activate so that if I booted once, I wouldn't have to much about with the boot screen, selecting the old BE, etc. Thinking about this now, the boot-once is harder for me in this situation. I'll boot it, oh good, this is working, let me trying this.. Some days later, I've forgotten about the boot-once and reboot. I think not-using boot-once is logically less complicated.

[dvl@r720-02:~] $ sudo bectl activate -t 13
Successfully activated boot environment 13
for next boot
[dvl@r720-02:~] $ bectl list
BE      Active Mountpoint Space Created
13      R      -          10.5G 2021-05-21 16:23
default N      /          936K  2017-04-12 21:36
[dvl@r720-02:~] $ 

Reboot.

[dvl@r720-02:~] $ uname -a
FreeBSD r720-02.unixathome.org 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr  9 04:24:09 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

Success!

Now it's time to update the zpool. BE WARNED: at present, the system does not mention that you need to update your boot code. You need to do that. In previous versions, you got this notice:

If you boot from pool 'zroot', don't forget to update boot code.
Assuming you use GPT partitioning and da0 is your boot disk
the following command will do it:
 
    gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

As you can see here, that does not happen now. An issue has been raised and a FreeBSD PR created.

zpools before

These are my zpools right after booting in FreeBSD 13:

[dvl@r720-02:~] $ zpool status
  pool: data01
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:00:03 with 0 errors on Thu May 20 03:30:41 2021
config:

	NAME                         STATE     READ WRITE CKSUM
	data01                       ONLINE       0     0     0
	  mirror-0                   ONLINE       0     0     0
	    gpt/S59VNS0N809087J_S00  ONLINE       0     0     0
	    gpt/S59VNJ0N631973D_S01  ONLINE       0     0     0
	  mirror-1                   ONLINE       0     0     0
	    gpt/S5B3NDFN807383E_S02  ONLINE       0     0     0
	    gpt/S5B3NDFN807386P_S03  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:01:35 with 0 errors on Thu May 20 03:32:17 2021
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    ada0p3  ONLINE       0     0     0
	    ada1p3  ONLINE       0     0     0

errors: No known data errors

Checkpoint

First, I set a checkpoint, just in case:

[dvl@r720-02:~] $ sudo zpool checkpoint zroot
[dvl@r720-02:~] $ zpool status zroot
  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:01:35 with 0 errors on Thu May 20 03:32:17 2021
checkpoint: created Fri May 21 17:26:03 2021, consumes 648K
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    ada0p3  ONLINE       0     0     0
	    ada1p3  ONLINE       0     0     0

errors: No known data errors

zpool upgrade

The upgrade:

[dvl@r720-02:~] $ sudo zpool upgrade zroot
This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':
  userobj_accounting
  encryption
  project_quota
  resilver_defer
  bookmark_v2
  redaction_bookmarks
  redacted_datasets
  bookmark_written
  log_spacemap
  livelist
  device_rebuild
  zstd_compress
  draid

[dvl@r720-02:~] $ zpool status
  pool: data01
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:00:03 with 0 errors on Thu May 20 03:30:41 2021
config:

	NAME                         STATE     READ WRITE CKSUM
	data01                       ONLINE       0     0     0
	  mirror-0                   ONLINE       0     0     0
	    gpt/S59VNS0N809087J_S00  ONLINE       0     0     0
	    gpt/S59VNJ0N631973D_S01  ONLINE       0     0     0
	  mirror-1                   ONLINE       0     0     0
	    gpt/S5B3NDFN807383E_S02  ONLINE       0     0     0
	    gpt/S5B3NDFN807386P_S03  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:01:35 with 0 errors on Thu May 20 03:32:17 2021
checkpoint: created Fri May 21 17:26:03 2021, consumes 16.1M
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    ada0p3  ONLINE       0     0     0
	    ada1p3  ONLINE       0     0     0

errors: No known data errors

Note there is no notice to update the bootcode. This has trapped, and will trap, people.

Applying the bootcode

This works for my system. Inspect your own partitions to be sure:

[dvl@r720-02:~] $ gpart show ada0 ada1
=>       40  234441568  ada0  GPT  (112G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048    4194304     2  freebsd-swap  (2.0G)
    4196352  230244352     3  freebsd-zfs  (110G)
  234440704        904        - free -  (452K)

=>       40  234441568  ada1  GPT  (112G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048    4194304     2  freebsd-swap  (2.0G)
    4196352  230244352     3  freebsd-zfs  (110G)
  234440704        904        - free -  (452K)

[dvl@r720-02:~] $ 

On EFI systems, or systems setup for 'BOTH', the freebsd-boot partition is #2

[dvl@r720-02:~] $ sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
partcode written to ada0p1
bootcode written to ada0
[dvl@r720-02:~] $ sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
partcode written to ada1p1
bootcode written to ada1
[dvl@r720-02:~] $ 

Reboot and remove the checkpoints

After a succcesful reboot, I removed the checkpoints:

[dvl@r720-02:~] $ sudo zpool checkpoint -d zroot
[dvl@r720-02:~] $ sudo zpool checkpoint -d data01
[dvl@r720-02:~] $ zpool status
  pool: data01
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:00:04 with 0 errors on Fri May 21 17:54:11 2021
config:

	NAME                         STATE     READ WRITE CKSUM
	data01                       ONLINE       0     0     0
	  mirror-0                   ONLINE       0     0     0
	    gpt/S59VNS0N809087J_S00  ONLINE       0     0     0
	    gpt/S59VNJ0N631973D_S01  ONLINE       0     0     0
	  mirror-1                   ONLINE       0     0     0
	    gpt/S5B3NDFN807383E_S02  ONLINE       0     0     0
	    gpt/S5B3NDFN807386P_S03  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:01:38 with 0 errors on Fri May 21 17:28:14 2021
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    ada0p3  ONLINE       0     0     0
	    ada1p3  ONLINE       0     0     0

errors: No known data errors
[dvl@r720-02:~] $ 

Oh wait. I see I didn't upgrade data01

Let's do that.

Upgrading one zpool

This is me, upgrading one zpool from which I do not boot:

[dvl@r720-02:~] $ sudo zpool checkpoint data01
[dvl@r720-02:~] $ sudo zpool upgrade data01
This system supports ZFS pool feature flags.

Enabled the following features on 'data01':
  userobj_accounting
  encryption
  project_quota
  resilver_defer
  bookmark_v2
  redaction_bookmarks
  redacted_datasets
  bookmark_written
  log_spacemap
  livelist
  device_rebuild
  zstd_compress
  draid

[dvl@r720-02:~] $ sudo zpool checkpoint -d data01
[dvl@r720-02:~] $ zpool status data01
  pool: data01
 state: ONLINE
  scan: scrub repaired 0B in 00:00:04 with 0 errors on Fri May 21 18:08:28 2021
config:

	NAME                         STATE     READ WRITE CKSUM
	data01                       ONLINE       0     0     0
	  mirror-0                   ONLINE       0     0     0
	    gpt/S59VNS0N809087J_S00  ONLINE       0     0     0
	    gpt/S59VNJ0N631973D_S01  ONLINE       0     0     0
	  mirror-1                   ONLINE       0     0     0
	    gpt/S5B3NDFN807383E_S02  ONLINE       0     0     0
	    gpt/S5B3NDFN807386P_S03  ONLINE       0     0     0

errors: No known data errors
[dvl@r720-02:~] $ 

Done. :)

man page differences

One great thing about FreeBSD 13 - the man pages for zfs and zpool are split up.

Try man zpool-create or man zfs-create

I find this much easier than the previous situation.

I also like that /etc/motd is gone.

And that newsyslog.conf files need to end in .conf:

$ tail -2 /etc/newsyslog.conf
 /etc/newsyslog.conf.d/[!.]*.conf
 /usr/local/etc/newsyslog.conf.d/[!.]*.conf
$ 

I appreciate this change.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive