I am going to upgrade a FreeBSD 12.2 host to FreeBSD 13.0 with one reboot. Actually, here, because of the zpool update, there is another reboot. This is significantly fewer than the standard upgrade process (yeah, critics claim 1 is not significantly less than 2; sigh). The instructions are taken from vermaden’s blog post on Upgrade FreeBSD with ZFS Boot Environments. I have seen a couple of references to this post lately and this upgrade was the perfect chance to try it.
This procedure assumes a standard layout for the main directories of FreeBSD.
In this post:
- All my boot partitions are 512K
- FreeBSD 12.2
- Updates via Boot Environment (BE)
Details about the host ZFS
The zpools:
[dvl@r720-02:~] $ zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data01 1.81T 1.99G 1.81T - - 1% 0% 1.00x ONLINE - zroot 109G 18.0G 91.0G - - 57% 16% 1.00x ONLINE -
The status:
[dvl@r720-02:~] $ zpool status pool: data01 state: ONLINE scan: scrub repaired 0 in 0 days 00:00:03 with 0 errors on Thu May 20 03:30:41 2021 config: NAME STATE READ WRITE CKSUM data01 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/S59VNS0N809087J_S00 ONLINE 0 0 0 gpt/S59VNJ0N631973D_S01 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/S5B3NDFN807383E_S02 ONLINE 0 0 0 gpt/S5B3NDFN807386P_S03 ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: scrub repaired 0 in 0 days 00:01:35 with 0 errors on Thu May 20 03:32:17 2021 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 errors: No known data errors [dvl@r720-02:~] $
The upgrade begins
[dvl@r720-02:~] $ sudo bectl create -r 13 Created successfully [dvl@r720-02:~] $ sudo bectl mount 13 /var/tmp/BE-13 Mounted successfully on '/var/tmp/BE-13' [dvl@r720-02:~] $ sudo chroot /var/tmp/BE-13 root@r720-02:/ # bash [root@r720-02:/] # mount -t devfs devfs /dev
If any of the above steps fail, stop and find out why. For one of my servers did not have a standard ZFS filesystem layout. It had been manually configured into a BE-compatible layout. Mistakes were made. By me.
I removed the old freebsd-update files. This was where I discovered that the filesystem layout on this host did not conform to standard FreeBSD practices. First, check to verify this directory exists and has content.
[root@r720-02:/] # rm -rf /var/db/freebsd-update
NOTE: if the above directory empties quickly, you may have a problem. I bet the directory does not exist. It should. if it does not, I bet your zroot/var is not properly set up for a BE. Check the directory first. Make sure it exists in this BE.
The first of several freebsd-update commands:
[root@r720-02:/] # mkdir /var/db/freebsd-update [root@r720-02:/] # freebsd-update upgrade -r 13.0-RELEASE src component not installed, skipped Looking up update.FreeBSD.org mirrors... 2 mirrors found. Fetching public key from update2.freebsd.org... done. Fetching metadata signature for 12.2-RELEASE from update2.freebsd.org... done. Fetching metadata index... done. Fetching 2 metadata files... done. Inspecting system... done. The following components of FreeBSD seem to be installed: kernel/generic world/base world/doc world/lib32 The following components of FreeBSD do not seem to be installed: kernel/generic-dbg world/base-dbg world/lib32-dbg Does this look reasonable (y/n)? y Fetching metadata signature for 13.0-RELEASE from update2.freebsd.org... done. Fetching metadata index... done. Fetching 1 metadata patches. done. Applying metadata patches... done. Fetching 1 metadata files... done. .... /bin/ln /bin/ls /bin/mkdir /bin/mv To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".
The next:
[root@r720-02:/] # freebsd-update install src component not installed, skipped Installing updates... Kernel updates have been installed. Please reboot and run "/usr/sbin/freebsd-update install" again to finish installing updates.
NOTE: I did *NOT* reboot, despite the instructions contained above.
[root@r720-02:/] # freebsd-update install src component not installed, skipped Installing updates...sysctl: unknown oid 'user.localbase' Scanning //usr/share/certs/blacklisted for certificates... Scanning //etc/ssl/blacklisted for certificates... Scanning //usr/share/certs/trusted for certificates... Scanning //etc/ssl/certs for certificates... Completing this upgrade requires removing old shared object files. Please rebuild all installed 3rd party software (e.g., programs installed from the ports tree) and then run "/usr/sbin/freebsd-update install" again to finish installing updates.
This is where I deviate from the instructions I was following.
I update from my own repo. Therefore, I updated /usr/local/etc/pkg/repos/local.conf to make this change:
- url: "pkg+http://fedex.int.unixathome.org/packages/122amd64-default-master-list/" + url: "pkg+http://fedex.int.unixathome.org/packages/13amd64-default-master-list/"
You’ll have to decide if that is something you need to do. Generally, if you are using the FreeBSD repos, you do not have to make this change.
[root@r720-02:/] # pkg bootstrap -f The package management tool is not yet installed on your system. Do you want to fetch and install it now? [y/N]: y Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/latest, please wait... Installing pkg-1.16.3... package pkg is already installed, forced install Extracting pkg-1.16.3: 100%
Then I returned to the script:
[root@r720-02:/] # pkg upgrade -f Updating FreeBSD repository catalogue... Fetching meta.conf: 100% 163 B 0.2kB/s 00:01 Fetching packagesite.txz: 100% 6 MiB 6.5MB/s 00:01 Processing entries: 100% FreeBSD repository update completed. 30513 packages processed. All repositories are up to date. Updating database digests format: 100% Checking for upgrades (128 candidates): 100% Processing candidates (128 candidates): 100% The following 138 package(s) will be affected (of 0 checked): New packages to be INSTALLED: mpdecimal: 2.5.1 [FreeBSD] p5-Authen-SASL: 2.16_1 [FreeBSD] p5-Digest-HMAC: 1.03_1 [FreeBSD] p5-GSSAPI: 0.28_1 [FreeBSD] p5-IO-Socket-INET6: 2.72_1 [FreeBSD] p5-IO-Socket-SSL: 2.070 [FreeBSD] p5-Mozilla-CA: 20200520 [FreeBSD] p5-Net-SSLeay: 1.88 [FreeBSD] ...
Next, I check to see if everything upgraded. This is based upon my upgrade to new ports repo recipe.
Remove any unneeded packages:
[root@knew:/] # pkg autoremove
Did everything get upgraded?
[root@knew:/] # pkg query -a '%n %t' | awk -v t=$(date -v -6H +%s) '$2 < t { print $1 }'
beadm
p5-Crypt-CBC
I removed both of those packages, because I don't use either.
[root@r720-02:/usr/local/etc] # freebsd-update install src component not installed, skipped Installing updates... done. [root@r720-02:/usr/local/etc] # [root@r720-02:/usr/local/etc] # exit exit root@r720-02:/ # exit exit [dvl@r720-02:~] $ sudo umount /var/tmp/BE-13/dev [dvl@r720-02:~] $ sudo bectl activate 13 Successfully activated boot environment 13 [dvl@r720-02:~] $ mount | grep /var/tmp zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls) [dvl@r720-02:~] $
There, that looks better. Checking:
[dvl@r720-02:~] $ bectl list BE Active Mountpoint Space Created 13 R - 10.5G 2021-05-21 16:23 default N / 936K 2017-04-12 21:36 [dvl@r720-02:~] $
Let's make that a temporary boot.
NOTE: I now believe this is not the way I want to proceed in future. After the upcoming zpool upgrade the old BE will be unable to boot. I used the temp activate so that if I booted once, I wouldn't have to much about with the boot screen, selecting the old BE, etc. Thinking about this now, the boot-once is harder for me in this situation. I'll boot it, oh good, this is working, let me trying this.. Some days later, I've forgotten about the boot-once and reboot. I think not-using boot-once is logically less complicated.
[dvl@r720-02:~] $ sudo bectl activate -t 13 Successfully activated boot environment 13 for next boot [dvl@r720-02:~] $ bectl list BE Active Mountpoint Space Created 13 R - 10.5G 2021-05-21 16:23 default N / 936K 2017-04-12 21:36 [dvl@r720-02:~] $
Reboot.
[dvl@r720-02:~] $ uname -a FreeBSD r720-02.unixathome.org 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 04:24:09 UTC 2021 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
Success!
Now it's time to update the zpool. BE WARNED: at present, the system does not mention that you need to update your boot code. You need to do that. In previous versions, you got this notice:
If you boot from pool 'zroot', don't forget to update boot code.
Assuming you use GPT partitioning and da0 is your boot disk
the following command will do it:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
As you can see here, that does not happen now. An issue has been raised and a FreeBSD PR created.
zpools before
These are my zpools right after booting in FreeBSD 13:
[dvl@r720-02:~] $ zpool status pool: data01 state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 00:00:03 with 0 errors on Thu May 20 03:30:41 2021 config: NAME STATE READ WRITE CKSUM data01 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/S59VNS0N809087J_S00 ONLINE 0 0 0 gpt/S59VNJ0N631973D_S01 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/S5B3NDFN807383E_S02 ONLINE 0 0 0 gpt/S5B3NDFN807386P_S03 ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 00:01:35 with 0 errors on Thu May 20 03:32:17 2021 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 errors: No known data errors
Checkpoint
First, I set a checkpoint, just in case:
[dvl@r720-02:~] $ sudo zpool checkpoint zroot [dvl@r720-02:~] $ zpool status zroot pool: zroot state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 00:01:35 with 0 errors on Thu May 20 03:32:17 2021 checkpoint: created Fri May 21 17:26:03 2021, consumes 648K config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 errors: No known data errors
zpool upgrade
The upgrade:
[dvl@r720-02:~] $ sudo zpool upgrade zroot This system supports ZFS pool feature flags. Enabled the following features on 'zroot': userobj_accounting encryption project_quota resilver_defer bookmark_v2 redaction_bookmarks redacted_datasets bookmark_written log_spacemap livelist device_rebuild zstd_compress draid [dvl@r720-02:~] $ zpool status pool: data01 state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 00:00:03 with 0 errors on Thu May 20 03:30:41 2021 config: NAME STATE READ WRITE CKSUM data01 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/S59VNS0N809087J_S00 ONLINE 0 0 0 gpt/S59VNJ0N631973D_S01 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/S5B3NDFN807383E_S02 ONLINE 0 0 0 gpt/S5B3NDFN807386P_S03 ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: scrub repaired 0B in 00:01:35 with 0 errors on Thu May 20 03:32:17 2021 checkpoint: created Fri May 21 17:26:03 2021, consumes 16.1M config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 errors: No known data errors
Note there is no notice to update the bootcode. This has trapped, and will trap, people.
Applying the bootcode
This works for my system. Inspect your own partitions to be sure:
[dvl@r720-02:~] $ gpart show ada0 ada1
=> 40 234441568 ada0 GPT (112G)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 4194304 2 freebsd-swap (2.0G)
4196352 230244352 3 freebsd-zfs (110G)
234440704 904 - free - (452K)
=> 40 234441568 ada1 GPT (112G)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 4194304 2 freebsd-swap (2.0G)
4196352 230244352 3 freebsd-zfs (110G)
234440704 904 - free - (452K)
[dvl@r720-02:~] $
On EFI systems, or systems setup for 'BOTH', the freebsd-boot partition is #2
[dvl@r720-02:~] $ sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 partcode written to ada0p1 bootcode written to ada0 [dvl@r720-02:~] $ sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1 partcode written to ada1p1 bootcode written to ada1 [dvl@r720-02:~] $
Reboot and remove the checkpoints
After a succcesful reboot, I removed the checkpoints:
[dvl@r720-02:~] $ sudo zpool checkpoint -d zroot [dvl@r720-02:~] $ sudo zpool checkpoint -d data01 [dvl@r720-02:~] $ zpool status pool: data01 state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 00:00:04 with 0 errors on Fri May 21 17:54:11 2021 config: NAME STATE READ WRITE CKSUM data01 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/S59VNS0N809087J_S00 ONLINE 0 0 0 gpt/S59VNJ0N631973D_S01 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/S5B3NDFN807383E_S02 ONLINE 0 0 0 gpt/S5B3NDFN807386P_S03 ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: scrub repaired 0B in 00:01:38 with 0 errors on Fri May 21 17:28:14 2021 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 errors: No known data errors [dvl@r720-02:~] $
Oh wait. I see I didn't upgrade data01
Let's do that.
Upgrading one zpool
This is me, upgrading one zpool from which I do not boot:
[dvl@r720-02:~] $ sudo zpool checkpoint data01 [dvl@r720-02:~] $ sudo zpool upgrade data01 This system supports ZFS pool feature flags. Enabled the following features on 'data01': userobj_accounting encryption project_quota resilver_defer bookmark_v2 redaction_bookmarks redacted_datasets bookmark_written log_spacemap livelist device_rebuild zstd_compress draid [dvl@r720-02:~] $ sudo zpool checkpoint -d data01 [dvl@r720-02:~] $ zpool status data01 pool: data01 state: ONLINE scan: scrub repaired 0B in 00:00:04 with 0 errors on Fri May 21 18:08:28 2021 config: NAME STATE READ WRITE CKSUM data01 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/S59VNS0N809087J_S00 ONLINE 0 0 0 gpt/S59VNJ0N631973D_S01 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/S5B3NDFN807383E_S02 ONLINE 0 0 0 gpt/S5B3NDFN807386P_S03 ONLINE 0 0 0 errors: No known data errors [dvl@r720-02:~] $
Done. :)
man page differences
One great thing about FreeBSD 13 - the man pages for zfs and zpool are split up.
Try man zpool-create or man zfs-create
I find this much easier than the previous situation.
I also like that /etc/motd is gone.
And that newsyslog.conf files need to end in .conf:
$ tail -2 /etc/newsyslog.conf/etc/newsyslog.conf.d/[!.]*.conf /usr/local/etc/newsyslog.conf.d/[!.]*.conf $
I appreciate this change.
