After yesterday’s failure to shrink a parition-based zpool.
After reading this post, you want also want to read this other successful post.
You will recall that expedition ended with:
dvl@r730-04:~ $ zpool add zroot mirror /dev/da2p3 /dev/da3p3 cannot use '/dev/da2p3': must be a block device or regular file
Today, I’m going to try the same thing with whole disks, just to see if that works.
You might also want to read Moving a zpool to smaller drives #ZFS #FreeBSD
zroot
For my own reference, I’m booting the r730-04 host off a pair of SATADOM devices:
[14:00 pro05 dvl ~/Downloads] % ssh dvl@10.55.0.215
Last login: Fri Nov 21 18:41:43 2025 from pro05.startpoint.vpn.unixathome.org
dvl@r730-04:~ $ zpool status
pool: zroot
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p3 ONLINE 0 0 0
ada1p3 ONLINE 0 0 0
errors: No known data errors
dvl@r730-04:~ $ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 83G 691M 82.3G - - 0% 0% 1.00x ONLINE -
dvl@r730-04:~ $ gpart show ada0 ada1
=> 40 242255584 ada0 GPT (116G)
40 532480 1 efi (260M)
532520 2008 - free - (1.0M)
534528 67108864 2 freebsd-swap (32G)
67643392 174610432 3 freebsd-zfs (83G)
242253824 1800 - free - (900K)
=> 40 242255584 ada1 GPT (116G)
40 532480 1 efi (260M)
532520 2008 - free - (1.0M)
534528 67108864 2 freebsd-swap (32G)
67643392 174610432 3 freebsd-zfs (83G)
242253824 1800 - free - (900K)
dvl@r730-04:~ $
First failed test – drives are too small
The other drives are:
root@r730-04:/home/dvl # diskinfo -v /dev/mfisyspd0 /dev/mfisyspd0 512 # sectorsize 1000204886016 # mediasize in bytes (932G) 1953525168 # mediasize in sectors 0 # stripesize 0 # stripeoffset 121601 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. # Disk descr. # Disk ident. # Attachment No # TRIM/UNMAP support Unknown # Rotation rate in RPM root@r730-04:/home/dvl # diskinfo -v /dev/mfisyspd1 /dev/mfisyspd1 512 # sectorsize 1000204886016 # mediasize in bytes (932G) 1953525168 # mediasize in sectors 0 # stripesize 0 # stripeoffset 121601 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. # Disk descr. # Disk ident. # Attachment No # TRIM/UNMAP support Unknown # Rotation rate in RPM root@r730-04:/home/dvl # diskinfo -v /dev/mfisyspd2 /dev/mfisyspd2 512 # sectorsize 32017047552 # mediasize in bytes (30G) 62533296 # mediasize in sectors 0 # stripesize 0 # stripeoffset 3892 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. # Disk descr. # Disk ident. # Attachment No # TRIM/UNMAP support Unknown # Rotation rate in RPM root@r730-04:/home/dvl # diskinfo -v /dev/mfisyspd3 /dev/mfisyspd3 512 # sectorsize 32017047552 # mediasize in bytes (30G) 62533296 # mediasize in sectors 0 # stripesize 0 # stripeoffset 3892 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. # Disk descr. # Disk ident. # Attachment No # TRIM/UNMAP support Unknown # Rotation rate in RPM root@r730-04:/home/dvl #
The test
Here I create the test zpool:
root@r730-04:/home/dvl # zpool create ztest mirror /dev/mfisyspd0 /dev/mfisyspd1 root@r730-04:/home/dvl # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 83G 1.01G 82.0G - - 0% 1% 1.00x ONLINE - ztest 928G 384K 928G - - 0% 0% 1.00x ONLINE -
It uses the 1T disks.
Next, I add a file to the zpool and add in the other two drives.
root@r730-04:/home/dvl # touch /ztest/things root@r730-04:/home/dvl # zpool add ztest mirror /dev/mfisyspd2 mfisyspd3 root@r730-04:/home/dvl # zpool status ztest pool: ztest state: ONLINE config: NAME STATE READ WRITE CKSUM ztest ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 mfisyspd0 ONLINE 0 0 0 mfisyspd1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 mfisyspd2 ONLINE 0 0 0 mfisyspd3 ONLINE 0 0 0 errors: No known data errors
Yes, the file is there:
root@r730-04:/home/dvl # ls -l /ztest/ total 1 -rw-r--r-- 1 root wheel 0 Nov 21 20:55 things
Let’s try removal:
root@r730-04:/home/dvl # zpool list ztest NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ztest 958G 612K 957G - - 0% 0% 1.00x ONLINE - root@r730-04:/home/dvl # zpool remove ztest mirror-0 cannot remove mirror-0: out of space
OK, fair enough. It’s try it again, with smaller drives.
reset
Let’s delete the zpool and tidy up:
root@r730-04:/home/dvl # zpool destroy ztest root@r730-04:/home/dvl # zpool labelclear -f /dev/mfisyspd0 root@r730-04:/home/dvl # zpool labelclear -f /dev/mfisyspd1 root@r730-04:/home/dvl # zpool labelclear -f /dev/mfisyspd2 root@r730-04:/home/dvl # zpool labelclear -f /dev/mfisyspd3 root@r730-04:/home/dvl #
I’m clearing the zpool labels to avoid future reluctance of ZFS to use those drives in new zpools.
Test #2 – with 30G and 240G drives
I removed the 1T drives and replaced them with 204G drives.
root@r730-04:/home/dvl # diskinfo -v /dev/mfisyspd3 /dev/mfisyspd3 512 # sectorsize 32017047552 # mediasize in bytes (30G) 62533296 # mediasize in sectors 0 # stripesize 0 # stripeoffset 3892 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. # Disk descr. # Disk ident. # Attachment No # TRIM/UNMAP support Unknown # Rotation rate in RPM root@r730-04:/home/dvl # diskinfo -v /dev/mfisyspd0 /dev/mfisyspd0 512 # sectorsize 240057409536 # mediasize in bytes (224G) 468862128 # mediasize in sectors 0 # stripesize 0 # stripeoffset 29185 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. # Disk descr. # Disk ident. # Attachment No # TRIM/UNMAP support Unknown # Rotation rate in RPM
I created the test pool:
root@r730-04:/home/dvl # zpool create ztest mirror /dev/mfisyspd0 /dev/mfisyspd1 root@r730-04:/home/dvl # zpool list ztest NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ztest 222G 372K 222G - - 0% 0% 1.00x ONLINE -
Add in the second pair of drives:
root@r730-04:/home/dvl # zpool add ztest mirror /dev/mfisyspd2 mfisyspd3 root@r730-04:/home/dvl # zpool status ztest pool: ztest state: ONLINE config: NAME STATE READ WRITE CKSUM ztest ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 mfisyspd0 ONLINE 0 0 0 mfisyspd1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 mfisyspd2 ONLINE 0 0 0 mfisyspd3 ONLINE 0 0 0 errors: No known data errors root@r730-04:/home/dvl #
And we have files:
root@r730-04:/home/dvl # touch /ztest/things root@r730-04:/home/dvl # ls -l /ztest/things -rw-r--r-- 1 root wheel 0 Nov 22 14:15 /ztest/things
Removal
Here. We. Go.
root@r730-04:/home/dvl # zpool remove ztest mirror-0 root@r730-04:/home/dvl # root@r730-04:/home/dvl # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 83G 1.01G 82.0G - - 0% 1% 1.00x ONLINE - ztest 29.5G 648K 29.5G - - 0% 0% 1.00x ONLINE - root@r730-04:/home/dvl # ls -l /ztest/things -rw-r--r-- 1 root wheel 0 Nov 22 14:15 /ztest/things root@r730-04:/home/dvl #
Success.
So this works with drives, but not partitions.
Let’s try it again, with partitions, and these same drives.
FYI, these were the log messages created during that remove:
Nov 22 14:17:58 r730-04 kernel: mfi0: 94057997 (817136279s/0x0002/info) - Inserted: PD 07(e0x20/s7) Nov 22 14:17:58 r730-04 kernel: mfisyspd3 numa-domain 0 on mfi0 Nov 22 14:17:58 r730-04 kernel: mfisyspd3: 30533MB (62533296 sectors) SYSPD volume (deviceid: 7) Nov 22 14:17:58 r730-04 kernel: mfisyspd3: SYSPD volume attached Nov 22 14:17:58 r730-04 kernel: mfi0: 94057998 (817136279s/0x0002/info) - Inserted: PD 07(e0x20/s7) Info: enclPd=20, scsiType=0, portMap=01, sasAddr=4433221103000000,0000000000000000 Nov 22 14:17:58 r730-04 kernel: mfi0: 94057999 (817136279s/0x0002/WARN) - PD 07(e0x20/s7) is not a certified drive Nov 22 14:19:19 r730-04 kernel: GEOM: mfisyspd0: the primary GPT table is corrupt or invalid. Nov 22 14:19:19 r730-04 kernel: GEOM: mfisyspd0: using the secondary instead -- recovery strongly advised. Nov 22 14:19:19 r730-04 kernel: GEOM: mfisyspd1: the primary GPT table is corrupt or invalid. Nov 22 14:19:19 r730-04 kernel: GEOM: mfisyspd1: using the secondary instead -- recovery strongly advised.
Once more, with partitions.
First, clean up:
root@r730-04:/home/dvl # zpool destroy ztest root@r730-04:/home/dvl # zpool labelclear -f /dev/mfisyspd0 root@r730-04:/home/dvl # zpool labelclear -f /dev/mfisyspd1 root@r730-04:/home/dvl # zpool labelclear -f /dev/mfisyspd2 root@r730-04:/home/dvl # zpool labelclear -f /dev/mfisyspd3 root@r730-04:/home/dvl #
Next, we partitions all drives, more or less, the same way:
root@r730-04:/home/dvl # gpart create -s gpt /dev/mfisyspd0
mfisyspd0 created
root@r730-04:/home/dvl # gpart create -s gpt /dev/mfisyspd1
mfisyspd1 created
root@r730-04:/home/dvl # gpart create -s gpt /dev/mfisyspd2
mfisyspd2 created
root@r730-04:/home/dvl # gpart create -s gpt /dev/mfisyspd3
mfisyspd3 created
root@r730-04:/home/dvl # gpart add -t freebsd-zfs /dev/mfisyspd0
mfisyspd0p1 added
root@r730-04:/home/dvl # gpart add -t freebsd-zfs /dev/mfisyspd1
mfisyspd1p1 added
root@r730-04:/home/dvl # gpart add -t freebsd-zfs /dev/mfisyspd2
mfisyspd2p1 added
root@r730-04:/home/dvl # gpart add -t freebsd-zfs /dev/mfisyspd3
mfisyspd3p1 added
root@r730-04:/home/dvl # gpart show mfisyspd0 mfisyspd1 mfisyspd2 mfisyspd3
=> 40 468862048 mfisyspd0 GPT (224G)
40 468862048 1 freebsd-zfs (224G)
=> 40 468862048 mfisyspd1 GPT (224G)
40 468862048 1 freebsd-zfs (224G)
=> 40 62533216 mfisyspd2 GPT (30G)
40 62533216 1 freebsd-zfs (30G)
=> 40 62533216 mfisyspd3 GPT (30G)
40 62533216 1 freebsd-zfs (30G)
root@r730-04:/home/dvl #
We create the larger zpool:
root@r730-04:/home/dvl # zpool create ztest mirror /dev/mfisyspd0p1 /dev/mfisyspd1p1 root@r730-04:/home/dvl # zpool list ztest NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ztest 222G 360K 222G - - 0% 0% 1.00x ONLINE -
Then we add in the two smaller drives:
root@r730-04:/home/dvl # zpool add ztest mirror /dev/mfisyspd2p1 mfisyspd3p1 root@r730-04:/home/dvl # zpool status ztest pool: ztest state: ONLINE config: NAME STATE READ WRITE CKSUM ztest ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 mfisyspd0p1 ONLINE 0 0 0 mfisyspd1p1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 mfisyspd2p1 ONLINE 0 0 0 mfisyspd3p1 ONLINE 0 0 0 errors: No known data errors
All commands are the same, just with p1 appended to the devices.
Let’s remove the large drives:
root@r730-04:/home/dvl # touch /ztest/things root@r730-04:/home/dvl # ls -l /ztest/things -rw-r--r-- 1 root wheel 0 Nov 22 14:27 /ztest/things root@r730-04:/home/dvl # zpool remove ztest mirror-0 root@r730-04:/home/dvl # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 83G 1.01G 82.0G - - 0% 1% 1.00x ONLINE - ztest 29.5G 1.08M 29.5G - - 0% 0% 1.00x ONLINE - root@r730-04:/home/dvl #
Good. So it DOES work with partitions.
So why did my previous attempt fail?
