Upgrading an NVMe zpool from 222G to 1TB drives

The FreshPorts dev, test, and stage websites are hosted on a server in my basement. Each instance consists of two jails:

  1. an ingress node – for pulling in new commits (and other data) into the database.
  2. a webserver node – for displaying the web pages.

The new drives:

Two 1TB Samsung 980 PRO NVMe drives in boxes
These arrived on Saturday. They will replace a 220GB zpool which frequently fills up.

Sometimes the zpool gets too close to full. I tweeted about one incident on March 2021. I held hope that some spare SSDs might be the answer. Eventually, I bought new SSDs after another full incident (I’ve not mentioned them all).

The first drive added to a spare PCIe interface:

An NVMe card has been attached to a PCIe card
Ready to install

They arrived on Saturday and I installed them this morning.

The slots before:

Main board slots: before. Two empty slots.
The main board slots: before

This post is about upgrading that zpool to contain the new devices.

In this post:

The slots after:

Main board slots: after - the new NVMe drive is second from the bottom
The main board slots: after

What uses all this space?

The ingress nodes are actually the storage hungry side.

[slocum dan ~] % zfs list tank_fast/data/dev-ingress01 tank_fast/data/dev-nginx01 nvd/freshports/dev nvd/freshports/dev-nginx0
NAME                           USED  AVAIL     REFER  MOUNTPOINT
nvd/freshports/dev            5.10G  62.7G       23K  /jails/dev-ingress01/var/db/freshports
nvd/freshports/dev-nginx01    98.4M  62.7G       24K  none
tank_fast/data/dev-ingress01  12.7G   172G       96K  none
tank_fast/data/dev-nginx01    57.1M   172G       96K  none

That’s 17GB for the ingress, compared with only 145MB for the website. Most of the data for the ingress node is the repos and the old text commits still on disk.

[slocum dan ~] % zfs list -r tank_fast/data/dev-ingress01 nvd/freshports/dev
NAME                                                                    USED  AVAIL     REFER  MOUNTPOINT
nvd/freshports/dev                                                     5.10G  62.7G       23K  /jails/dev-ingress01/var/db/freshports
nvd/freshports/dev/cache                                                106K  62.7G     35.5K  /jails/dev-ingress01/var/db/freshports/cache
nvd/freshports/dev/cache/html                                            42K  62.7G       42K  /jails/dev-ingress01/var/db/freshports/cache/html
nvd/freshports/dev/cache/spooling                                        28K  62.7G       28K  /jails/dev-ingress01/var/db/freshports/cache/spooling
nvd/freshports/dev/ingress                                             5.08G  62.7G       23K  /jails/dev-ingress01/var/db/ingress
nvd/freshports/dev/ingress/message-queues                              1.80M  62.7G     1.33M  /jails/dev-ingress01/var/db/ingress/message-queues
nvd/freshports/dev/ingress/repos                                       5.08G  62.7G     2.28G  /jails/dev-ingress01/var/db/ingress/repos
nvd/freshports/dev/ingress/repos/src                                   2.80G  62.7G     2.80G  /jails/dev-ingress01/var/db/ingress/repos/src
nvd/freshports/dev/message-queues                                      16.1M  62.7G     15.2M  /jails/dev-ingress01/var/db/freshports/message-queues
tank_fast/data/dev-ingress01                                           12.7G   172G       96K  none
tank_fast/data/dev-ingress01/messages                                  12.7G   172G       96K  none
tank_fast/data/dev-ingress01/messages/archive                          12.7G   172G     12.7G  /jails/dev-ingress01/var/db/freshports/message-queues/archive
tank_fast/data/dev-ingress01/usr.local.lib.perl5.site_perl.FreshPorts  3.12M   172G     3.12M  /jails/dev-ingress01/usr/local/lib/perl5/site_perl/FreshPorts
tank_fast/data/dev-ingress01/usr.local.libexec.freshports              2.18M   172G     2.18M  /jails/dev-ingress01/usr/local/libexec/freshports

The biggest eater of space is the jail within the jail, specifically, the distfiles directory.

[slocum dan ~] % sudo du -ch -d 1 /jails/dev-ingress01/jails/freshports/usr/ports
 19M	/jails/dev-ingress01/jails/freshports/usr/ports/games
 22M	/jails/dev-ingress01/jails/freshports/usr/ports/net
 55K	/jails/dev-ingress01/jails/freshports/usr/ports/base
5.7M	/jails/dev-ingress01/jails/freshports/usr/ports/java
 12M	/jails/dev-ingress01/jails/freshports/usr/ports/cad
 15M	/jails/dev-ingress01/jails/freshports/usr/ports/databases
136K	/jails/dev-ingress01/jails/freshports/usr/ports/vietnamese
 13M	/jails/dev-ingress01/jails/freshports/usr/ports/audio
472K	/jails/dev-ingress01/jails/freshports/usr/ports/korean
219K	/jails/dev-ingress01/jails/freshports/usr/ports/Tools
1.2M	/jails/dev-ingress01/jails/freshports/usr/ports/french
 21M	/jails/dev-ingress01/jails/freshports/usr/ports/graphics
4.4M	/jails/dev-ingress01/jails/freshports/usr/ports/japanese
235K	/jails/dev-ingress01/jails/freshports/usr/ports/russian
 16M	/jails/dev-ingress01/jails/freshports/usr/ports/textproc
2.2M	/jails/dev-ingress01/jails/freshports/usr/ports/astro
 48K	/jails/dev-ingress01/jails/freshports/usr/ports/ukrainian
 61K	/jails/dev-ingress01/jails/freshports/usr/ports/arabic
4.8M	/jails/dev-ingress01/jails/freshports/usr/ports/emulators
 41G	/jails/dev-ingress01/jails/freshports/usr/ports/distfiles
 65M	/jails/dev-ingress01/jails/freshports/usr/ports/www
3.2M	/jails/dev-ingress01/jails/freshports/usr/ports/comms
4.5M	/jails/dev-ingress01/jails/freshports/usr/ports/deskutils
 11M	/jails/dev-ingress01/jails/freshports/usr/ports/misc
909K	/jails/dev-ingress01/jails/freshports/usr/ports/news
100K	/jails/dev-ingress01/jails/freshports/usr/ports/x11-servers
406K	/jails/dev-ingress01/jails/freshports/usr/ports/accessibility
8.4M	/jails/dev-ingress01/jails/freshports/usr/ports/editors
1.0M	/jails/dev-ingress01/jails/freshports/usr/ports/ftp
 11M	/jails/dev-ingress01/jails/freshports/usr/ports/science
 13M	/jails/dev-ingress01/jails/freshports/usr/ports/mail
2.7M	/jails/dev-ingress01/jails/freshports/usr/ports/x11-fonts
 25M	/jails/dev-ingress01/jails/freshports/usr/ports/security
2.8M	/jails/dev-ingress01/jails/freshports/usr/ports/net-p2p
4.4M	/jails/dev-ingress01/jails/freshports/usr/ports/x11-toolkits
 58K	/jails/dev-ingress01/jails/freshports/usr/ports/hebrew
7.3M	/jails/dev-ingress01/jails/freshports/usr/ports/net-mgmt
3.7M	/jails/dev-ingress01/jails/freshports/usr/ports/net-im
 26K	/jails/dev-ingress01/jails/freshports/usr/ports/Keywords
8.6M	/jails/dev-ingress01/jails/freshports/usr/ports/x11
 35K	/jails/dev-ingress01/jails/freshports/usr/ports/portuguese
 13K	/jails/dev-ingress01/jails/freshports/usr/ports/.hooks
3.6M	/jails/dev-ingress01/jails/freshports/usr/ports/biology
664K	/jails/dev-ingress01/jails/freshports/usr/ports/x11-fm
158K	/jails/dev-ingress01/jails/freshports/usr/ports/german
501K	/jails/dev-ingress01/jails/freshports/usr/ports/polish
 31K	/jails/dev-ingress01/jails/freshports/usr/ports/hungarian
397K	/jails/dev-ingress01/jails/freshports/usr/ports/x11-clocks
 17M	/jails/dev-ingress01/jails/freshports/usr/ports/math
3.6M	/jails/dev-ingress01/jails/freshports/usr/ports/chinese
1.2M	/jails/dev-ingress01/jails/freshports/usr/ports/Mk
2.3M	/jails/dev-ingress01/jails/freshports/usr/ports/dns
7.0M	/jails/dev-ingress01/jails/freshports/usr/ports/finance
 24M	/jails/dev-ingress01/jails/freshports/usr/ports/sysutils
2.5M	/jails/dev-ingress01/jails/freshports/usr/ports/x11-wm
 82M	/jails/dev-ingress01/jails/freshports/usr/ports/devel
1.2M	/jails/dev-ingress01/jails/freshports/usr/ports/benchmarks
8.5M	/jails/dev-ingress01/jails/freshports/usr/ports/multimedia
1.2M	/jails/dev-ingress01/jails/freshports/usr/ports/shells
1.6M	/jails/dev-ingress01/jails/freshports/usr/ports/irc
1.2G	/jails/dev-ingress01/jails/freshports/usr/ports/.git
821K	/jails/dev-ingress01/jails/freshports/usr/ports/ports-mgmt
5.7M	/jails/dev-ingress01/jails/freshports/usr/ports/print
494K	/jails/dev-ingress01/jails/freshports/usr/ports/x11-drivers
 63M	/jails/dev-ingress01/jails/freshports/usr/ports/x11-themes
513K	/jails/dev-ingress01/jails/freshports/usr/ports/Templates
 23M	/jails/dev-ingress01/jails/freshports/usr/ports/lang
1.4M	/jails/dev-ingress01/jails/freshports/usr/ports/converters
2.1M	/jails/dev-ingress01/jails/freshports/usr/ports/archivers
 43G	/jails/dev-ingress01/jails/freshports/usr/ports
 43G	total

FreshPort uses the distfiles to extract information about the port. It doesn’t delete them after using them to save downloading them again.

The drives in the system

After the system booted up, I found this:

[slocum dan ~] % sysctl kern.disks
kern.disks: da11 da10 da9 da8 da7 da6 da5 da4 da3 da2 da1 da0 ada1 ada0 nvd2 nvd1 nvd0
[slocum dan ~] % gpart show nvd0 nvd1 nvd2                                                                                                                               12:55:41
=>       40  488397088  nvd0  GPT  (233G)
         40  488397088     1  freebsd-zfs  (233G)

gpart: No such geom: nvd1.
[slocum dan ~] % gpart show nvd2
gpart: No such geom: nvd2.
[slocum dan ~] % 

This means neither nvd1 nor nvd2 have partitions.

What does dmesg have to say about this?

[slocum dan ~] % egrep -e 'nvd[0,1,2]' /var/run/dmesg.boot
nvd0: <WDC WDS250G2B0C-00PXH0> NVMe namespace
nvd0: 238475MB (488397168 512 byte sectors)
nvd1: <BPXP> NVMe namespace
nvd1: 228936MB (468862128 512 byte sectors)
nvd2: <Samsung SSD 980 PRO with Heatsink 1TB> NVMe namespace
nvd2: 953869MB (1953525168 512 byte sectors)

The new drive is nvd2. So what’s up with nvd1 not having any partitions. It’s part of the existing (and cleverly named) nvd zpool. See my recent post about my rediscovery of that situation.

smartctl

Let’s look at the drive using smartctl:

[slocum dan ~] % sudo smartctl -a /dev/nvme2                                                                                                                             13:20:29
smartctl 7.3 2022-02-28 r5338 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 980 PRO with Heatsink 1TB
Serial Number:                      [redacted]
Firmware Version:                   4B2QGXA7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 1,000,204,886,016 [1.00 TB]
Unallocated NVM Capacity:           0
Controller ID:                      6
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,000,204,886,016 [1.00 TB]
Namespace 1 Utilization:            40,960 [40.9 KB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 b22140998d
Local Time is:                      Tue Oct 18 13:20:37 2022 UTC
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x0057):     Comp Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f):         S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         128 Pages
Warning  Comp. Temp. Threshold:     82 Celsius
Critical Comp. Temp. Threshold:     85 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     8.49W       -        -    0  0  0  0        0       0
 1 +     4.48W       -        -    1  1  1  1        0     200
 2 +     3.18W       -        -    2  2  2  2        0    1000
 3 -   0.0400W       -        -    3  3  3  3     2000    1200
 4 -   0.0050W       -        -    4  4  4  4      500    9500

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        41 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    12 [6.14 MB]
Data Units Written:                 2 [1.02 MB]
Host Read Commands:                 1,619
Host Write Commands:                120
Controller Busy Time:               0
Power Cycles:                       1
Power On Hours:                     1
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               41 Celsius
Temperature Sensor 2:               46 Celsius

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

Partitioning the new drive

This is similar to previous posts:

  1. create a partitioning schem
  2. add a ZFS partition
  3. tell zfs to replace one drive with another (i.e. resilver)

Let’s get start by creating the partitioning scheme.

[slocum dan ~] % sudo gpart create -s gpt nvd2
nvd2 created
[slocum dan ~] % gpart show nvd2
=>        40  1953525088  nvd2  GPT  (932G)
          40  1953525088        - free -  (932G)

[slocum dan ~] % 

Then create a ZFS partition:

[slocum dan ~] % sudo gpart add -t freebsd-zfs -a 4K nvd2
nvd2p1 added
[slocum dan ~] % gpart show nvd2
=>        40  1953525088  nvd2  GPT  (932G)
          40  1953525088     1  freebsd-zfs  (932G)

[slocum dan ~] % 

I want to point out that I’m going against my own advice to not give ZFS your entire drive, but to save a few MB.

Let’s fix that. Through trial and error, because I didn’t do the correct math, I came up with this:

[slocum dan ~] % sudo gpart delete -i 1 nvd2
nvd2p1 deleted
[slocum dan ~] % sudo gpart add -t freebsd-zfs -a 4K -s 953850M -l Drive_Serial_Number nvd2
nvd2p1 added
[slocum dan ~] % gpart show nvd2
=>        40  1953525088  nvd2  GPT  (932G)
          40  1953484800     1  freebsd-zfs  (931G)
  1953484840       40288        - free -  (20M)

[slocum dan ~] % 

That unused 20MB is my insurance plan. Should I need to replace this with another 1TB drive, it can be short by up to 20MB.

Is this where someone tells me I should have chosen a specific size because this size will cause some problem?

I also supplied the drive serial number as the label – this can be useful when replacing drives – it helps you identify which drive to remove. I should also label the drive. I’m not sure the serial number is printed on the drive. You can view that label with this command:

[slocum dan ~] % gpart show -l nvd2
=>        40  1953525088  nvd2  GPT  (932G)
          40  1953484800     1  Drive_Serial_Number  (931G)
  1953484840       40288        - free -  (20M)

[slocum dan ~] % 

Next, add this drive into the zpool. That deserves a separate section.

Adding the drive into the zpool

Why did I add in the new drive and not remove the old drive? Safety. When you can, it is best to do that. It keeps the existing zpool in optimal condition. If I removed one of the drives and just swapped in the new drive, the zpool would be degraded. With this approach, that never happens.

This command always gives me grief. It’s the commands I use so rarely that cause this. After issuing the following command, I immediately issued the status command, just to see what was happening.

[slocum dan ~] % sudo zpool replace nvd diskid/DISK-C26D07890DC400015192p1 nvd2p1

I will outline the arguments to zpool replace

  • nvd – the name of the zpool
  • diskid/DISK-C26D07890DC400015192p1 – the vdev which is leaving the zpool
  • nvd2p1 – the vdev which is coming into the zpool
[slocum dan ~] % zpool status nvd
  pool: nvd
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Oct 18 13:37:34 2022
	16.2G scanned at 1.48G/s, 6.76M issued at 629K/s, 152G total
	8.22M resilvered, 0.00% done, no estimated completion time
config:

	NAME                                      STATE     READ WRITE CKSUM
	nvd                                       ONLINE       0     0     0
	  mirror-0                                ONLINE       0     0     0
	    nvd0p1                                ONLINE       0     0     0
	    replacing-1                           ONLINE       0     0     0
	      diskid/DISK-C26D07890DC400015192p1  ONLINE       0     0     0
	      nvd2p1                              ONLINE       0     0     0  (resilvering)

errors: No known data errors
[slocum dan ~] % 

Checking in on progress:

[slocum dan ~] % zpool status nvd
  pool: nvd
 state: ONLINE
  scan: resilvered 154G in 00:03:36 with 0 errors on Tue Oct 18 13:41:10 2022
config:

	NAME        STATE     READ WRITE CKSUM
	nvd         ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nvd0p1  ONLINE       0     0     0
	    nvd2p1  ONLINE       0     0     0

errors: No known data errors
[slocum dan ~] % 

It’s done. In the time it took me to type the above, it finished.

Next, I repeat the process with the second drive. That will be later today, perhaps at lunch.

Over lunch: which drive to pull?

I have to pull nvd1 from the nvd zpool. Which one is that? Based on the dmesg shown earlier, I know it’s the BPXP drive, and not the WDC drive. If I didn’t have brand / model names to distinguish them, I’d hope serial numbers were visible. Or perhaps this might help:

[slocum dan ~] % pciconf -lv
...
nvme0@pci0:2:0:0:       class=0x010802 rev=0x01 hdr=0x00 vendor=0x15b7 device=0x5019 subvendor=0x15b7 subdevice=0x5019
    vendor     = 'Sandisk Corp'
    class      = mass storage
    subclass   = NVM
...
nvme1@pci0:5:0:0:       class=0x010802 rev=0x01 hdr=0x00 vendor=0x1987 device=0x5012 subvendor=0x1987 subdevice=0x5012
    vendor     = 'Phison Electronics Corporation'
    device     = 'E12 NVMe Controller'
    class      = mass storage
    subclass   = NVM
nvme2@pci0:6:0:0:       class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d device=0xa80a subvendor=0x144d subdevice=0xa801
    vendor     = 'Samsung Electronics Co Ltd'
    device     = 'NVMe SSD Controller PM9A1/PM9A3/980PRO'
    class      = mass storage
    subclass   = NVM

Not all output is shown. But I can see that the drive I want to remove is in slot 5, and not slots 2 or 6.

Let’s go see if that’s right.

No, it wasn’t. I don’t know what those number are, but they don’t match up with physical slots, unless they are not numbers consecutively.

I did find the BPXP drive. The Samsung drive would not fit; the screw was not long enough.

Photo of nvd1 / BPXP NVMe drive on a PCIe slot after being removed from the server.
The first removed NVMe drive

When I removed the WDC drive, I noticed the elastic bands had broken. They were holding the heatsink against the drive. Fortunately, the adhesive was still doing its job and the heatsink had not fallen, perhaps against the other card, causing a short.

The Samsung was mounted on the other card, the one with the WDC drive. The WDC drive was moved to the card which had contained the BPXP drive. In the photo, the mounting screw for the BPXP drive is not yet installed. I did attach it before installing the card in the server.

The broken elastics are at the bottom right of the photo.

Photo showing BPXP drive at top, the existing WDC drive in the middle, and the new SAMSUNG drive at the bottom
Let’s move some things around so they fit

All in.

Photo of main board PCIe slots, showing the cards reinserted
PCIe slots after adding 2nd new NVMe drive

After the install

After the system booted up, I found this:

[slocum dan ~] % egrep -e 'nvd[0,1,2]' /var/run/dmesg.boot
nvd0: <WDC WDS250G2B0C-00PXH0> NVMe namespace
nvd0: 238475MB (488397168 512 byte sectors)
nvd1: <Samsung SSD 980 PRO with Heatsink 1TB> NVMe namespace
nvd1: 953869MB (1953525168 512 byte sectors)
nvd2: <Samsung SSD 980 PRO with Heatsink 1TB> NVMe namespace
nvd2: 953869MB (1953525168 512 byte sectors)
[slocum dan ~] % gpart show nvd1
gpart: No such geom: nvd1.
[slocum dan ~] % gpart show nvd2
=>        40  1953525088  nvd2  GPT  (932G)
          40  1953484800     1  freebsd-zfs  (931G)
  1953484840       40288        - free -  (20M)

[slocum dan ~] % 
[slocum dan ~] % zpool status nvd
  pool: nvd
 state: ONLINE
  scan: resilvered 154G in 00:03:36 with 0 errors on Tue Oct 18 13:41:10 2022
config:

	NAME        STATE     READ WRITE CKSUM
	nvd         ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nvd0p1  ONLINE       0     0     0
	    nvd2p1  ONLINE       0     0     0

errors: No known data errors
[slocum dan ~] % 

Some notes:

  • Lines 4-5 show the new drive.
  • Lines 8-9 confirm it has no partitions.
  • Line 16 confirms the zpool is still OK and I did not remove the wrong drive.

Prepare the new drive

I do exactly what I did for the other drive.

For the record. It, like the first one, does not appear to be a used drive. The Host Read Commands value is low. So is the Host Write Commands value. I have no doubt someone knows how to zero that.

[slocum dan ~] % sudo smartctl -a /dev/nvme1
smartctl 7.3 2022-02-28 r5338 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 980 PRO with Heatsink 1TB
Serial Number:                      [redacted]
Firmware Version:                   4B2QGXA7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 1,000,204,886,016 [1.00 TB]
Unallocated NVM Capacity:           0
Controller ID:                      6
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,000,204,886,016 [1.00 TB]
Namespace 1 Utilization:            0
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 b221409d56
Local Time is:                      Tue Oct 18 17:35:26 2022 UTC
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x0057):     Comp Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f):         S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         128 Pages
Warning  Comp. Temp. Threshold:     82 Celsius
Critical Comp. Temp. Threshold:     85 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     8.49W       -        -    0  0  0  0        0       0
 1 +     4.48W       -        -    1  1  1  1        0     200
 2 +     3.18W       -        -    2  2  2  2        0    1000
 3 -   0.0400W       -        -    3  3  3  3     2000    1200
 4 -   0.0050W       -        -    4  4  4  4      500    9500

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        40 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    8 [4.09 MB]
Data Units Written:                 0
Host Read Commands:                 5,694
Host Write Commands:                0
Controller Busy Time:               0
Power Cycles:                       2
Power On Hours:                     0
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               40 Celsius
Temperature Sensor 2:               44 Celsius

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

[slocum dan ~] % 

Configuring the drive:

[slocum dan ~] % sudo gpart create -s gpt nvd1
nvd1 created
[slocum dan ~] % sudo gpart add -t freebsd-zfs -a 4K -s 953850M -l Drive_Serial_Number nvd1
nvd1p1 added
[slocum dan ~] % gpart show nvd1
=>        40  1953525088  nvd1  GPT  (932G)
          40  1953484800     1  freebsd-zfs  (931G)
  1953484840       40288        - free -  (20M)

[slocum dan ~] % gpart show -l nvd1
=>        40  1953525088  nvd1  GPT  (932G)
          40  1953484800     1  Drive_Serial_Number  (931G)
  1953484840       40288        - free -  (20M)

[slocum dan ~] % 

Next, replace the existing vdev with the new vdev:

[slocum dan ~] % sudo zpool replace nvd nvd0p1 nvd1p1                                                                                                                    17:41:37
[slocum dan ~] % zpool status nvd                                                                                                                                        17:42:43
  pool: nvd
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Oct 18 17:42:38 2022
	18.6G scanned at 1.55G/s, 6.88M issued at 587K/s, 153G total
	8.22M resilvered, 0.00% done, no estimated completion time
config:

	NAME             STATE     READ WRITE CKSUM
	nvd              ONLINE       0     0     0
	  mirror-0       ONLINE       0     0     0
	    replacing-0  ONLINE       0     0     0
	      nvd0p1     ONLINE       0     0     0
	      nvd1p1     ONLINE       0     0     0  (resilvering)
	    nvd2p1       ONLINE       0     0     0

errors: No known data errors

Not long after, we’re done:

[slocum dan ~] % zpool status nvd
  pool: nvd
 state: ONLINE
  scan: resilvered 154G in 00:02:25 with 0 errors on Tue Oct 18 17:45:03 2022
config:

	NAME        STATE     READ WRITE CKSUM
	nvd         ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nvd1p1  ONLINE       0     0     0
	    nvd2p1  ONLINE       0     0     0

errors: No known data errors
[slocum dan ~] % 

But where’s my space?

[slocum dan ~] % zpool list nvd
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
nvd    222G   153G  69.5G        -        8G    45%    68%  1.00x    ONLINE  -

I recalled there was an auto feature, which expanded the zpool size once all the drives had beens swapped. What was it called?

[slocum dan ~] % zpool get all nvd | grep auto
nvd   autoreplace                    off                            default
nvd   autoexpand                     off                            default
nvd   autotrim                       off                            default

Ahh, autoreplace.

How do I fix this now? I found this reference which showed me the way.

[slocum dan ~] % zpool status
  pool: nvd
 state: ONLINE
  scan: resilvered 154G in 00:02:25 with 0 errors on Tue Oct 18 17:45:03 2022
config:

	NAME        STATE     READ WRITE CKSUM
	nvd         ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nvd1p1  ONLINE       0     0     0
	    nvd2p1  ONLINE       0     0     0

[slocum dan ~] % sudo zpool set autoexpand=on nvd

[slocum dan ~] % sudo zpool online -e nvd nvd1p1

[slocum dan ~] % sudo zpool online -e nvd nvd2p1

[slocum dan ~] % zpool list nvd
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
nvd    930G   153G   777G        -         -    10%    16%  1.00x    ONLINE  -
[slocum dan ~] % 

BOOM, the drives are taken online (not that they were offline), and the -e option says to “Expand the device to use all available space”. Done.

Here is the full list you never wanted to see:

[slocum dan ~] % zfs list -r nvd                                                                                                                                         17:59:33
NAME                                                               USED  AVAIL     REFER  MOUNTPOINT
nvd                                                                153G   748G       23K  none
nvd/freshports                                                     152G   748G       23K  none
nvd/freshports/dan                                                38.9M   748G     38.9M  none
nvd/freshports/dev                                                5.10G   748G       23K  /jails/dev-ingress01/var/db/freshports
nvd/freshports/dev-nginx01                                        98.8M   748G       24K  none
nvd/freshports/dev-nginx01/var                                    98.7M   748G       24K  none
nvd/freshports/dev-nginx01/var/db                                 98.7M   748G       24K  none
nvd/freshports/dev-nginx01/var/db/freshports                      98.7M   748G       24K  none
nvd/freshports/dev-nginx01/var/db/freshports/cache                98.7M   748G       24K  /var/db/freshports/cache
nvd/freshports/dev-nginx01/var/db/freshports/cache/categories       56K   748G       41K  /var/db/freshports/cache/categories
nvd/freshports/dev-nginx01/var/db/freshports/cache/commits        88.4M   748G     88.4M  /var/db/freshports/cache/commits
nvd/freshports/dev-nginx01/var/db/freshports/cache/daily           234K   748G      219K  /var/db/freshports/cache/daily
nvd/freshports/dev-nginx01/var/db/freshports/cache/general        9.23M   748G     9.22M  /var/db/freshports/cache/general
nvd/freshports/dev-nginx01/var/db/freshports/cache/news            299K   748G      284K  /var/db/freshports/cache/news
nvd/freshports/dev-nginx01/var/db/freshports/cache/packages         38K   748G       24K  /var/db/freshports/cache/packages
nvd/freshports/dev-nginx01/var/db/freshports/cache/pages            38K   748G       24K  /var/db/freshports/cache/pages
nvd/freshports/dev-nginx01/var/db/freshports/cache/ports           249K   748G      234K  /var/db/freshports/cache/ports
nvd/freshports/dev-nginx01/var/db/freshports/cache/spooling         63K   748G       37K  /var/db/freshports/cache/spooling
nvd/freshports/dev/cache                                           110K   748G     35.5K  /jails/dev-ingress01/var/db/freshports/cache
nvd/freshports/dev/cache/html                                       46K   748G       46K  /jails/dev-ingress01/var/db/freshports/cache/html
nvd/freshports/dev/cache/spooling                                   28K   748G       28K  /jails/dev-ingress01/var/db/freshports/cache/spooling
nvd/freshports/dev/ingress                                        5.08G   748G       23K  /jails/dev-ingress01/var/db/ingress
nvd/freshports/dev/ingress/message-queues                         1.73M   748G     1.33M  /jails/dev-ingress01/var/db/ingress/message-queues
nvd/freshports/dev/ingress/repos                                  5.08G   748G     2.28G  /jails/dev-ingress01/var/db/ingress/repos
nvd/freshports/dev/ingress/repos/src                              2.80G   748G     2.80G  /jails/dev-ingress01/var/db/ingress/repos/src
nvd/freshports/dev/message-queues                                 16.8M   748G     16.0M  /jails/dev-ingress01/var/db/freshports/message-queues
nvd/freshports/jailed                                              134G   748G       24K  none
nvd/freshports/jailed/dev-ingress01                               44.6G   748G       24K  none
nvd/freshports/jailed/dev-ingress01/jails                         44.3G   748G       24K  /jails
nvd/freshports/jailed/dev-ingress01/jails/freshports              44.3G   748G     44.3G  /jails/freshports
nvd/freshports/jailed/dev-ingress01/mkjail                         340M   748G      340M  /var/db/mkjail
nvd/freshports/jailed/stage-ingress01                             44.6G   748G       24K  none
nvd/freshports/jailed/stage-ingress01/jails                       44.3G   748G       24K  /jails
nvd/freshports/jailed/stage-ingress01/jails/freshports            44.3G   748G     44.3G  /jails/freshports
nvd/freshports/jailed/stage-ingress01/mkjail                       340M   748G      340M  /var/db/mkjail
nvd/freshports/jailed/test-ingress01                              44.6G   748G       24K  none
nvd/freshports/jailed/test-ingress01/jails                        44.3G   748G       24K  /jails
nvd/freshports/jailed/test-ingress01/jails/freshports             44.3G   748G     44.3G  /jails/freshports
nvd/freshports/jailed/test-ingress01/mkjail                        340M   748G      340M  /var/db/mkjail
nvd/freshports/mobile-nginx01                                      462K   748G       24K  none
nvd/freshports/mobile-nginx01/var                                  438K   748G       24K  none
nvd/freshports/mobile-nginx01/var/db                               414K   748G       24K  none
nvd/freshports/mobile-nginx01/var/db/freshports                    390K   748G       24K  none
nvd/freshports/mobile-nginx01/var/db/freshports/cache              366K   748G       24K  /var/db/freshports/cache
nvd/freshports/mobile-nginx01/var/db/freshports/cache/categories    38K   748G       24K  /var/db/freshports/cache/categories
nvd/freshports/mobile-nginx01/var/db/freshports/cache/commits       38K   748G       24K  /var/db/freshports/cache/commits
nvd/freshports/mobile-nginx01/var/db/freshports/cache/daily         38K   748G       24K  /var/db/freshports/cache/daily
nvd/freshports/mobile-nginx01/var/db/freshports/cache/general       38K   748G       24K  /var/db/freshports/cache/general
nvd/freshports/mobile-nginx01/var/db/freshports/cache/news          38K   748G       24K  /var/db/freshports/cache/news
nvd/freshports/mobile-nginx01/var/db/freshports/cache/packages      38K   748G       24K  /var/db/freshports/cache/packages
nvd/freshports/mobile-nginx01/var/db/freshports/cache/pages         38K   748G       24K  /var/db/freshports/cache/pages
nvd/freshports/mobile-nginx01/var/db/freshports/cache/ports         38K   748G       24K  /var/db/freshports/cache/ports
nvd/freshports/mobile-nginx01/var/db/freshports/cache/spooling      38K   748G       24K  /var/db/freshports/cache/spooling
nvd/freshports/stage-ingress01                                    5.14G   748G       24K  none
nvd/freshports/stage-ingress01/var                                5.14G   748G       24K  none
nvd/freshports/stage-ingress01/var/db                             5.14G   748G       24K  none
nvd/freshports/stage-ingress01/var/db/freshports                  14.8M   748G       24K  /jails/stage-ingress01/var/db/freshports
nvd/freshports/stage-ingress01/var/db/freshports/cache             144K   748G       24K  /jails/stage-ingress01/var/db/freshports/cache
nvd/freshports/stage-ingress01/var/db/freshports/cache/html         58K   748G       43K  /jails/stage-ingress01/var/db/freshports/cache/html
nvd/freshports/stage-ingress01/var/db/freshports/cache/spooling   61.5K   748G     61.5K  /jails/stage-ingress01/var/db/freshports/cache/spooling
nvd/freshports/stage-ingress01/var/db/freshports/message-queues   14.6M   748G     13.8M  /jails/stage-ingress01/var/db/freshports/message-queues
nvd/freshports/stage-ingress01/var/db/ingress                     5.13G   748G       24K  /jails/stage-ingress01/var/db/ingress
nvd/freshports/stage-ingress01/var/db/ingress/message-queues      1.32M   748G      961K  /jails/stage-ingress01/var/db/ingress/message-queues
nvd/freshports/stage-ingress01/var/db/ingress/repos               5.13G   748G     5.12G  /jails/stage-ingress01/var/db/ingress/repos
nvd/freshports/stage-nginx01                                      96.9M   748G       24K  none
nvd/freshports/stage-nginx01/var                                  96.9M   748G       24K  none
nvd/freshports/stage-nginx01/var/db                               96.9M   748G       24K  none
nvd/freshports/stage-nginx01/var/db/freshports                    96.8M   748G       24K  none
nvd/freshports/stage-nginx01/var/db/freshports/cache              96.8M   748G       24K  /var/db/freshports/cache
nvd/freshports/stage-nginx01/var/db/freshports/cache/categories   6.40M   748G     6.39M  /var/db/freshports/cache/categories
nvd/freshports/stage-nginx01/var/db/freshports/cache/commits      38.4M   748G     38.4M  /var/db/freshports/cache/commits
nvd/freshports/stage-nginx01/var/db/freshports/cache/daily        1.82M   748G     1.80M  /var/db/freshports/cache/daily
nvd/freshports/stage-nginx01/var/db/freshports/cache/general      10.5M   748G     10.4M  /var/db/freshports/cache/general
nvd/freshports/stage-nginx01/var/db/freshports/cache/news          397K   748G      382K  /var/db/freshports/cache/news
nvd/freshports/stage-nginx01/var/db/freshports/cache/packages       38K   748G       24K  /var/db/freshports/cache/packages
nvd/freshports/stage-nginx01/var/db/freshports/cache/pages          38K   748G       24K  /var/db/freshports/cache/pages
nvd/freshports/stage-nginx01/var/db/freshports/cache/ports        39.1M   748G     39.1M  /var/db/freshports/cache/ports
nvd/freshports/stage-nginx01/var/db/freshports/cache/spooling       66K   748G       41K  /var/db/freshports/cache/spooling
nvd/freshports/test-ingress01                                     8.01G   748G       24K  none
nvd/freshports/test-ingress01/var                                 8.01G   748G       24K  none
nvd/freshports/test-ingress01/var/db                              8.01G   748G       24K  none
nvd/freshports/test-ingress01/var/db/freshports                   6.74M   748G       24K  none
nvd/freshports/test-ingress01/var/db/freshports/cache              132K   748G       24K  /jails/test-ingress01/var/db/freshports/cache
nvd/freshports/test-ingress01/var/db/freshports/cache/html          48K   748G       48K  /jails/test-ingress01/var/db/freshports/cache/html
nvd/freshports/test-ingress01/var/db/freshports/cache/spooling    60.5K   748G     60.5K  /jails/test-ingress01/var/db/freshports/cache/spooling
nvd/freshports/test-ingress01/var/db/freshports/message-queues    6.59M   748G     5.80M  /jails/test-ingress01/var/db/freshports/message-queues
nvd/freshports/test-ingress01/var/db/ingress                      8.00G   748G       24K  /jails/test-ingress01/var/db/ingress
nvd/freshports/test-ingress01/var/db/ingress/message-queues       1.34M   748G      962K  /jails/test-ingress01/var/db/ingress/message-queues
nvd/freshports/test-ingress01/var/db/ingress/repos                8.00G   748G     5.10G  /jails/test-ingress01/var/db/ingress/repos
nvd/freshports/test-nginx01                                       33.6M   748G       24K  none
nvd/freshports/test-nginx01/var                                   33.6M   748G       24K  none
nvd/freshports/test-nginx01/var/db                                33.6M   748G       24K  none
nvd/freshports/test-nginx01/var/db/freshports                     33.6M   748G       24K  none
nvd/freshports/test-nginx01/var/db/freshports/cache               33.5M   748G       24K  /var/db/freshports/cache
nvd/freshports/test-nginx01/var/db/freshports/cache/categories     360K   748G      345K  /var/db/freshports/cache/categories
nvd/freshports/test-nginx01/var/db/freshports/cache/commits       21.2M   748G     21.2M  /var/db/freshports/cache/commits
nvd/freshports/test-nginx01/var/db/freshports/cache/daily         5.01M   748G     4.99M  /var/db/freshports/cache/daily
nvd/freshports/test-nginx01/var/db/freshports/cache/general       3.82M   748G     3.79M  /var/db/freshports/cache/general
nvd/freshports/test-nginx01/var/db/freshports/cache/news           300K   748G      285K  /var/db/freshports/cache/news
nvd/freshports/test-nginx01/var/db/freshports/cache/packages        38K   748G       24K  /var/db/freshports/cache/packages
nvd/freshports/test-nginx01/var/db/freshports/cache/pages           38K   748G       24K  /var/db/freshports/cache/pages
nvd/freshports/test-nginx01/var/db/freshports/cache/ports         2.72M   748G     2.71M  /var/db/freshports/cache/ports
nvd/freshports/test-nginx01/var/db/freshports/cache/spooling        64K   748G       39K  /var/db/freshports/cache/spooling

Thank you. I hope this helps.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Leave a Comment