Dec 122020
 

In this post I will replace a working, but suspect, drive with another drive. No down time. The server is knew.

In this post:

  1. FreeBSD 12.2
  2. ZFS
  3. TOSHIBA MD04ACA500 5TB drive – the suspect drive: da17
  4. TOSHIBA HDWE150 5TB drive – the replacement: da22
  5. None of these drives are under warranty

What drives are in this server?

I have had good luck with Toshiba DT01ACA300 3TB drives (presenting as Hitachi HDS723030BLE640), starting a few years back, and more recently, with Toshiba 5TB drives (Toshiba X300 presenting as TOSHIBA HDWE150 or TOSHIBA MD04ACA5 FP2A). Also great drives for me.

FYI Those 3TB drives presented as Hitachi HDS723030BLE640. They were great for me.

Why am I replacing

Deciding when to replace a drive seems to be an art, not a science. Opinions vary.

I find similar with asking about drive recommendations. Name a brand and someone will report a terrible experience with them.

In my case, I’m seeing these entries in /var/log/messages:

Dec 12 09:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors
Dec 12 09:53:04 knew syslogd: last message repeated 1 times
Dec 12 10:23:03 knew syslogd: last message repeated 1 times
Dec 12 10:53:04 knew syslogd: last message repeated 1 times
Dec 12 11:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors
Dec 12 11:53:05 knew syslogd: last message repeated 1 times
Dec 12 12:23:04 knew syslogd: last message repeated 1 times
Dec 12 12:53:03 knew syslogd: last message repeated 1 times
Dec 12 13:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors
Dec 12 13:53:04 knew syslogd: last message repeated 1 times

The full smartctl output for this drive is in this recent blog post.

Here is a graph of the Reallocated_Event_Count going up since August, where is was zero before that.

LibreNMS graph showing reallocated event count going from 0 to 40 between Aug 2020 and late Oct 2020

LibreNMS graph showing reallocated event count going from 0 to 40 between Aug 2020 and late Oct 2020

This is the only drive in the system with a non-zero value for Current_Pending_Sector.

I think the Current_Pending_Sector and Reallocated_Event_Count refer to the same thing.

Both old and new drives are in the system

It is vital to note that both drives are in the system. The zpool is not degraded.

This is important. The failing drive *COULD* be removed from the system and the replacement drive put in the same drive bay and replaced. This is not optimal. When the drive is removed from the system the zpool will degrade.

Keeping both old and new drives in the system means that the zpool does not degrade during the replacement process.

Maintaining the vdev, and therefore zpool integrity is important.

This zpool consists of two raidz2 vdevs, each containing 10 5TB drives.

[dan@knew:~] $ zpool status system
  pool: system
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: scrub repaired 2.27M in 0 days 16:45:33 with 0 errors on Tue Dec  8 21:40:28 2020
config:

	NAME        STATE     READ WRITE CKSUM
	system      ONLINE       0     0     0
	  raidz2-0  ONLINE       0     0     0
	    da3p3   ONLINE       0     0     0
	    da10p3  ONLINE       0     0     0
	    da9p3   ONLINE       0     0     0
	    da2p3   ONLINE       0     0     0
	    da13p3  ONLINE       0     0     0
	    da15p3  ONLINE       0     0     0
	    da11p3  ONLINE       0     0     0
	    da14p3  ONLINE       0     0     0
	    da8p3   ONLINE       0     0     0
	    da7p3   ONLINE       0     0     0
	  raidz2-1  ONLINE       0     0     0
	    da5p1   ONLINE       0     0     0
	    da6p1   ONLINE       0     0     0
	    da19p1  ONLINE       0     0     0
	    da12p1  ONLINE       0     0     0
	    da4p1   ONLINE       0     0     0
	    da1p1   ONLINE       0     0     0
	    da17p1  ONLINE       0     0     0
	    da16p1  ONLINE       0     0     0
	    da0p1   ONLINE       0     0     0
	    da18p1  ONLINE       0     0     0

errors: No known data errors
[dan@knew:~] $ 

The Some supported features are not enabled on the pool notice is related to a recent upgrade of this host from FreeBSD 12.1 to FreeBSD 12.2. The new ZFS features are not yet enabled.

Preparation of the new drive

I say new drive, but this drive was purchased some time ago and has been sitting in the system and powered on for some time. It has 17742 power on hours, so it’s been sitting there for 2 years, just waiting for this situation.

I’m running FreeBSD and it has long been the best practice to use partitioned drives for ZFS under FreeBSD. You might well ask why.

First step, partition the new drive. gpart can do that for you, using the partition of an existing drive.

[dan@knew:~] $ gpart backup da17 | sudo gpart restore da22
[dan@knew:~] $ gpart show da17 da22
=>        34  9767541101  da17  GPT  (4.5T)
          34           6        - free -  (3.0K)
          40  9766000000     1  freebsd-zfs  (4.5T)
  9766000040     1541095        - free -  (752M)

=>        40  9767541088  da22  GPT  (4.5T)
          40  9766000000     1  freebsd-zfs  (4.5T)
  9766000040     1541088        - free -  (752M)

[dan@knew:~] $ 

One thing I often do is label the drive partition using the serial number of the drive. This can be useful when it comes time to pull a drive from the system and helps to identify that you do have the correct drive.

[dan@knew:~] $ sudo gpart modify -i 1 -l 48JGK3AYF57D da22
da22p1 modified

[dan@knew:~] $ gpart show -l da22
=>        40  9767541088  da22  GPT  (4.5T)
          40  9766000000     1  48JGK3AYF57D  (4.5T)
  9766000040     1541088        - free -  (752M)

[dan@knew:~] $ 

That label also appears in this directory:

[dan@knew:~] $ ls -l /dev/gpt/
total 0
crw-r-----  1 root  operator  0x951 Dec 12 15:27 48JGK3AYF57D
crw-r-----  1 root  operator  0x10a Dec  8 22:21 BTJR5090037J240AGN.boot.internal
crw-r-----  1 root  operator  0x165 Dec  8 22:21 BTJR509003N7240AGN.boot.internal
crw-r-----  1 root  operator  0x17c Dec  8 22:21 S3Z8NB0KB11776R.Slot.11
crw-r-----  1 root  operator  0x17b Dec  8 22:21 S3Z8NB0KB11784L.Slot.05
[dan@knew:~] $ 

You can see some other labels I have created to indicate the boot drives, which are mounted inside the chassis, as opposed to a drive bay. Also shown are drives from slot 11 and 05. Presumably they are still in those locations.

Just in case this drive has previously been used in a zpool, let’s check for any existing labels (note /dev/da22 also works for this command):

[dan@knew:~] $ sudo zdb -l /dev/gpt/48JGK3AYF57D
------------------------------------
LABEL 0
------------------------------------
failed to unpack label 0
------------------------------------
LABEL 1
------------------------------------
failed to unpack label 1
------------------------------------
LABEL 2
------------------------------------
failed to unpack label 2
------------------------------------
LABEL 3
------------------------------------
failed to unpack label 3
[dan@knew:~] $ 

If I found anything other than failed, I would have used zpool labelclear as I did back in Feb 2016.

Starting the replace command

This is the command, with OLD drive first, and new drive last.

[dan@knew:~] $ sudo zpool replace system da17p1 da22p1
[dan@knew:~] $ 

It is normal and expected to see something like this in /var/log/messages after issuing a replace command:

Dec 12 15:35:47 knew ZFS[70753]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$18428760864140250121

Now, we let the replacement drive resilver.

[dan@knew:~] $ zpool status system
  pool: system
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Dec 12 15:35:49 2020
	2.70T scanned at 283M/s, 1.33T issued at 1.23G/s, 58.5T total
	0 resilvered, 2.27% done, 0 days 13:10:05 to go
config:

	NAME             STATE     READ WRITE CKSUM
	system           ONLINE       0     0     0
	  raidz2-0       ONLINE       0     0     0
	    da3p3        ONLINE       0     0     0
	    da10p3       ONLINE       0     0     0
	    da9p3        ONLINE       0     0     0
	    da2p3        ONLINE       0     0     0
	    da13p3       ONLINE       0     0     0
	    da15p3       ONLINE       0     0     0
	    da11p3       ONLINE       0     0     0
	    da14p3       ONLINE       0     0     0
	    da8p3        ONLINE       0     0     0
	    da7p3        ONLINE       0     0     0
	  raidz2-1       ONLINE       0     0     0
	    da5p1        ONLINE       0     0     0
	    da6p1        ONLINE       0     0     0
	    da19p1       ONLINE       0     0     0
	    da12p1       ONLINE       0     0     0
	    da4p1        ONLINE       0     0     0
	    da1p1        ONLINE       0     0     0
	    replacing-6  ONLINE       0     0     0
	      da17p1     ONLINE       0     0     0
	      da22p1     ONLINE       0     0     0
	    da16p1       ONLINE       0     0     0
	    da0p1        ONLINE       0     0     0
	    da18p1       ONLINE       0     0     0

errors: No known data errors
[dan@knew:~] $ 

It claims 15.5 hours. Given it is now Sat Dec 12 15:37:09 UTC 2020, that would be at about 07:20 UTC tomorrow.

I have other things to do. I’ll complete this block post tomorrow.

EDIT: already, at 15:48:23, we have 03:44:42 to go. Oh, and now it’s back up to 04:56:09, oh and now 05:23:39

I think I’ll stop checking do some of my other tasks.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive