Replacing a failed drive in ZFS on FreeBSD

1 Comment / Disks, FreeBSD, Open Source, ZFS / By

I’m using FreeBSD 10.2-RC2. One of my new hard 3TB drives was acting up. I removed it and sent it off to be replaced.

In the meantime, I received some 5TB drives and I added one of them into the existing vdev of the zpool.

First, I partitioned the drive much like the other drives in the system. In my case, this was:

$ sudo gpart create -s gpt da0
da0 created

$ gpart show da0
=>        34  9767541101  da0  GPT  (4.5T)
          34  9767541101       - free -  (4.5T)

$ sudo gpart add -a 4k -s 5860533095 -t freebsd-zfs  -l disk_653EK93PFS9A da0

$ gpart show da0
=>        34  9767541101  da0  GPT  (4.5T)
          34           6       - free -  (3.0K)
          40  5860533088    1  freebsd-zfs  (2.7T)
  5860533128  3907008007       - free -  (1.8T)

Here is the status from earlier today:

$ zpool status
  pool: benchmarking
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using 'zpool online' or replace the device with
	'zpool replace'.
  scan: resilvered 0 in 0h0m with 0 errors on Sun Jul 26 14:44:20 2015
config:

	NAME                      STATE     READ WRITE CKSUM
	benchmarking              DEGRADED     0     0     0
	  raidz3-0                DEGRADED     0     0     0
	    gpt/disk_Z2T2UJJAS    ONLINE       0     0     0
	    gpt/disk_13Q8U6GYS    ONLINE       0     0     0
	    gpt/disk_256BWVLGS    ONLINE       0     0     0
	    gpt/disk_256BY66GS    ONLINE       0     0     0
	    gpt/disk_255BV69GS    ONLINE       0     0     0
	    gpt/disk_255BS4NGS    ONLINE       0     0     0
	    gpt/disk_255BUT1GS    ONLINE       0     0     0
	    17862094558491243428  OFFLINE      0     0     0  was /dev/gpt/disk_255BTBPGS

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Sat Jul 25 14:50:11 2015
config:

	NAME          STATE     READ WRITE CKSUM
	zroot         ONLINE       0     0     0
	  mirror-0    ONLINE       0     0     0
	    gpt/zfs0  ONLINE       0     0     0
	    gpt/zfs1  ONLINE       0     0     0

errors: No known data errors

Reading zpool Administration, specifically the Dealing with Failed Devices section, I decided to issue this command:

$ sudo zpool replace benchmarking 17862094558491243428 gpt/disk_653EK93PFS9
$

The status now shows:

$ zpool status
  pool: benchmarking
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Aug  3 21:01:53 2015
        3.36G scanned out of 1.71T at 287M/s, 1h44m to go
        390M resilvered, 0.19% done
config:

	NAME                         STATE     READ WRITE CKSUM
	benchmarking                 DEGRADED     0     0     0
	  raidz3-0                   DEGRADED     0     0     0
	    gpt/disk_Z2T2UJJAS       ONLINE       0     0     0
	    gpt/disk_13Q8U6GYS       ONLINE       0     0     0
	    gpt/disk_256BWVLGS       ONLINE       0     0     0
	    gpt/disk_256BY66GS       ONLINE       0     0     0
	    gpt/disk_255BV69GS       ONLINE       0     0     0
	    gpt/disk_255BS4NGS       ONLINE       0     0     0
	    gpt/disk_255BUT1GS       ONLINE       0     0     0
	    replacing-7              OFFLINE      0     0     0
	      17862094558491243428   OFFLINE      0     0     0  was /dev/gpt/disk_255BTBPGS
	      gpt/disk_653EK93PFS9A  ONLINE       0     0     0  (resilvering)

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Sat Jul 25 14:50:11 2015
config:

	NAME          STATE     READ WRITE CKSUM
	zroot         ONLINE       0     0     0
	  mirror-0    ONLINE       0     0     0
	    gpt/zfs0  ONLINE       0     0     0
	    gpt/zfs1  ONLINE       0     0     0

errors: No known data errors

These are the messages which showed up in /var/log/messages:

Aug  3 21:01:52 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=17468409599595688358 vdev_guid=4796034587839379020''
Aug  3 21:01:52 varm ZFS: vdev state changed, pool_guid=17468409599595688358 vdev_guid=4796034587839379020

Then I waited for the resilvering to complete. I waited about 90 minutes. Now I see:

$ zpool status
  pool: benchmarking
 state: ONLINE
  scan: resilvered 202G in 1h28m with 0 errors on Mon Aug  3 22:29:55 2015
config:

	NAME                       STATE     READ WRITE CKSUM
	benchmarking               ONLINE       0     0     0
	  raidz3-0                 ONLINE       0     0     0
	    gpt/disk_Z2T2UJJAS     ONLINE       0     0     0
	    gpt/disk_13Q8U6GYS     ONLINE       0     0     0
	    gpt/disk_256BWVLGS     ONLINE       0     0     0
	    gpt/disk_256BY66GS     ONLINE       0     0     0
	    gpt/disk_255BV69GS     ONLINE       0     0     0
	    gpt/disk_255BS4NGS     ONLINE       0     0     0
	    gpt/disk_255BUT1GS     ONLINE       0     0     0
	    gpt/disk_653EK93PFS9A  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Sat Jul 25 14:50:11 2015
config:

	NAME          STATE     READ WRITE CKSUM
	zroot         ONLINE       0     0     0
	  mirror-0    ONLINE       0     0     0
	    gpt/zfs0  ONLINE       0     0     0
	    gpt/zfs1  ONLINE       0     0     0

errors: No known data errors

Done. Thank you for coming. Please tell your friends.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

1 thought on “Replacing a failed drive in ZFS on FreeBSD”

  1. Dan Langille

    Later, I noticed these. I don’t know what they mean.

    Aug  3 23:10:54 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=9211134403469172140''
Aug  3 23:10:54 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=9211134403469172140
Aug  3 23:10:54 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=17267024055838141310''
Aug  3 23:10:54 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=17267024055838141310
Aug  3 23:10:54 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=8256405295929053313''
Aug  3 23:10:54 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=8256405295929053313
Aug  3 23:10:54 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=6598645275852267849''
Aug  3 23:10:55 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=6598645275852267849
Aug  3 23:10:55 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=15350583303802625739''
Aug  3 23:10:55 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=15350583303802625739
Aug  3 23:10:55 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=6765781070189751992''
Aug  3 23:10:55 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=6765781070189751992
Aug  3 23:10:55 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=16723636486208499889''
Aug  3 23:10:55 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=16723636486208499889
Aug  3 23:10:55 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=17294169494618601773''
Aug  3 23:10:55 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=17294169494618601773
Aug  3 23:10:55 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=3450389178295691477''
Aug  3 23:10:55 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=3450389178295691477
Aug  3 23:10:55 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=17564453064704307814''
Aug  3 23:10:55 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=17564453064704307814
Aug  3 23:10:55 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=5764823468107978234 vdev_guid=3040563092296029026''
Aug  3 23:10:55 varm ZFS: vdev state changed, pool_guid=5764823468107978234 vdev_guid=3040563092296029026

Leave a Comment

Scroll to Top