Aug 162015
 

I’m using FreeBSD 10.2-RC2 here, with a bunch of HDD. Here are the current pools:

$ sudo zpool list
NAME            SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
benchmarking   21.8T  1.71T  20.0T         -     3%     7%  1.00x  DEGRADED  -
music            30T  1.86T  28.1T         -     3%     6%  1.00x  ONLINE  -
random_mirror  2.72T   308K  2.72T         -     0%     0%  1.00x  ONLINE  -
zroot           220G  1.29G   219G         -     0%     0%  1.00x  ONLINE  -

The degraded music pool contains a drive I will be replacing later in this post. But first, I want to replace a mis-matched drive in the benchmarking pool. I noticed that when I saw this output:

  pool: music
 state: ONLINE
  scan: none requested
config:

	NAME                       STATE     READ WRITE CKSUM
	music                      ONLINE       0     0     0
	  raidz3-0                 ONLINE       0     0     0
	    gpt/disk_653AK2MXFS9A  ONLINE       0     0     0
	    gpt/disk_653EK93XFS9A  ONLINE       0     0     0
	    gpt/disk_653DK7WCFS9A  ONLINE       0     0     0
	    gpt/disk_6525K2DGFS9A  ONLINE       0     0     0
	    gpt/disk_652FK58FFS9A  ONLINE       0     0     0
	    gpt/disk_653BK12IFS9A  ONLINE       0     0     0
	    gpt/disk_653EK93QFS9A  ONLINE       0     0     0
	    gpt/disk_653IK1IBFS9A  ONLINE       0     0     0
	    gpt/disk_6539K3OJFS9A  ONLINE       0     0     0
	    gpt/disk_653BK12FFS9A  ONLINE       0     0     0
	    gpt/disk_256BYDPGS     ONLINE       0     0     0

The serial number on line 19 is of a different format. I suspected I was using different size drives because I knew the box had both 3 and 5 TB drives. Let’s compare the device on line 18 with the one on line 19. I looked through the output of gpart list and found the two entries:

Geom name: da15
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 9767541134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da15p1
   Mediasize: 3000592941056 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   rawuuid: e27010ea-3a34-11e5-952f-0cc47a4cb140
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: disk_653BK12FFS9A
   length: 3000592941056
   offset: 20480
   type: freebsd-zfs
   index: 1
   end: 5860533127
   start: 40
Consumers:
1. Name: da15
   Mediasize: 5000981078016 (4.5T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3

Geom name: da19
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da19p1
   Mediasize: 3000592941056 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: e3ae48ab-3a34-11e5-952f-0cc47a4cb140
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: disk_256BYDPGS
   length: 3000592941056
   offset: 20480
   type: freebsd-zfs
   index: 1
   end: 5860533127
   start: 40
Consumers:
1. Name: da19
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

Lines 28 shows that the first device (da15) has 4.5T (which is what a 5TB drive gives you) and line 62 (da19) shows a 2.7TB drive (yes, that’s a 3TB drive).

Looking in /var/run/dmesg.boot, the above can be confirmed:

$ grep da15 /var/run/dmesg.boot 
da15 at mps2 bus 0 scbus12 target 15 lun 0
da15: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SPC-4 SCSI device
da15: Serial Number         653BK12FFS9A
da15: 600.000MB/s transfers
da15: Command Queueing enabled
da15: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)

$ grep da19 /var/run/dmesg.boot 
da19 at mps3 bus 0 scbus13 target 8 lun 0
da19: <ATA TOSHIBA DT01ACA3 ABB0> Fixed Direct Access SPC-4 SCSI device
da19: Serial Number            256BYDPGS
da19: 600.000MB/s transfers
da19: Command Queueing enabled
da19: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)

The goal

My goal: remove da19 (disk_256BYDPGS) and replace it with another drive.

The new drive

I have a new drive, da23, and here is how I prepared it for the pool.

$ sudo gpart create -s gpt da23
da23 created

$ sudo gpart add  -a 4k -s 4600GB -t freebsd-zfs  -l disk_653DK7WBFS9A da23
da23p1 added

$ gpart show da23
=>        34  9767541101  da23  GPT  (4.5T)
          34           6        - free -  (3.0K)
          40  9646899200     1  freebsd-zfs  (4.5T)
  9646899240   120641895        - free -  (58G)

I tried to specify the size as 4.5T but failed: gpart: Invalid size param: Invalid argument

Removing the existing device

$ sudo zpool offline music gpt/disk_256BYDPGS
$ zpool status music
  pool: music
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using 'zpool online' or replace the device with
	'zpool replace'.
  scan: none requested
config:

	NAME                       STATE     READ WRITE CKSUM
	music                      DEGRADED     0     0     0
	  raidz3-0                 DEGRADED     0     0     0
	    gpt/disk_653AK2MXFS9A  ONLINE       0     0     0
	    gpt/disk_653EK93XFS9A  ONLINE       0     0     0
	    gpt/disk_653DK7WCFS9A  ONLINE       0     0     0
	    gpt/disk_6525K2DGFS9A  ONLINE       0     0     0
	    gpt/disk_652FK58FFS9A  ONLINE       0     0     0
	    gpt/disk_653BK12IFS9A  ONLINE       0     0     0
	    gpt/disk_653EK93QFS9A  ONLINE       0     0     0
	    gpt/disk_653IK1IBFS9A  ONLINE       0     0     0
	    gpt/disk_6539K3OJFS9A  ONLINE       0     0     0
	    gpt/disk_653BK12FFS9A  ONLINE       0     0     0
	    3040563092296029026    OFFLINE      0     0     0  was /dev/gpt/disk_256BYDPGS

errors: No known data errors

Adding in the new drive

This is the command I used to replace the drive:

$ sudo zpool replace music 3040563092296029026 gpt/disk_653DK7WBFS9A
$ zpool status music
  pool: music
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Aug 15 17:50:55 2015
        3.45G scanned out of 1.86T at 208M/s, 2h36m to go
        320M resilvered, 0.18% done
config:

	NAME                         STATE     READ WRITE CKSUM
	music                        DEGRADED     0     0     0
	  raidz3-0                   DEGRADED     0     0     0
	    gpt/disk_653AK2MXFS9A    ONLINE       0     0     0
	    gpt/disk_653EK93XFS9A    ONLINE       0     0     0
	    gpt/disk_653DK7WCFS9A    ONLINE       0     0     0
	    gpt/disk_6525K2DGFS9A    ONLINE       0     0     0
	    gpt/disk_652FK58FFS9A    ONLINE       0     0     0
	    gpt/disk_653BK12IFS9A    ONLINE       0     0     0
	    gpt/disk_653EK93QFS9A    ONLINE       0     0     0
	    gpt/disk_653IK1IBFS9A    ONLINE       0     0     0
	    gpt/disk_6539K3OJFS9A    ONLINE       0     0     0
	    gpt/disk_653BK12FFS9A    ONLINE       0     0     0
	    replacing-10             OFFLINE      0     0     0
	      3040563092296029026    OFFLINE      0     0     0  was /dev/gpt/disk_256BYDPGS
	      gpt/disk_653DK7WBFS9A  ONLINE       0     0     0  (resilvering)

errors: No known data errors

I’ll just wait…

For extra points

I returned one of the Toshiba 3TB HDD which had smartctl errors shortly after starting up. The replacement arrived last week from Amazon. I put the drive into the system tonight and prepared it. Here’s what I did:

$ sudo gpart create -s gpt da7
da7 created

$ sudo gpart add  -a 4k    -s 5860533095 -t freebsd-zfs  -l disk_35AL161GS da7
da7p1 added

Here is the pool before I replace the missing device:

$ zpool status benchmarking
  pool: benchmarking
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 202G in 1h28m with 0 errors on Mon Aug  3 22:29:55 2015
config:

	NAME                     STATE     READ WRITE CKSUM
	benchmarking             DEGRADED     0     0     0
	  raidz3-0               DEGRADED     0     0     0
	    gpt/disk_Z2T2UJJAS   ONLINE       0     0     0
	    gpt/disk_13Q8U6GYS   ONLINE       0     0     0
	    gpt/disk_256BWVLGS   ONLINE       0     0     0
	    gpt/disk_256BY66GS   ONLINE       0     0     0
	    gpt/disk_255BV69GS   ONLINE       0     0     0
	    gpt/disk_255BS4NGS   ONLINE       0     0     0
	    gpt/disk_255BUT1GS   ONLINE       0     0     0
	    4796034587839379020  UNAVAIL      0     0     0  was /dev/gpt/disk_653EK93PFS9A

errors: No known data errors

Here is the command I used to replace the missing device:

$ sudo zpool replace benchmarking 4796034587839379020 gpt/disk_35AL161GS

Just like that, it was busy resilvering:

$ zpool status benchmarking
  pool: benchmarking
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Aug 15 18:21:01 2015
        263G scanned out of 1.71T at 763M/s, 0h33m to go
        30.6G resilvered, 15.00% done
config:

	NAME                       STATE     READ WRITE CKSUM
	benchmarking               DEGRADED     0     0     0
	  raidz3-0                 DEGRADED     0     0     0
	    gpt/disk_Z2T2UJJAS     ONLINE       0     0     0
	    gpt/disk_13Q8U6GYS     ONLINE       0     0     0
	    gpt/disk_256BWVLGS     ONLINE       0     0     0
	    gpt/disk_256BY66GS     ONLINE       0     0     0
	    gpt/disk_255BV69GS     ONLINE       0     0     0
	    gpt/disk_255BS4NGS     ONLINE       0     0     0
	    gpt/disk_255BUT1GS     ONLINE       0     0     0
	    replacing-7            UNAVAIL      0     0     0
	      4796034587839379020  UNAVAIL      0     0     0  was /dev/gpt/disk_653EK93PFS9A
	      gpt/disk_35AL161GS   ONLINE       0     0     0  (resilvering)

errors: No known data errors

Waiting waiting waiting

After 90 minutes, both resilverings were complete:

$ zpool status benchmarking music
  pool: benchmarking
 state: ONLINE
  scan: resilvered 202G in 0h39m with 0 errors on Sat Aug 15 19:00:07 2015
config:

	NAME                    STATE     READ WRITE CKSUM
	benchmarking            ONLINE       0     0     0
	  raidz3-0              ONLINE       0     0     0
	    gpt/disk_Z2T2UJJAS  ONLINE       0     0     0
	    gpt/disk_13Q8U6GYS  ONLINE       0     0     0
	    gpt/disk_256BWVLGS  ONLINE       0     0     0
	    gpt/disk_256BY66GS  ONLINE       0     0     0
	    gpt/disk_255BV69GS  ONLINE       0     0     0
	    gpt/disk_255BS4NGS  ONLINE       0     0     0
	    gpt/disk_255BUT1GS  ONLINE       0     0     0
	    gpt/disk_35AL161GS  ONLINE       0     0     0

errors: No known data errors

  pool: music
 state: ONLINE
  scan: resilvered 173G in 1h25m with 0 errors on Sat Aug 15 19:15:59 2015
config:

	NAME                       STATE     READ WRITE CKSUM
	music                      ONLINE       0     0     0
	  raidz3-0                 ONLINE       0     0     0
	    gpt/disk_653AK2MXFS9A  ONLINE       0     0     0
	    gpt/disk_653EK93XFS9A  ONLINE       0     0     0
	    gpt/disk_653DK7WCFS9A  ONLINE       0     0     0
	    gpt/disk_6525K2DGFS9A  ONLINE       0     0     0
	    gpt/disk_652FK58FFS9A  ONLINE       0     0     0
	    gpt/disk_653BK12IFS9A  ONLINE       0     0     0
	    gpt/disk_653EK93QFS9A  ONLINE       0     0     0
	    gpt/disk_653IK1IBFS9A  ONLINE       0     0     0
	    gpt/disk_6539K3OJFS9A  ONLINE       0     0     0
	    gpt/disk_653BK12FFS9A  ONLINE       0     0     0
	    gpt/disk_653DK7WBFS9A  ONLINE       0     0     0

errors: No known data errors

Interesting messages

I noticed these messages in /var/log/messages:

Aug 15 18:21:01 varm ZFS: vdev state changed, pool_guid=17468409599595688358 vdev_guid=17185678106577923296
Aug 15 18:36:18 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=13168585638132050393 vdev_guid=8485381449328651362''
Aug 15 18:36:18 varm ZFS: vdev state changed, pool_guid=13168585638132050393 vdev_guid=8485381449328651362
Aug 15 18:36:18 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=13168585638132050393 vdev_guid=8421655518573488526''
Aug 15 18:36:18 varm ZFS: vdev state changed, pool_guid=13168585638132050393 vdev_guid=8421655518573488526
Aug 15 18:36:18 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=13168585638132050393 vdev_guid=6595461994760359315''
Aug 15 18:36:18 varm ZFS: vdev state changed, pool_guid=13168585638132050393 vdev_guid=6595461994760359315
Aug 15 18:37:14 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=15362233628480096924 vdev_guid=6417995594589973460''
Aug 15 18:37:14 varm ZFS: vdev state changed, pool_guid=15362233628480096924 vdev_guid=6417995594589973460
Aug 15 18:37:14 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=15362233628480096924 vdev_guid=12132241653334970585''
Aug 15 18:37:14 varm ZFS: vdev state changed, pool_guid=15362233628480096924 vdev_guid=12132241653334970585
Aug 15 18:37:15 varm devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=15362233628480096924 vdev_guid=12827680528713607564''
Aug 15 18:37:15 varm ZFS: vdev state changed, pool_guid=15362233628480096924 vdev_guid=12827680528713607564

Anyone care to tell me what they refer to?

Trolling

This completes our device replacements for today. Please tip your servers, mention @mwlauthor on Twitter (without referencing this post), and have a nice day. Seriously, please just mention him. :)

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive