Aug 292022
 

This post is based on a tweet and was created after a followup incident occurred today. The post was created to consolidate the information into a blog post so I could easily find it later.

Details about this host (disks, zpool, gpart, etc) are in this post.

On March 15 2022, I noticed these messages in /var/log/messages:

Mar 15 13:36:02 r720-01 kernel: mps1: mpssas_prepare_remove: Sending reset for target ID 22
Mar 15 13:36:02 r720-01 kernel: da13 at mps1 bus 0 scbus8 target 22 lun 0
Mar 15 13:36:02 r720-01 kernel: da13: <ATA Samsung SSD 850 3B6Q>  s/n S3PTNF0JA11513Y      detached
Mar 15 13:36:02 r720-01 kernel: mps1: No pending commands: starting remove_device
Mar 15 13:36:02 r720-01 kernel: mps1: Unfreezing devq for target ID 22
Mar 15 13:36:03 r720-01 kernel: (da13:mps1:0:22:0): Periph destroyed
Mar 15 13:36:03 r720-01 ZFS[76470]: vdev state changed, pool_guid=1975810868733347630 vdev_guid=11376585178559251170
Mar 15 13:36:03 r720-01 ZFS[76474]: vdev is removed, pool_guid=1975810868733347630 vdev_guid=11376585178559251170
Mar 15 13:36:04 r720-01 kernel: da13 at mps1 bus 0 scbus8 target 22 lun 0
Mar 15 13:36:04 r720-01 kernel: da13: <ATA Samsung SSD 850 3B6Q> Fixed Direct Access SPC-4 SCSI device
Mar 15 13:36:04 r720-01 kernel: da13: Serial Number S3PTNF0JA11513Y     
Mar 15 13:36:04 r720-01 kernel: da13: 600.000MB/s transfers
Mar 15 13:36:04 r720-01 kernel: da13: Command Queueing enabled
Mar 15 13:36:04 r720-01 kernel: da13: 476940MB (976773168 512 byte sectors)
Mar 15 13:36:04 r720-01 kernel: da13: quirks=0x8<4K>

The zpool array was degraded. I don’t have a record of that message.

The tweet also mentions:

kernel mps1: mpssas_prepare_remove: Sending reset for target ID 22

The tweet mentions (via this gist) “It is lovely how zfs status tells you want commands to run.” – so presumably it told me to do this (as also found in the gist):

[r720-01 dan ~] % sudo zpool online tank_fast da13p1
[r720-01 dan ~] % zpool status tank_fast
  pool: tank_fast
 state: ONLINE
  scan: resilvered 607M in 00:00:05 with 0 errors on Wed Mar 16 17:37:56 2022
config:

	NAME        STATE     READ WRITE CKSUM
	tank_fast   ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    da14p1  ONLINE       0     0     0
	    da13p1  ONLINE       0     0     0
	  mirror-1  ONLINE       0     0     0
	    da11p1  ONLINE       0     0     0
	    da12p1  ONLINE       0     0     0

errors: No known data errors
[r720-01 dan ~] % 

Right after the above, this showed up in the logs:

Mar 16 17:37:51 r720-01 ZFS[64034]: vdev state changed, pool_guid=1975810868733347630 vdev_guid=11376585178559251170

Today, August 29 2022, I found this in /var/log/messages:

Aug 29 04:15:44 r720-01 kernel: mps1: IOC Fault 0x40007e23, Resetting
Aug 29 04:15:44 r720-01 kernel: mps1: Reinitializing controller
Aug 29 04:15:44 r720-01 kernel: mps1: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
Aug 29 04:15:44 r720-01 kernel: mps1: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>

zpool status

The zpool status is fine:

[r720-01 dan ~] % zpool status                                                                                                                                     12:32:28
  pool: data01
 state: ONLINE
  scan: scrub repaired 0B in 00:27:01 with 0 errors on Mon Aug 29 04:36:50 2022
config:

	NAME           STATE     READ WRITE CKSUM
	data01         ONLINE       0     0     0
	  mirror-0     ONLINE       0     0     0
	    gpt/data1  ONLINE       0     0     0
	    gpt/data2  ONLINE       0     0     0
	  mirror-1     ONLINE       0     0     0
	    gpt/data3  ONLINE       0     0     0
	    gpt/data4  ONLINE       0     0     0
	  mirror-2     ONLINE       0     0     0
	    gpt/data5  ONLINE       0     0     0
	    gpt/data6  ONLINE       0     0     0
	  mirror-3     ONLINE       0     0     0
	    gpt/data7  ONLINE       0     0     0
	    gpt/data8  ONLINE       0     0     0

errors: No known data errors

  pool: tank_fast
 state: ONLINE
  scan: scrub repaired 0B in 00:10:26 with 0 errors on Mon Aug 29 04:20:44 2022
config:

	NAME        STATE     READ WRITE CKSUM
	tank_fast   ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    da14p1  ONLINE       0     0     0
	    da13p1  ONLINE       0     0     0
	  mirror-1  ONLINE       0     0     0
	    da11p1  ONLINE       0     0     0
	    da12p1  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:03:05 with 0 errors on Mon Aug 29 04:13:26 2022
config:

	NAME          STATE     READ WRITE CKSUM
	zroot         ONLINE       0     0     0
	  mirror-0    ONLINE       0     0     0
	    gpt/zfs0  ONLINE       0     0     0
	    gpt/zfs1  ONLINE       0     0     0

errors: No known data errors
[r720-01 dan ~] %                                                                                                                                                  12:33:44
Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive