data04: DEGRADED – oh and have I mentioned WatchGoose?

Leave a Comment / nvme, ZFS / By /

I saw this today in my logs on r730-01. It looks similar to a previous DEGRADED even back on July 2. In this case, the device is S7KGNU0Y915666E, not S7KGNU0Y912955D as it was back then.

Sep 12 03:46:54 r730-01 kernel: nvme1: Resetting controller due to a timeout.
Sep 12 03:46:54 r730-01 kernel: nvme1: event="start"
Sep 12 03:46:54 r730-01 kernel: nvme1: Waiting for reset to complete
Sep 12 03:47:14 r730-01 syslogd: last message repeated 197 times
Sep 12 03:47:14 r730-01 ZFS[96518]: vdev I/O failure, zpool=data04 path=/dev/gpt/S7KGNU0Y915666E offset=270336 size=8192 error=6
Sep 12 03:47:14 r730-01 kernel: nvme1: Waiting for reset to complete
Sep 12 03:47:14 r730-01 kernel: nvme1: controller ready did not become 0 within 20500 ms
Sep 12 03:47:14 r730-01 kernel: nvme1: event="timed_out"
Sep 12 03:47:14 r730-01 kernel: nvme1: failing outstanding i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: FLUSH (00) sqid:1 cid:112 nsid:1
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:1 cid:112 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: failing outstanding i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:120 nsid:1 lba:2320185448 len:2048
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): FLUSH (00). NCB: opc=0 fuse=0 nsid=1 prp1=0 prp2=0 cdw=0 0 0 0 0 0
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:120 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:121 nsid:1 lba:2320187496 len:2048
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:121 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:123 nsid:1 lba:2320189544 len:1368
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:123 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b3868 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:118 nsid:1 lba:2320190912 len:2048
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b4068 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b4868 0 557 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:118 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:125 nsid:1 lba:2320192960 len:2048
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b4dc0 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:125 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:113 nsid:1 lba:2320195008 len:1368
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b55c0 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:113 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 5, Retries exhausted
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:115 nsid:1 lba:2320196376 len:2048
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:115 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:114 nsid:1 lba:2320198424 len:2048
Sep 12 03:47:14 r730-01 kernel: nda1 at nvme1 bus 0 scbus17 target 0 lun 1
Sep 12 03:47:14 r730-01 kernel: nda1: <Samsung SSD 990 PRO 4TB 4B2QJXD7 S7KGNU0Y915666E> s/n S7KGNU0Y915666E detached
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:114 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:5 cid:122 nsid:1 lba:2320200472 len:1368
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:122 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: failing queued i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:7 cid:0 nsid:1 lba:4317358168 len:8
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b5dc0 0 557 0 0 0
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:7 cid:0 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:7 cid:0 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: nvme1: failing outstanding i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: WRITE (01) sqid:11 cid:125 nsid:1 lba:4127749432 len:8
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:11 cid:125 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: WRITE (01) sqid:11 cid:119 nsid:1 lba:5324663584 len:8
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:11 cid:119 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: failing queued i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:13 cid:0 nsid:1 lba:5554129904 len:8
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:13 cid:0 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: failing outstanding i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:14 cid:115 nsid:1 lba:4123885016 len:32
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:14 cid:115 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: READ (02) sqid:14 cid:125 nsid:1 lba:4123885456 len:8
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:14 cid:125 cdw0:0
Sep 12 03:47:14 r730-01 kernel: nvme1: failing outstanding i/o
Sep 12 03:47:14 r730-01 kernel: nvme1: WRITE (01) sqid:15 cid:104 nsid:1 lba:4127749424 len:8
Sep 12 03:47:14 r730-01 kernel: nvme1: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:15 cid:104 cdw0:0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b6318 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b6b18 0 7ff 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8a4b7318 0 557 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=155a858 1 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=f6087538 0 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=3d5feb20 1 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=4b0d4bf0 1 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=f5cd7dd8 0 1f 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=f5cd7f90 0 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=f6087530 0 7 0 0 0
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): CAM status: NVME Status Error
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
Sep 12 03:47:14 r730-01 kernel: (nda1:nvme1:0:0:1): Error 6, Periph was invalidated
Sep 12 03:47:14 r730-01 kernel: nvme1: Failed controller, stopping watchdog timeout.
Sep 12 03:47:15 r730-01 ZFS[96546]: vdev state changed, pool_guid=1760090623298529734 vdev_guid=10692465753147695115
Sep 12 03:47:15 r730-01 ZFS[96550]: vdev is removed, pool_guid=1760090623298529734 vdev_guid=10692465753147695115
Sep 12 03:47:15 r730-01 kernel: nvme1: Failed controller, stopping watchdog timeout.
Sep 12 03:47:15 r730-01 syslogd: last message repeated 1 times
Sep 12 03:47:15 r730-01 syslogd: last message repeated 1 times
Sep 12 03:47:15 r730-01 kernel: (nda1:nvme1:0:0:1): Periph destroyed
Sep 12 03:47:15 r730-01 kernel: nvme1: Failed controller, stopping watchdog timeout.
Sep 12 03:47:15 r730-01 syslogd: last message repeated 1 times

The zpool status is:

[12:16 r730-01 dvl ~] % zpool status data04
  pool: data04
 state: DEGRADED
status: One or more devices have been removed.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using zpool online' or replace the device with
	'zpool replace'.
  scan: scrub repaired 0B in 01:20:36 with 0 errors on Sat Sep 12 04:29:12 2026
config:

	NAME                     STATE     READ WRITE CKSUM
	data04                   DEGRADED     0     0     0
	  raidz2-0               DEGRADED     0     0     0
	    gpt/S7KGNU0Y722875X  ONLINE       0     0     0
	    gpt/S7KGNU0Y915666E  REMOVED      0     0     0
	    gpt/S7KGNU0Y912937J  ONLINE       0     0     0
	    gpt/S7KGNU0Y912955D  ONLINE       0     0     0
	    gpt/S7U8NJ0Y716854P  ONLINE       0     0     0
	    gpt/S7U8NJ0Y716801F  ONLINE       0     0     0
	    gpt/S757NS0Y700758M  ONLINE       0     0     0
	    gpt/S757NS0Y700760R  ONLINE       0     0     0

errors: No known data errors

What’s next?

I will reboot this host after dbclone finishes restoring database backups

.

I’ll take this opportunity to mention WatchGoose – a third party website for monitoring tasks, such as cronjobs. If I’d checked here first, I’d have seen dbclone had not finished yet today. There is a free tier for this service.

WatchGoose
WatchGoose

Full disclosure: They offered free accounts to FreeBSD Developers. I got one. And I plan to add more monitoring such as:

  • has the Bacula database backup completed? (one host)
  • has the Bacula database backup been rsynced? (three hosts)
  • has the Bacula configuration been rsynced? (three hosts)
  • have the zfs snapshots been created? (several hosts)
  • have the data backup rsyncs run? (several hosts)
  • have the FreshPorts package checks been pulled? (several hosts)
  • have the database backups been rsynced to dbclone? (several hosts)

It’s mainly a matter of time for implementing that.

I plan to post this, then create another post as I resolve the issue.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Related Posts

Leave a Comment

Scroll to Top