Yesterday, I started a zpool replace. It finished overnight, and dropped the suspect drive out of the vdev.
The resilver finished in the middle of the night:
[dan@knew:~] $ zpool status system pool: system state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: resilvered 2.04T in 0 days 16:15:52 with 0 errors on Sun Dec 13 07:51:41 2020 config: NAME STATE READ WRITE CKSUM system ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da3p3 ONLINE 0 0 0 da10p3 ONLINE 0 0 0 da9p3 ONLINE 0 0 0 da2p3 ONLINE 0 0 0 da13p3 ONLINE 0 0 0 da15p3 ONLINE 0 0 0 da11p3 ONLINE 0 0 0 da14p3 ONLINE 0 0 0 da8p3 ONLINE 0 0 0 da7p3 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 da5p1 ONLINE 0 0 0 da6p1 ONLINE 0 0 0 da19p1 ONLINE 0 0 0 da12p1 ONLINE 0 0 0 da4p1 ONLINE 0 0 0 da1p1 ONLINE 0 0 0 da22p1 ONLINE 0 0 0 da16p1 ONLINE 0 0 0 da0p1 ONLINE 0 0 0 da18p1 ONLINE 0 0 0 errors: No known data errors
As you can see, da22p1 has taken the place of da17p1. The resilver took
As the new drive is being resilvered, I started looking at metrics.
Here is gstat. Look at how da22, the replacement drive is getting all the writes, compared to the others.
dT: 5.001s w: 5.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 27 1 1 0.2 26 156 0.0 0.1| nvd0
0 30 4 9 0.1 25 156 0.0 0.2| nvd1
0 25 0 0 0.0 22 172 0.2 0.5| ada0
0 26 0 0 0.0 22 172 0.3 0.6| ada1
2 459 374 25421 8.4 70 1329 8.1 52.4| da0
1 469 382 25382 8.1 71 1327 8.1 50.0| da1
0 75 0 0 0.0 60 930 6.9 20.9| da2
0 76 0 0 0.0 61 932 8.0 21.8| da3
0 491 407 25419 4.8 69 1333 7.9 40.5| da4
1 495 411 25420 4.3 69 1333 7.8 40.6| da5
1 497 412 25370 4.4 70 1341 8.6 43.4| da6
0 74 0 0 0.0 59 925 5.2 17.2| da7
0 75 0 0 0.0 60 932 5.4 17.9| da8
0 77 0 0 0.0 62 930 10.5 26.6| da9
0 77 0 0 0.0 62 935 10.2 24.1| da10
0 76 0 0 0.0 61 921 8.3 23.1| da11
1 485 400 25386 5.2 69 1323 8.1 45.5| da12
0 76 0 0 0.0 61 928 6.7 17.7| da13
0 76 0 0 0.0 61 931 6.2 21.5| da14
0 76 0 1 35.0 61 924 5.7 17.6| da15
2 495 409 25365 4.9 71 1335 8.2 45.1| da16
1 466 379 25424 7.3 72 1338 8.5 54.3| da17
2 474 389 25372 7.0 70 1331 11.3 56.2| da18
1 473 388 25395 7.9 70 1325 9.0 51.8| da19
0 1 0 0 0.0 0 0 0.0 0.0| da20
0 1 0 0 0.0 0 0 0.0 0.1| da21
1 533 0 0 0.0 518 26362 9.7 63.2| da22
LibreNMS also shows the increased disk activity: 16.25 hours.
While that happened, the system was operational and several jails were updated from FreeBSD 12.1 to FreeBSD 12.2
Other things during the resilver
Please ignore nvd0 and nvd1 in the legend. There is a long list of drives in this system and they are not shown. Those drives are not involved in the resilver.
Well, that’s extensive
This is all the messages I found related to this suspect drive.
EDIT This list has been updated to include mps0 since first publishd.
[dan@knew:~] $ egrep -e 'ZFS|smart|mps0' /var/log/messages.1 Dec 12 00:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 01:23:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 03:23:05 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 05:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 07:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 09:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 11:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 13:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 15:23:05 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 15:35:47 knew ZFS[70753]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$18428760864140250121 Dec 12 15:53:05 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 17:23:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 19:23:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 21:23:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 21:37:36 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1758 loginfo 31080000 Dec 12 21:37:36 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1694 loginfo 31080000 Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 bd f5 78 00 00 01 00 00 00 Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 bd f6 78 00 00 01 00 00 00 Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 bd f4 78 00 00 01 00 00 00 Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): Info: 0x241bdf4b0 Dec 12 21:37:36 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:37:40 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1186 loginfo 31080000 Dec 12 21:37:40 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1587 loginfo 31080000 Dec 12 21:37:40 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1187 loginfo 31080000 Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 02 a0 00 00 01 00 00 00 Dec 12 21:37:40 knew kernel: mps0: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:40 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 721 loginfo 31080000 Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 03 a0 00 00 01 00 00 00 Dec 12 21:37:40 knew kernel: mps0: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 46 19 50 38 00 00 00 10 00 00 Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 46 19 52 38 00 00 00 10 00 00 Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 04 a0 00 00 00 18 00 00 Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): WRITE(10). CDB: 2a 00 65 3a 20 80 00 00 08 00 Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 02 38 00 00 00 20 00 00 Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): Info: 0x241be0248 Dec 12 21:37:40 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:37:43 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 480 loginfo 31080000 Dec 12 21:37:43 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2052 loginfo 31080000 Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 0a 18 00 00 00 38 00 00 Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): WRITE(16). CDB: 8a 00 00 00 00 02 41 be 02 40 00 00 00 08 00 00 Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 09 18 00 00 00 b8 00 00 Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): Info: 0x241be0918 Dec 12 21:37:43 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:37:43 knew ZFS[87517]: checksum mismatch, zpool=$system path=$ offset=$4962763681792 size=$4096 Dec 12 21:37:43 knew ZFS[87518]: checksum mismatch, zpool=$system path=$ offset=$4962761875456 size=$12288 Dec 12 21:37:43 knew ZFS[87519]: checksum mismatch, zpool=$system path=$ offset=$4962761920512 size=$12288 Dec 12 21:37:43 knew ZFS[87520]: checksum mismatch, zpool=$system path=$ offset=$4962762067968 size=$4096 Dec 12 21:37:43 knew ZFS[87521]: checksum mismatch, zpool=$system path=$ offset=$4962761965568 size=$16384 Dec 12 21:37:43 knew ZFS[87522]: checksum mismatch, zpool=$system path=$ offset=$4962762137600 size=$8192 Dec 12 21:37:43 knew ZFS[87523]: checksum mismatch, zpool=$system path=$ offset=$4962762129408 size=$8192 Dec 12 21:37:43 knew ZFS[87524]: checksum mismatch, zpool=$system path=$ offset=$4962762145792 size=$8192 Dec 12 21:37:43 knew ZFS[87525]: checksum mismatch, zpool=$system path=$ offset=$4962762121216 size=$8192 Dec 12 21:37:43 knew ZFS[87526]: checksum mismatch, zpool=$system path=$ offset=$4962762153984 size=$8192 Dec 12 21:37:43 knew ZFS[87527]: checksum mismatch, zpool=$system path=$ offset=$4962762113024 size=$8192 Dec 12 21:37:43 knew ZFS[87528]: checksum mismatch, zpool=$system path=$ offset=$4962762104832 size=$8192 Dec 12 21:37:43 knew ZFS[87529]: checksum mismatch, zpool=$system path=$ offset=$4962762166272 size=$4096 Dec 12 21:37:43 knew ZFS[87530]: checksum mismatch, zpool=$system path=$ offset=$4962762096640 size=$8192 Dec 12 21:37:43 knew ZFS[87531]: checksum mismatch, zpool=$system path=$ offset=$4962762080256 size=$8192 Dec 12 21:37:43 knew ZFS[87532]: checksum mismatch, zpool=$system path=$ offset=$4962762088448 size=$8192 Dec 12 21:37:43 knew ZFS[87533]: checksum mismatch, zpool=$system path=$ offset=$4962762256384 size=$4096 Dec 12 21:37:43 knew ZFS[87534]: checksum mismatch, zpool=$system path=$ offset=$4962762182656 size=$8192 Dec 12 21:37:43 knew ZFS[87535]: checksum mismatch, zpool=$system path=$ offset=$4962762248192 size=$4096 Dec 12 21:37:43 knew ZFS[87536]: checksum mismatch, zpool=$system path=$ offset=$4962762190848 size=$8192 Dec 12 21:37:43 knew ZFS[87537]: checksum mismatch, zpool=$system path=$ offset=$4962762252288 size=$4096 Dec 12 21:37:43 knew ZFS[87538]: checksum mismatch, zpool=$system path=$ offset=$4962762240000 size=$4096 Dec 12 21:37:43 knew ZFS[87539]: checksum mismatch, zpool=$system path=$ offset=$4962762174464 size=$8192 Dec 12 21:37:43 knew ZFS[87540]: checksum mismatch, zpool=$system path=$ offset=$4962762264576 size=$4096 Dec 12 21:37:43 knew ZFS[87541]: checksum mismatch, zpool=$system path=$ offset=$4962762276864 size=$4096 Dec 12 21:37:43 knew ZFS[87542]: checksum mismatch, zpool=$system path=$ offset=$4962762231808 size=$8192 Dec 12 21:37:43 knew ZFS[87543]: checksum mismatch, zpool=$system path=$ offset=$4962762268672 size=$8192 Dec 12 21:37:43 knew ZFS[87544]: checksum mismatch, zpool=$system path=$ offset=$4962762244096 size=$4096 Dec 12 21:37:43 knew ZFS[87545]: checksum mismatch, zpool=$system path=$ offset=$4962763677696 size=$4096 Dec 12 21:37:43 knew ZFS[87546]: checksum mismatch, zpool=$system path=$ offset=$4962763689984 size=$4096 Dec 12 21:37:43 knew ZFS[87547]: checksum mismatch, zpool=$system path=$ offset=$4962762010624 size=$57344 Dec 12 21:37:46 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2123 loginfo 31080000 Dec 12 21:37:46 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 729 loginfo 31080000 Dec 12 21:37:46 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1496 loginfo 31080000 Dec 12 21:37:46 knew kernel: mps0: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 10 d0 00 00 00 c8 00 00 Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 10 30 00 00 00 58 00 00 Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): WRITE(10). CDB: 2a 00 92 b7 e5 c8 00 01 00 00 Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 46 19 52 38 00 00 00 10 00 00 Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 0f 30 00 00 01 00 00 00 Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): Info: 0x241be0fe0 Dec 12 21:37:46 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:37:46 knew ZFS[87555]: checksum mismatch, zpool=$system path=$ offset=$4962764607488 size=$12288 Dec 12 21:37:46 knew ZFS[87556]: checksum mismatch, zpool=$system path=$ offset=$4962764591104 size=$12288 Dec 12 21:37:46 knew ZFS[87557]: checksum mismatch, zpool=$system path=$ offset=$4962764578816 size=$12288 Dec 12 21:37:46 knew ZFS[87558]: checksum mismatch, zpool=$system path=$ offset=$4962764619776 size=$53248 Dec 12 21:37:46 knew ZFS[87559]: checksum mismatch, zpool=$system path=$ offset=$4962765537280 size=$8192 Dec 12 21:37:46 knew ZFS[87560]: checksum mismatch, zpool=$system path=$ offset=$4962765545472 size=$8192 Dec 12 21:37:46 knew ZFS[87561]: checksum mismatch, zpool=$system path=$ offset=$4962765443072 size=$8192 Dec 12 21:37:46 knew ZFS[87562]: checksum mismatch, zpool=$system path=$ offset=$4962765463552 size=$12288 Dec 12 21:37:46 knew ZFS[87563]: checksum mismatch, zpool=$system path=$ offset=$4962765475840 size=$8192 Dec 12 21:37:46 knew ZFS[87564]: checksum mismatch, zpool=$system path=$ offset=$4962765430784 size=$12288 Dec 12 21:37:46 knew ZFS[87567]: checksum mismatch, zpool=$system path=$ offset=$4962765484032 size=$12288 Dec 12 21:37:46 knew ZFS[87568]: checksum mismatch, zpool=$system path=$ offset=$4962765451264 size=$12288 Dec 12 21:37:46 knew ZFS[87573]: checksum mismatch, zpool=$system path=$ offset=$4962765524992 size=$12288 Dec 12 21:37:46 knew ZFS[87574]: checksum mismatch, zpool=$system path=$ offset=$4962765377536 size=$53248 Dec 12 21:37:48 knew ZFS[87783]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$5840140110512920130 Dec 12 21:37:51 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1639 loginfo 31080000 Dec 12 21:37:51 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1180 loginfo 31080000 Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 17 10 00 00 00 18 00 00 Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): WRITE(10). CDB: 2a 00 64 67 6d 40 00 00 20 00 Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 16 80 00 00 00 40 00 00 Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): Info: 0x241be16b8 Dec 12 21:37:51 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:37:51 knew ZFS[87971]: checksum mismatch, zpool=$system path=$ offset=$4962766356480 size=$12288 Dec 12 21:37:55 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1605 loginfo 31080000 Dec 12 21:37:55 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2009 loginfo 31080000 Dec 12 21:37:55 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1855 loginfo 31080000 Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 1e 18 00 00 01 00 00 00 Dec 12 21:37:55 knew kernel: mps0: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:55 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1530 loginfo 31080000 Dec 12 21:37:55 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2086 loginfo 31080000 Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): WRITE(16). CDB: 8a 00 00 00 00 02 41 be 15 80 00 00 01 00 00 00 Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): WRITE(16). CDB: 8a 00 00 00 00 02 41 be 16 80 00 00 00 28 00 00 Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 1f 18 00 00 01 00 00 00 Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 20 18 00 00 00 08 00 00 Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): WRITE(10). CDB: 2a 00 64 67 6e 10 00 00 20 00 Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 1d 48 00 00 00 60 00 00 Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): Info: 0x241be1d80 Dec 12 21:37:55 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:37:55 knew ZFS[87973]: checksum mismatch, zpool=$system path=$ offset=$4962766249984 size=$8192 Dec 12 21:37:55 knew ZFS[87974]: checksum mismatch, zpool=$system path=$ offset=$4962766336000 size=$8192 Dec 12 21:37:55 knew ZFS[87975]: checksum mismatch, zpool=$system path=$ offset=$4962766237696 size=$12288 Dec 12 21:37:55 knew ZFS[87976]: checksum mismatch, zpool=$system path=$ offset=$4962766323712 size=$12288 Dec 12 21:37:55 knew ZFS[87977]: checksum mismatch, zpool=$system path=$ offset=$4962766225408 size=$12288 Dec 12 21:37:55 knew ZFS[87978]: checksum mismatch, zpool=$system path=$ offset=$4962766311424 size=$12288 Dec 12 21:37:55 knew ZFS[87979]: checksum mismatch, zpool=$system path=$ offset=$4962766344192 size=$12288 Dec 12 21:37:55 knew ZFS[87980]: checksum mismatch, zpool=$system path=$ offset=$4962766217216 size=$8192 Dec 12 21:37:55 knew ZFS[87981]: checksum mismatch, zpool=$system path=$ offset=$4962766204928 size=$12288 Dec 12 21:37:55 knew ZFS[87982]: checksum mismatch, zpool=$system path=$ offset=$4962766258176 size=$53248 Dec 12 21:37:58 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1559 loginfo 31080000 Dec 12 21:37:58 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 301 loginfo 31080000 Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): WRITE(16). CDB: 8a 00 00 00 00 02 41 be 1d 48 00 00 00 18 00 00 Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 2b a8 00 00 00 98 00 00 Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 2b 08 00 00 00 50 00 00 Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): Info: 0x241be2b18 Dec 12 21:37:58 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:37:58 knew ZFS[87984]: checksum mismatch, zpool=$system path=$ offset=$4962767224832 size=$12288 Dec 12 21:37:58 knew ZFS[87985]: checksum mismatch, zpool=$system path=$ offset=$4962767265792 size=$8192 Dec 12 21:37:58 knew ZFS[87986]: checksum mismatch, zpool=$system path=$ offset=$4962769031168 size=$8192 Dec 12 21:37:58 knew ZFS[87987]: checksum mismatch, zpool=$system path=$ offset=$4962769027072 size=$4096 Dec 12 21:38:02 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2044 loginfo 31080000 Dec 12 21:38:02 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 834 loginfo 31080000 Dec 12 21:38:02 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2173 loginfo 31080000 Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 3a 20 00 00 00 88 00 00 Dec 12 21:38:02 knew kernel: mps0: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 3d 48 00 00 00 30 00 00 Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): WRITE(16). CDB: 8a 00 00 00 00 02 41 be 2b 08 00 00 00 08 00 00 Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): WRITE(10). CDB: 2a 00 64 67 6e 50 00 00 20 00 Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 39 20 00 00 01 00 00 00 Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): Info: 0x241be3928 Dec 12 21:38:02 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:38:02 knew ZFS[87996]: checksum mismatch, zpool=$system path=$ offset=$4962769063936 size=$4096 Dec 12 21:38:02 knew ZFS[87997]: checksum mismatch, zpool=$system path=$ offset=$4962771062784 size=$4096 Dec 12 21:38:02 knew ZFS[87998]: checksum mismatch, zpool=$system path=$ offset=$4962771066880 size=$4096 Dec 12 21:38:02 knew ZFS[87999]: checksum mismatch, zpool=$system path=$ offset=$4962771042304 size=$8192 Dec 12 21:38:03 knew ZFS[88000]: checksum mismatch, zpool=$system path=$ offset=$4962770984960 size=$8192 Dec 12 21:38:03 knew ZFS[88001]: checksum mismatch, zpool=$system path=$ offset=$4962770898944 size=$8192 Dec 12 21:38:03 knew ZFS[88002]: checksum mismatch, zpool=$system path=$ offset=$4962770890752 size=$8192 Dec 12 21:38:03 knew ZFS[88003]: checksum mismatch, zpool=$system path=$ offset=$4962770976768 size=$8192 Dec 12 21:38:03 knew ZFS[88004]: checksum mismatch, zpool=$system path=$ offset=$4962770882560 size=$8192 Dec 12 21:38:03 knew ZFS[88005]: checksum mismatch, zpool=$system path=$ offset=$4962770968576 size=$8192 Dec 12 21:38:03 knew ZFS[88006]: checksum mismatch, zpool=$system path=$ offset=$4962770993152 size=$8192 Dec 12 21:38:03 knew ZFS[88007]: checksum mismatch, zpool=$system path=$ offset=$4962770874368 size=$8192 Dec 12 21:38:03 knew ZFS[88008]: checksum mismatch, zpool=$system path=$ offset=$4962770907136 size=$61440 Dec 12 21:38:03 knew ZFS[88009]: checksum mismatch, zpool=$system path=$ offset=$4962771034112 size=$8192 Dec 12 21:38:03 knew ZFS[88010]: checksum mismatch, zpool=$system path=$ offset=$4962771050496 size=$8192 Dec 12 21:38:03 knew ZFS[88011]: checksum mismatch, zpool=$system path=$ offset=$4962771070976 size=$4096 Dec 12 21:38:04 knew ZFS[88032]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$5840140110512920130 Dec 12 21:38:07 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1387 loginfo 31080000 Dec 12 21:38:07 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 281 loginfo 31080000 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): WRITE(10). CDB: 2a 00 92 b7 ef 50 00 01 00 00 Dec 12 21:38:07 knew kernel: mps0: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 8d e8 00 00 01 00 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 997 loginfo 31080000 Dec 12 21:38:07 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2061 loginfo 31080000 Dec 12 21:38:07 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2015 loginfo 31080000 Dec 12 21:38:07 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1706 loginfo 31080000 Dec 12 21:38:07 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 526 loginfo 31080000 Dec 12 21:38:07 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1266 loginfo 31080000 Dec 12 21:38:07 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 380 loginfo 31080000 Dec 12 21:38:07 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 733 loginfo 31080000 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 8e e8 00 00 01 00 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 8b 70 00 00 01 00 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 8c 70 00 00 01 00 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 8d 70 00 00 00 30 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 8f e8 00 00 01 00 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 90 e8 00 00 01 00 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 91 e8 00 00 01 00 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 92 e8 00 00 01 00 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 93 e8 00 00 00 40 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 8a 70 00 00 01 00 00 00 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Info: 0x241be8ae0 Dec 12 21:38:07 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:38:11 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 595 loginfo 31080000 Dec 12 21:38:11 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be ad 20 00 00 00 90 00 00 Dec 12 21:38:11 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:11 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:11 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be ac 20 00 00 01 00 00 00 Dec 12 21:38:11 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:38:11 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:38:11 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:38:11 knew kernel: (da17:mps0:0:27:0): Info: 0x241beac60 Dec 12 21:38:11 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:38:16 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 363 loginfo 31080000 Dec 12 21:38:16 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be ae f8 00 00 00 b0 00 00 Dec 12 21:38:16 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:16 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:16 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be ad f8 00 00 01 00 00 00 Dec 12 21:38:16 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:38:16 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:38:16 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:38:16 knew kernel: (da17:mps0:0:27:0): Info: 0x241beae80 Dec 12 21:38:16 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:38:16 knew ZFS[88455]: checksum mismatch, zpool=$system path=$ offset=$4962781937664 size=$4096 Dec 12 21:38:16 knew ZFS[88456]: checksum mismatch, zpool=$system path=$ offset=$4962781347840 size=$8192 Dec 12 21:38:16 knew ZFS[88457]: checksum mismatch, zpool=$system path=$ offset=$4962781282304 size=$8192 Dec 12 21:38:16 knew ZFS[88458]: checksum mismatch, zpool=$system path=$ offset=$4962781315072 size=$8192 Dec 12 21:38:16 knew ZFS[88459]: checksum mismatch, zpool=$system path=$ offset=$4962781335552 size=$12288 Dec 12 21:38:16 knew ZFS[88460]: checksum mismatch, zpool=$system path=$ offset=$4962781302784 size=$12288 Dec 12 21:38:16 knew ZFS[88461]: checksum mismatch, zpool=$system path=$ offset=$4962781323264 size=$12288 Dec 12 21:38:16 knew ZFS[88462]: checksum mismatch, zpool=$system path=$ offset=$4962781290496 size=$12288 Dec 12 21:38:16 knew ZFS[88463]: checksum mismatch, zpool=$system path=$ offset=$4962781270016 size=$12288 Dec 12 21:38:17 knew ZFS[88464]: checksum mismatch, zpool=$system path=$ offset=$4962781413376 size=$4096 Dec 12 21:38:17 knew ZFS[88465]: checksum mismatch, zpool=$system path=$ offset=$4962781384704 size=$4096 Dec 12 21:38:17 knew ZFS[88466]: checksum mismatch, zpool=$system path=$ offset=$4962781392896 size=$4096 Dec 12 21:38:17 knew ZFS[88467]: checksum mismatch, zpool=$system path=$ offset=$4962781409280 size=$4096 Dec 12 21:38:17 knew ZFS[88468]: checksum mismatch, zpool=$system path=$ offset=$4962781405184 size=$4096 Dec 12 21:38:17 knew ZFS[88469]: checksum mismatch, zpool=$system path=$ offset=$4962781401088 size=$4096 Dec 12 21:38:17 knew ZFS[88470]: checksum mismatch, zpool=$system path=$ offset=$4962781396992 size=$4096 Dec 12 21:38:17 knew ZFS[88471]: checksum mismatch, zpool=$system path=$ offset=$4962781388800 size=$4096 Dec 12 21:38:17 knew ZFS[88472]: checksum mismatch, zpool=$system path=$ offset=$4962781376512 size=$4096 Dec 12 21:38:17 knew ZFS[88473]: checksum mismatch, zpool=$system path=$ offset=$4962781380608 size=$4096 Dec 12 21:38:17 knew ZFS[88474]: checksum mismatch, zpool=$system path=$ offset=$4962781372416 size=$4096 Dec 12 21:38:17 knew ZFS[88475]: checksum mismatch, zpool=$system path=$ offset=$4962781360128 size=$4096 Dec 12 21:38:17 knew ZFS[88476]: checksum mismatch, zpool=$system path=$ offset=$4962781364224 size=$8192 Dec 12 21:38:17 knew ZFS[88477]: checksum mismatch, zpool=$system path=$ offset=$4962781437952 size=$4096 Dec 12 21:38:17 knew ZFS[88478]: checksum mismatch, zpool=$system path=$ offset=$4962781429760 size=$8192 Dec 12 21:38:17 knew ZFS[88479]: checksum mismatch, zpool=$system path=$ offset=$4962781425664 size=$4096 Dec 12 21:38:17 knew ZFS[88480]: checksum mismatch, zpool=$system path=$ offset=$4962781421568 size=$4096 Dec 12 21:38:17 knew ZFS[88481]: checksum mismatch, zpool=$system path=$ offset=$4962781458432 size=$4096 Dec 12 21:38:17 knew ZFS[88482]: checksum mismatch, zpool=$system path=$ offset=$4962781487104 size=$4096 Dec 12 21:38:17 knew ZFS[88483]: checksum mismatch, zpool=$system path=$ offset=$4962781450240 size=$4096 Dec 12 21:38:17 knew ZFS[88484]: checksum mismatch, zpool=$system path=$ offset=$4962781454336 size=$4096 Dec 12 21:38:17 knew ZFS[88485]: checksum mismatch, zpool=$system path=$ offset=$4962781483008 size=$4096 Dec 12 21:38:17 knew ZFS[88486]: checksum mismatch, zpool=$system path=$ offset=$4962781474816 size=$8192 Dec 12 21:38:17 knew ZFS[88487]: checksum mismatch, zpool=$system path=$ offset=$4962781466624 size=$8192 Dec 12 21:38:17 knew ZFS[88488]: checksum mismatch, zpool=$system path=$ offset=$4962781446144 size=$4096 Dec 12 21:38:17 knew ZFS[88489]: checksum mismatch, zpool=$system path=$ offset=$4962781495296 size=$4096 Dec 12 21:38:17 knew ZFS[88490]: checksum mismatch, zpool=$system path=$ offset=$4962781462528 size=$4096 Dec 12 21:38:17 knew ZFS[88491]: checksum mismatch, zpool=$system path=$ offset=$4962781581312 size=$4096 Dec 12 21:38:17 knew ZFS[88492]: checksum mismatch, zpool=$system path=$ offset=$4962781925376 size=$4096 Dec 12 21:38:17 knew ZFS[88493]: checksum mismatch, zpool=$system path=$ offset=$4962781913088 size=$4096 Dec 12 21:38:17 knew ZFS[88494]: checksum mismatch, zpool=$system path=$ offset=$4962781917184 size=$8192 Dec 12 21:38:17 knew ZFS[88495]: checksum mismatch, zpool=$system path=$ offset=$4962781892608 size=$8192 Dec 12 21:38:17 knew ZFS[88496]: checksum mismatch, zpool=$system path=$ offset=$4962781929472 size=$8192 Dec 12 21:38:17 knew ZFS[88497]: checksum mismatch, zpool=$system path=$ offset=$4962781941760 size=$8192 Dec 12 21:38:17 knew ZFS[88498]: checksum mismatch, zpool=$system path=$ offset=$4962781704192 size=$4096 Dec 12 21:38:17 knew ZFS[88499]: checksum mismatch, zpool=$system path=$ offset=$4962781851648 size=$4096 Dec 12 21:38:17 knew ZFS[88500]: checksum mismatch, zpool=$system path=$ offset=$4962781777920 size=$4096 Dec 12 21:38:17 knew ZFS[88501]: checksum mismatch, zpool=$system path=$ offset=$4962781904896 size=$8192 Dec 12 21:38:17 knew ZFS[88502]: checksum mismatch, zpool=$system path=$ offset=$4962781888512 size=$4096 Dec 12 21:38:17 knew ZFS[88503]: checksum mismatch, zpool=$system path=$ offset=$4962781876224 size=$4096 Dec 12 21:38:17 knew ZFS[88504]: checksum mismatch, zpool=$system path=$ offset=$4962781765632 size=$4096 Dec 12 21:38:17 knew ZFS[88505]: checksum mismatch, zpool=$system path=$ offset=$4962781528064 size=$45056 Dec 12 21:38:17 knew ZFS[88506]: checksum mismatch, zpool=$system path=$ offset=$4962781614080 size=$4096 Dec 12 21:38:17 knew ZFS[88507]: checksum mismatch, zpool=$system path=$ offset=$4962781831168 size=$8192 Dec 12 21:38:17 knew ZFS[88508]: checksum mismatch, zpool=$system path=$ offset=$4962781720576 size=$8192 Dec 12 21:38:17 knew ZFS[88509]: checksum mismatch, zpool=$system path=$ offset=$4962781794304 size=$8192 Dec 12 21:38:17 knew ZFS[88510]: checksum mismatch, zpool=$system path=$ offset=$4962781782016 size=$8192 Dec 12 21:38:17 knew ZFS[88511]: checksum mismatch, zpool=$system path=$ offset=$4962781671424 size=$8192 Dec 12 21:38:17 knew ZFS[88512]: checksum mismatch, zpool=$system path=$ offset=$4962781708288 size=$8192 Dec 12 21:38:17 knew ZFS[88513]: checksum mismatch, zpool=$system path=$ offset=$4962781650944 size=$4096 Dec 12 21:38:17 knew ZFS[88514]: checksum mismatch, zpool=$system path=$ offset=$4962781827072 size=$4096 Dec 12 21:38:17 knew ZFS[88515]: checksum mismatch, zpool=$system path=$ offset=$4962781609984 size=$4096 Dec 12 21:38:17 knew ZFS[88516]: checksum mismatch, zpool=$system path=$ offset=$4962781679616 size=$4096 Dec 12 21:38:17 knew ZFS[88517]: checksum mismatch, zpool=$system path=$ offset=$4962781900800 size=$4096 Dec 12 21:38:17 knew ZFS[88518]: checksum mismatch, zpool=$system path=$ offset=$4962781642752 size=$4096 Dec 12 21:38:17 knew ZFS[88519]: checksum mismatch, zpool=$system path=$ offset=$4962781593600 size=$4096 Dec 12 21:38:17 knew ZFS[88520]: checksum mismatch, zpool=$system path=$ offset=$4962781745152 size=$8192 Dec 12 21:38:17 knew ZFS[88521]: checksum mismatch, zpool=$system path=$ offset=$4962781818880 size=$8192 Dec 12 21:38:17 knew ZFS[88522]: checksum mismatch, zpool=$system path=$ offset=$4962781802496 size=$4096 Dec 12 21:38:17 knew ZFS[88523]: checksum mismatch, zpool=$system path=$ offset=$4962781863936 size=$4096 Dec 12 21:38:17 knew ZFS[88524]: checksum mismatch, zpool=$system path=$ offset=$4962781880320 size=$8192 Dec 12 21:38:17 knew ZFS[88525]: checksum mismatch, zpool=$system path=$ offset=$4962781855744 size=$8192 Dec 12 21:38:17 knew ZFS[88526]: checksum mismatch, zpool=$system path=$ offset=$4962781618176 size=$4096 Dec 12 21:38:17 knew ZFS[88527]: checksum mismatch, zpool=$system path=$ offset=$4962781622272 size=$8192 Dec 12 21:38:17 knew ZFS[88528]: checksum mismatch, zpool=$system path=$ offset=$4962781790208 size=$4096 Dec 12 21:38:17 knew ZFS[88529]: checksum mismatch, zpool=$system path=$ offset=$4962781814784 size=$4096 Dec 12 21:38:17 knew ZFS[88530]: checksum mismatch, zpool=$system path=$ offset=$4962781696000 size=$8192 Dec 12 21:38:17 knew ZFS[88531]: checksum mismatch, zpool=$system path=$ offset=$4962781630464 size=$12288 Dec 12 21:38:17 knew ZFS[88532]: checksum mismatch, zpool=$system path=$ offset=$4962781753344 size=$4096 Dec 12 21:38:17 knew ZFS[88533]: checksum mismatch, zpool=$system path=$ offset=$4962781757440 size=$8192 Dec 12 21:38:17 knew ZFS[88534]: checksum mismatch, zpool=$system path=$ offset=$4962781663232 size=$4096 Dec 12 21:38:17 knew ZFS[88535]: checksum mismatch, zpool=$system path=$ offset=$4962781843456 size=$8192 Dec 12 21:38:17 knew ZFS[88536]: checksum mismatch, zpool=$system path=$ offset=$4962781732864 size=$8192 Dec 12 21:38:17 knew ZFS[88537]: checksum mismatch, zpool=$system path=$ offset=$4962781646848 size=$4096 Dec 12 21:38:17 knew ZFS[88538]: checksum mismatch, zpool=$system path=$ offset=$4962781728768 size=$4096 Dec 12 21:38:17 knew ZFS[88539]: checksum mismatch, zpool=$system path=$ offset=$4962781601792 size=$4096 Dec 12 21:38:17 knew ZFS[88540]: checksum mismatch, zpool=$system path=$ offset=$4962781605888 size=$4096 Dec 12 21:38:17 knew ZFS[88541]: checksum mismatch, zpool=$system path=$ offset=$4962781655040 size=$8192 Dec 12 21:38:17 knew ZFS[88542]: checksum mismatch, zpool=$system path=$ offset=$4962781597696 size=$4096 Dec 12 21:38:17 knew ZFS[88543]: checksum mismatch, zpool=$system path=$ offset=$4962781868032 size=$8192 Dec 12 21:38:17 knew ZFS[88544]: checksum mismatch, zpool=$system path=$ offset=$4962781806592 size=$8192 Dec 12 21:38:17 knew ZFS[88545]: checksum mismatch, zpool=$system path=$ offset=$4962781839360 size=$4096 Dec 12 21:38:17 knew ZFS[88546]: checksum mismatch, zpool=$system path=$ offset=$4962781769728 size=$8192 Dec 12 21:38:17 knew ZFS[88547]: checksum mismatch, zpool=$system path=$ offset=$4962781741056 size=$4096 Dec 12 21:38:17 knew ZFS[88548]: checksum mismatch, zpool=$system path=$ offset=$4962781691904 size=$4096 Dec 12 21:38:17 knew ZFS[88549]: checksum mismatch, zpool=$system path=$ offset=$4962781716480 size=$4096 Dec 12 21:38:17 knew ZFS[88550]: checksum mismatch, zpool=$system path=$ offset=$4962781667328 size=$4096 Dec 12 21:38:17 knew ZFS[88551]: checksum mismatch, zpool=$system path=$ offset=$4962781683712 size=$8192 Dec 12 21:38:19 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1741 loginfo 31080000 Dec 12 21:38:19 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1811 loginfo 31080000 Dec 12 21:38:19 knew kernel: mps0: (da17:mps0:0:27:0): WRITE(16). CDB: 8a 00 00 00 00 02 41 be af 78 00 00 00 10 00 00 Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): WRITE(10). CDB: 2a 00 65 3a 2a d8 00 00 10 00 Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:19 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1732 loginfo 31080000 Dec 12 21:38:19 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1623 loginfo 31080000 Dec 12 21:38:19 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1074 loginfo 31080000 Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b2 88 00 00 00 e0 00 00 Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b3 c8 00 00 01 00 00 00 Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b4 c8 00 00 01 00 00 00 Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b5 c8 00 00 00 90 00 00 Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b1 88 00 00 01 00 00 00 Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): Info: 0x241beb238 Dec 12 21:38:19 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:38:19 knew ZFS[88553]: checksum mismatch, zpool=$system path=$ offset=$4962786385920 size=$8192 Dec 12 21:38:19 knew ZFS[88554]: checksum mismatch, zpool=$system path=$ offset=$4962786140160 size=$4096 Dec 12 21:38:19 knew ZFS[88555]: checksum mismatch, zpool=$system path=$ offset=$4962786127872 size=$4096 Dec 12 21:38:19 knew ZFS[88556]: checksum mismatch, zpool=$system path=$ offset=$4962786115584 size=$4096 Dec 12 21:38:19 knew ZFS[88557]: checksum mismatch, zpool=$system path=$ offset=$4962786103296 size=$4096 Dec 12 21:38:19 knew ZFS[88558]: checksum mismatch, zpool=$system path=$ offset=$4962786119680 size=$8192 Dec 12 21:38:19 knew ZFS[88559]: checksum mismatch, zpool=$system path=$ offset=$4962786107392 size=$8192 Dec 12 21:38:19 knew ZFS[88560]: checksum mismatch, zpool=$system path=$ offset=$4962786131968 size=$8192 Dec 12 21:38:19 knew ZFS[88561]: checksum mismatch, zpool=$system path=$ offset=$4962786144256 size=$8192 Dec 12 21:38:19 knew ZFS[88562]: checksum mismatch, zpool=$system path=$ offset=$4962786041856 size=$4096 Dec 12 21:38:19 knew ZFS[88563]: checksum mismatch, zpool=$system path=$ offset=$4962785935360 size=$8192 Dec 12 21:38:19 knew ZFS[88564]: checksum mismatch, zpool=$system path=$ offset=$4962785923072 size=$8192 Dec 12 21:38:19 knew ZFS[88565]: checksum mismatch, zpool=$system path=$ offset=$4962786033664 size=$8192 Dec 12 21:38:19 knew ZFS[88566]: checksum mismatch, zpool=$system path=$ offset=$4962786066432 size=$4096 Dec 12 21:38:19 knew ZFS[88567]: checksum mismatch, zpool=$system path=$ offset=$4962785865728 size=$8192 Dec 12 21:38:19 knew ZFS[88568]: checksum mismatch, zpool=$system path=$ offset=$4962785984512 size=$8192 Dec 12 21:38:19 knew ZFS[88569]: checksum mismatch, zpool=$system path=$ offset=$4962786082816 size=$8192 Dec 12 21:38:19 knew ZFS[88570]: checksum mismatch, zpool=$system path=$ offset=$4962785980416 size=$4096 Dec 12 21:38:19 knew ZFS[88571]: checksum mismatch, zpool=$system path=$ offset=$4962785861632 size=$4096 Dec 12 21:38:19 knew ZFS[88572]: checksum mismatch, zpool=$system path=$ offset=$4962786078720 size=$4096 Dec 12 21:38:19 knew ZFS[88573]: checksum mismatch, zpool=$system path=$ offset=$4962785837056 size=$4096 Dec 12 21:38:19 knew ZFS[88574]: checksum mismatch, zpool=$system path=$ offset=$4962785824768 size=$4096 Dec 12 21:38:19 knew ZFS[88575]: checksum mismatch, zpool=$system path=$ offset=$4962785968128 size=$4096 Dec 12 21:38:19 knew ZFS[88576]: checksum mismatch, zpool=$system path=$ offset=$4962786058240 size=$8192 Dec 12 21:38:19 knew ZFS[88577]: checksum mismatch, zpool=$system path=$ offset=$4962785910784 size=$8192 Dec 12 21:38:19 knew ZFS[88578]: checksum mismatch, zpool=$system path=$ offset=$4962785996800 size=$8192 Dec 12 21:38:19 knew ZFS[88579]: checksum mismatch, zpool=$system path=$ offset=$4962786070528 size=$8192 Dec 12 21:38:19 knew ZFS[88580]: checksum mismatch, zpool=$system path=$ offset=$4962785972224 size=$8192 Dec 12 21:38:19 knew ZFS[88581]: checksum mismatch, zpool=$system path=$ offset=$4962786095104 size=$8192 Dec 12 21:38:19 knew ZFS[88582]: checksum mismatch, zpool=$system path=$ offset=$4962785902592 size=$8192 Dec 12 21:38:19 knew ZFS[88583]: checksum mismatch, zpool=$system path=$ offset=$4962786021376 size=$8192 Dec 12 21:38:19 knew ZFS[88584]: checksum mismatch, zpool=$system path=$ offset=$4962786054144 size=$4096 Dec 12 21:38:19 knew ZFS[88585]: checksum mismatch, zpool=$system path=$ offset=$4962785955840 size=$4096 Dec 12 21:38:19 knew ZFS[88586]: checksum mismatch, zpool=$system path=$ offset=$4962785959936 size=$8192 Dec 12 21:38:19 knew ZFS[88587]: checksum mismatch, zpool=$system path=$ offset=$4962786004992 size=$4096 Dec 12 21:38:19 knew ZFS[88588]: checksum mismatch, zpool=$system path=$ offset=$4962785898496 size=$4096 Dec 12 21:38:19 knew ZFS[88589]: checksum mismatch, zpool=$system path=$ offset=$4962785878016 size=$8192 Dec 12 21:38:19 knew ZFS[88590]: checksum mismatch, zpool=$system path=$ offset=$4962786045952 size=$8192 Dec 12 21:38:19 knew ZFS[88591]: checksum mismatch, zpool=$system path=$ offset=$4962785890304 size=$8192 Dec 12 21:38:19 knew ZFS[88592]: checksum mismatch, zpool=$system path=$ offset=$4962785828864 size=$8192 Dec 12 21:38:19 knew ZFS[88593]: checksum mismatch, zpool=$system path=$ offset=$4962785947648 size=$8192 Dec 12 21:38:19 knew ZFS[88594]: checksum mismatch, zpool=$system path=$ offset=$4962785841152 size=$8192 Dec 12 21:38:19 knew ZFS[88595]: checksum mismatch, zpool=$system path=$ offset=$4962786029568 size=$4096 Dec 12 21:38:19 knew ZFS[88596]: checksum mismatch, zpool=$system path=$ offset=$4962785931264 size=$4096 Dec 12 21:38:19 knew ZFS[88597]: checksum mismatch, zpool=$system path=$ offset=$4962786017280 size=$4096 Dec 12 21:38:19 knew ZFS[88598]: checksum mismatch, zpool=$system path=$ offset=$4962785992704 size=$4096 Dec 12 21:38:19 knew ZFS[88599]: checksum mismatch, zpool=$system path=$ offset=$4962785853440 size=$8192 Dec 12 21:38:19 knew ZFS[88600]: checksum mismatch, zpool=$system path=$ offset=$4962786091008 size=$4096 Dec 12 21:38:19 knew ZFS[88601]: checksum mismatch, zpool=$system path=$ offset=$4962785943552 size=$4096 Dec 12 21:38:19 knew ZFS[88602]: checksum mismatch, zpool=$system path=$ offset=$4962786009088 size=$8192 Dec 12 21:38:19 knew ZFS[88603]: checksum mismatch, zpool=$system path=$ offset=$4962785918976 size=$4096 Dec 12 21:38:19 knew ZFS[88604]: checksum mismatch, zpool=$system path=$ offset=$4962785873920 size=$4096 Dec 12 21:38:19 knew ZFS[88605]: checksum mismatch, zpool=$system path=$ offset=$4962785886208 size=$4096 Dec 12 21:38:19 knew ZFS[88606]: checksum mismatch, zpool=$system path=$ offset=$4962785849344 size=$4096 Dec 12 21:38:19 knew ZFS[88607]: checksum mismatch, zpool=$system path=$ offset=$4962785816576 size=$8192 Dec 12 21:38:23 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 895 loginfo 31080000 Dec 12 21:38:23 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 347 loginfo 31080000 Dec 12 21:38:23 knew kernel: mps0: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b3 c8 00 00 01 00 00 00 Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): Retrying command, 2 more tries remain Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b4 c8 00 00 01 00 00 00 Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): Retrying command, 2 more tries remain Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b5 c8 00 00 00 90 00 00 Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): Retrying command, 2 more tries remain Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b2 88 00 00 00 e0 00 00 Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): Info: 0x241beb2e0 Dec 12 21:38:23 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:38:23 knew ZFS[88678]: checksum mismatch, zpool=$system path=$ offset=$4962786394112 size=$4096 Dec 12 21:38:23 knew ZFS[88679]: checksum mismatch, zpool=$system path=$ offset=$4962786381824 size=$4096 Dec 12 21:38:23 knew ZFS[88680]: checksum mismatch, zpool=$system path=$ offset=$4962786369536 size=$4096 Dec 12 21:38:23 knew ZFS[88681]: checksum mismatch, zpool=$system path=$ offset=$4962786357248 size=$4096 Dec 12 21:38:23 knew ZFS[88682]: checksum mismatch, zpool=$system path=$ offset=$4962786398208 size=$8192 Dec 12 21:38:23 knew ZFS[88683]: checksum mismatch, zpool=$system path=$ offset=$4962786406400 size=$4096 Dec 12 21:38:23 knew ZFS[88684]: checksum mismatch, zpool=$system path=$ offset=$4962786361344 size=$8192 Dec 12 21:38:23 knew ZFS[88685]: checksum mismatch, zpool=$system path=$ offset=$4962786373632 size=$8192 Dec 12 21:38:23 knew ZFS[88686]: checksum mismatch, zpool=$system path=$ offset=$4962786320384 size=$4096 Dec 12 21:38:23 knew ZFS[88687]: checksum mismatch, zpool=$system path=$ offset=$4962786275328 size=$8192 Dec 12 21:38:23 knew ZFS[88688]: checksum mismatch, zpool=$system path=$ offset=$4962786283520 size=$4096 Dec 12 21:38:23 knew ZFS[88689]: checksum mismatch, zpool=$system path=$ offset=$4962786349056 size=$8192 Dec 12 21:38:23 knew ZFS[88690]: checksum mismatch, zpool=$system path=$ offset=$4962786258944 size=$4096 Dec 12 21:38:23 knew ZFS[88691]: checksum mismatch, zpool=$system path=$ offset=$4962786324480 size=$8192 Dec 12 21:38:23 knew ZFS[88692]: checksum mismatch, zpool=$system path=$ offset=$4962786299904 size=$8192 Dec 12 21:38:23 knew ZFS[88693]: checksum mismatch, zpool=$system path=$ offset=$4962786238464 size=$8192 Dec 12 21:38:23 knew ZFS[88694]: checksum mismatch, zpool=$system path=$ offset=$4962786308096 size=$4096 Dec 12 21:38:23 knew ZFS[88695]: checksum mismatch, zpool=$system path=$ offset=$4962786271232 size=$4096 Dec 12 21:38:23 knew ZFS[88696]: checksum mismatch, zpool=$system path=$ offset=$4962786344960 size=$4096 Dec 12 21:38:23 knew ZFS[88697]: checksum mismatch, zpool=$system path=$ offset=$4962786201600 size=$8192 Dec 12 21:38:23 knew ZFS[88698]: checksum mismatch, zpool=$system path=$ offset=$4962786189312 size=$8192 Dec 12 21:38:23 knew ZFS[88699]: checksum mismatch, zpool=$system path=$ offset=$4962786287616 size=$8192 Dec 12 21:38:23 knew ZFS[88700]: checksum mismatch, zpool=$system path=$ offset=$4962786336768 size=$8192 Dec 12 21:38:23 knew ZFS[88701]: checksum mismatch, zpool=$system path=$ offset=$4962786226176 size=$8192 Dec 12 21:38:23 knew ZFS[88702]: checksum mismatch, zpool=$system path=$ offset=$4962786246656 size=$4096 Dec 12 21:38:23 knew ZFS[88703]: checksum mismatch, zpool=$system path=$ offset=$4962786332672 size=$4096 Dec 12 21:38:23 knew ZFS[88704]: checksum mismatch, zpool=$system path=$ offset=$4962786263040 size=$8192 Dec 12 21:38:23 knew ZFS[88705]: checksum mismatch, zpool=$system path=$ offset=$4962786312192 size=$8192 Dec 12 21:38:23 knew ZFS[88706]: checksum mismatch, zpool=$system path=$ offset=$4962786213888 size=$8192 Dec 12 21:38:23 knew ZFS[88707]: checksum mismatch, zpool=$system path=$ offset=$4962786295808 size=$4096 Dec 12 21:38:23 knew ZFS[88708]: checksum mismatch, zpool=$system path=$ offset=$4962786197504 size=$4096 Dec 12 21:38:23 knew ZFS[88709]: checksum mismatch, zpool=$system path=$ offset=$4962786250752 size=$8192 Dec 12 21:38:23 knew ZFS[88710]: checksum mismatch, zpool=$system path=$ offset=$4962786234368 size=$4096 Dec 12 21:38:23 knew ZFS[88711]: checksum mismatch, zpool=$system path=$ offset=$4962786222080 size=$4096 Dec 12 21:38:23 knew ZFS[88712]: checksum mismatch, zpool=$system path=$ offset=$4962786209792 size=$4096 Dec 12 21:38:28 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1503 loginfo 31080000 Dec 12 21:38:28 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b6 a0 00 00 00 18 00 00 Dec 12 21:38:28 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:28 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:28 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b5 c8 00 00 00 90 00 00 Dec 12 21:38:28 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:38:28 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:38:28 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:38:28 knew kernel: (da17:mps0:0:27:0): Info: 0x241beb640 Dec 12 21:38:28 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:38:32 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 974 loginfo 31080000 Dec 12 21:38:32 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1335 loginfo 31080000 Dec 12 21:38:32 knew kernel: mps0: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b9 70 00 00 01 00 00 00 Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be ba 70 00 00 00 30 00 00 Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:32 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 148 loginfo 31080000 Dec 12 21:38:32 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 552 loginfo 31080000 Dec 12 21:38:32 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 982 loginfo 31080000 Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be bb 28 00 00 01 00 00 00 Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be bc 28 00 00 01 00 00 00 Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be bd 28 00 00 00 30 00 00 Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): WRITE(16). CDB: 8a 00 00 00 00 02 41 be af 88 00 00 00 20 00 00 Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be b8 70 00 00 01 00 00 00 Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): Info: 0x241beb870 Dec 12 21:38:32 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:38:35 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 701 loginfo 31080000 Dec 12 21:38:35 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2129 loginfo 31080000 Dec 12 21:38:35 knew kernel: mps0: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be ba 70 00 00 00 30 00 00 Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): Retrying command, 2 more tries remain Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be bc 28 00 00 01 00 00 00 Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): Retrying command, 2 more tries remain Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be bd 28 00 00 00 30 00 00 Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): Retrying command, 2 more tries remain Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be bb 28 00 00 01 00 00 00 Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): Info: 0x241bebba0 Dec 12 21:38:35 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:38:35 knew ZFS[88723]: checksum mismatch, zpool=$system path=$ offset=$4962786889728 size=$4096 Dec 12 21:38:35 knew ZFS[88724]: checksum mismatch, zpool=$system path=$ offset=$4962786865152 size=$4096 Dec 12 21:38:35 knew ZFS[88725]: checksum mismatch, zpool=$system path=$ offset=$4962786877440 size=$4096 Dec 12 21:38:35 knew ZFS[88726]: checksum mismatch, zpool=$system path=$ offset=$4962786852864 size=$4096 Dec 12 21:38:35 knew ZFS[88727]: checksum mismatch, zpool=$system path=$ offset=$4962786881536 size=$8192 Dec 12 21:38:35 knew ZFS[88728]: checksum mismatch, zpool=$system path=$ offset=$4962786869248 size=$8192 Dec 12 21:38:35 knew ZFS[88730]: checksum mismatch, zpool=$system path=$ offset=$4962786893824 size=$8192 Dec 12 21:38:35 knew ZFS[88731]: checksum mismatch, zpool=$system path=$ offset=$4962786856960 size=$8192 Dec 12 21:38:35 knew ZFS[88732]: checksum mismatch, zpool=$system path=$ offset=$4962786840576 size=$4096 Dec 12 21:38:35 knew ZFS[88733]: checksum mismatch, zpool=$system path=$ offset=$4962786816000 size=$4096 Dec 12 21:38:35 knew ZFS[88734]: checksum mismatch, zpool=$system path=$ offset=$4962786803712 size=$4096 Dec 12 21:38:35 knew ZFS[88735]: checksum mismatch, zpool=$system path=$ offset=$4962786729984 size=$4096 Dec 12 21:38:35 knew ZFS[88736]: checksum mismatch, zpool=$system path=$ offset=$4962786779136 size=$4096 Dec 12 21:38:35 knew ZFS[88737]: checksum mismatch, zpool=$system path=$ offset=$4962786742272 size=$4096 Dec 12 21:38:35 knew ZFS[88738]: checksum mismatch, zpool=$system path=$ offset=$4962786844672 size=$8192 Dec 12 21:38:35 knew ZFS[88739]: checksum mismatch, zpool=$system path=$ offset=$4962786807808 size=$8192 Dec 12 21:38:35 knew ZFS[88740]: checksum mismatch, zpool=$system path=$ offset=$4962786709504 size=$8192 Dec 12 21:38:35 knew ZFS[88741]: checksum mismatch, zpool=$system path=$ offset=$4962786656256 size=$4096 Dec 12 21:38:35 knew ZFS[88742]: checksum mismatch, zpool=$system path=$ offset=$4962786791424 size=$4096 Dec 12 21:38:35 knew ZFS[88743]: checksum mismatch, zpool=$system path=$ offset=$4962786734080 size=$8192 Dec 12 21:38:35 knew ZFS[88744]: checksum mismatch, zpool=$system path=$ offset=$4962786828288 size=$4096 Dec 12 21:38:35 knew ZFS[88745]: checksum mismatch, zpool=$system path=$ offset=$4962786697216 size=$8192 Dec 12 21:38:35 knew ZFS[88746]: checksum mismatch, zpool=$system path=$ offset=$4962786832384 size=$8192 Dec 12 21:38:35 knew ZFS[88747]: checksum mismatch, zpool=$system path=$ offset=$4962786766848 size=$4096 Dec 12 21:38:35 knew ZFS[88748]: checksum mismatch, zpool=$system path=$ offset=$4962786754560 size=$4096 Dec 12 21:38:35 knew ZFS[88749]: checksum mismatch, zpool=$system path=$ offset=$4962786783232 size=$8192 Dec 12 21:38:35 knew ZFS[88750]: checksum mismatch, zpool=$system path=$ offset=$4962786717696 size=$4096 Dec 12 21:38:35 knew ZFS[88751]: checksum mismatch, zpool=$system path=$ offset=$4962786693120 size=$4096 Dec 12 21:38:35 knew ZFS[88752]: checksum mismatch, zpool=$system path=$ offset=$4962786795520 size=$8192 Dec 12 21:38:35 knew ZFS[88753]: checksum mismatch, zpool=$system path=$ offset=$4962786820096 size=$8192 Dec 12 21:38:35 knew ZFS[88754]: checksum mismatch, zpool=$system path=$ offset=$4962786770944 size=$8192 Dec 12 21:38:35 knew ZFS[88755]: checksum mismatch, zpool=$system path=$ offset=$4962786672640 size=$8192 Dec 12 21:38:35 knew ZFS[88756]: checksum mismatch, zpool=$system path=$ offset=$4962786705408 size=$4096 Dec 12 21:38:35 knew ZFS[88757]: checksum mismatch, zpool=$system path=$ offset=$4962786668544 size=$4096 Dec 12 21:38:35 knew ZFS[88758]: checksum mismatch, zpool=$system path=$ offset=$4962786680832 size=$4096 Dec 12 21:38:35 knew ZFS[88759]: checksum mismatch, zpool=$system path=$ offset=$4962786721792 size=$8192 Dec 12 21:38:35 knew ZFS[88760]: checksum mismatch, zpool=$system path=$ offset=$4962786684928 size=$8192 Dec 12 21:38:35 knew ZFS[88761]: checksum mismatch, zpool=$system path=$ offset=$4962786746368 size=$8192 Dec 12 21:38:35 knew ZFS[88762]: checksum mismatch, zpool=$system path=$ offset=$4962786660352 size=$8192 Dec 12 21:38:35 knew ZFS[88763]: checksum mismatch, zpool=$system path=$ offset=$4962786758656 size=$8192 Dec 12 21:38:37 knew ZFS[88766]: checksum mismatch, zpool=$system path=$ offset=$4962787246080 size=$4096 Dec 12 21:38:37 knew ZFS[88767]: checksum mismatch, zpool=$system path=$ offset=$4962787233792 size=$4096 Dec 12 21:38:37 knew ZFS[88768]: checksum mismatch, zpool=$system path=$ offset=$4962787258368 size=$4096 Dec 12 21:38:37 knew ZFS[88769]: checksum mismatch, zpool=$system path=$ offset=$4962787270656 size=$4096 Dec 12 21:38:37 knew ZFS[88770]: checksum mismatch, zpool=$system path=$ offset=$4962787262464 size=$8192 Dec 12 21:38:37 knew ZFS[88771]: checksum mismatch, zpool=$system path=$ offset=$4962787250176 size=$8192 Dec 12 21:38:37 knew ZFS[88772]: checksum mismatch, zpool=$system path=$ offset=$4962787237888 size=$8192 Dec 12 21:38:37 knew ZFS[88773]: checksum mismatch, zpool=$system path=$ offset=$4962787209216 size=$4096 Dec 12 21:38:37 knew ZFS[88774]: checksum mismatch, zpool=$system path=$ offset=$4962787147776 size=$4096 Dec 12 21:38:37 knew ZFS[88775]: checksum mismatch, zpool=$system path=$ offset=$4962787024896 size=$4096 Dec 12 21:38:37 knew ZFS[88776]: checksum mismatch, zpool=$system path=$ offset=$4962787225600 size=$8192 Dec 12 21:38:37 knew ZFS[88777]: checksum mismatch, zpool=$system path=$ offset=$4962786951168 size=$4096 Dec 12 21:38:37 knew ZFS[88778]: checksum mismatch, zpool=$system path=$ offset=$4962787151872 size=$8192 Dec 12 21:38:37 knew ZFS[88779]: checksum mismatch, zpool=$system path=$ offset=$4962787127296 size=$8192 Dec 12 21:38:37 knew ZFS[88780]: checksum mismatch, zpool=$system path=$ offset=$4962787078144 size=$8192 Dec 12 21:38:37 knew ZFS[88781]: checksum mismatch, zpool=$system path=$ offset=$4962787160064 size=$4096 Dec 12 21:38:37 knew ZFS[88782]: checksum mismatch, zpool=$system path=$ offset=$4962787196928 size=$4096 Dec 12 21:38:37 knew ZFS[88783]: checksum mismatch, zpool=$system path=$ offset=$4962787221504 size=$4096 Dec 12 21:38:37 knew ZFS[88784]: checksum mismatch, zpool=$system path=$ offset=$4962787061760 size=$4096 Dec 12 21:38:37 knew ZFS[88785]: checksum mismatch, zpool=$system path=$ offset=$4962787123200 size=$4096 Dec 12 21:38:37 knew ZFS[88786]: checksum mismatch, zpool=$system path=$ offset=$4962787164160 size=$8192 Dec 12 21:38:37 knew ZFS[88787]: checksum mismatch, zpool=$system path=$ offset=$4962787188736 size=$8192 Dec 12 21:38:37 knew ZFS[88788]: checksum mismatch, zpool=$system path=$ offset=$4962787201024 size=$8192 Dec 12 21:38:37 knew ZFS[88789]: checksum mismatch, zpool=$system path=$ offset=$4962787135488 size=$4096 Dec 12 21:38:37 knew ZFS[88790]: checksum mismatch, zpool=$system path=$ offset=$4962787016704 size=$8192 Dec 12 21:38:37 knew ZFS[88791]: checksum mismatch, zpool=$system path=$ offset=$4962787139584 size=$8192 Dec 12 21:38:37 knew ZFS[88792]: checksum mismatch, zpool=$system path=$ offset=$4962786979840 size=$8192 Dec 12 21:38:37 knew ZFS[88793]: checksum mismatch, zpool=$system path=$ offset=$4962786963456 size=$4096 Dec 12 21:38:37 knew ZFS[88794]: checksum mismatch, zpool=$system path=$ offset=$4962787090432 size=$8192 Dec 12 21:38:37 knew ZFS[88795]: checksum mismatch, zpool=$system path=$ offset=$4962787184640 size=$4096 Dec 12 21:38:37 knew ZFS[88796]: checksum mismatch, zpool=$system path=$ offset=$4962787102720 size=$8192 Dec 12 21:38:37 knew ZFS[88797]: checksum mismatch, zpool=$system path=$ offset=$4962787012608 size=$4096 Dec 12 21:38:37 knew ZFS[88798]: checksum mismatch, zpool=$system path=$ offset=$4962787098624 size=$4096 Dec 12 21:38:37 knew ZFS[88799]: checksum mismatch, zpool=$system path=$ offset=$4962787172352 size=$4096 Dec 12 21:38:37 knew ZFS[88800]: checksum mismatch, zpool=$system path=$ offset=$4962787065856 size=$8192 Dec 12 21:38:37 knew ZFS[88801]: checksum mismatch, zpool=$system path=$ offset=$4962787086336 size=$4096 Dec 12 21:38:37 knew ZFS[88802]: checksum mismatch, zpool=$system path=$ offset=$4962787053568 size=$8192 Dec 12 21:38:37 knew ZFS[88803]: checksum mismatch, zpool=$system path=$ offset=$4962787176448 size=$8192 Dec 12 21:38:37 knew ZFS[88804]: checksum mismatch, zpool=$system path=$ offset=$4962787041280 size=$8192 Dec 12 21:38:37 knew ZFS[88805]: checksum mismatch, zpool=$system path=$ offset=$4962787004416 size=$8192 Dec 12 21:38:37 knew ZFS[88806]: checksum mismatch, zpool=$system path=$ offset=$4962786992128 size=$8192 Dec 12 21:38:37 knew ZFS[88807]: checksum mismatch, zpool=$system path=$ offset=$4962787110912 size=$4096 Dec 12 21:38:37 knew ZFS[88808]: checksum mismatch, zpool=$system path=$ offset=$4962787049472 size=$4096 Dec 12 21:38:37 knew ZFS[88809]: checksum mismatch, zpool=$system path=$ offset=$4962787037184 size=$4096 Dec 12 21:38:37 knew ZFS[88810]: checksum mismatch, zpool=$system path=$ offset=$4962787074048 size=$4096 Dec 12 21:38:37 knew ZFS[88811]: checksum mismatch, zpool=$system path=$ offset=$4962787115008 size=$8192 Dec 12 21:38:37 knew ZFS[88812]: checksum mismatch, zpool=$system path=$ offset=$4962786975744 size=$4096 Dec 12 21:38:37 knew ZFS[88813]: checksum mismatch, zpool=$system path=$ offset=$4962787028992 size=$8192 Dec 12 21:38:37 knew ZFS[88814]: checksum mismatch, zpool=$system path=$ offset=$4962787000320 size=$4096 Dec 12 21:38:37 knew ZFS[88815]: checksum mismatch, zpool=$system path=$ offset=$4962786988032 size=$4096 Dec 12 21:38:37 knew ZFS[88816]: checksum mismatch, zpool=$system path=$ offset=$4962786967552 size=$8192 Dec 12 21:38:37 knew ZFS[88817]: checksum mismatch, zpool=$system path=$ offset=$4962786955264 size=$8192 Dec 12 21:38:37 knew ZFS[88818]: checksum mismatch, zpool=$system path=$ offset=$4962787213312 size=$8192 Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2020 loginfo 31080000 Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1849 loginfo 31080000 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 46 19 52 38 00 00 00 10 00 00 Dec 12 21:38:42 knew kernel: mps0: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be be c0 00 00 01 00 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2100 loginfo 31080000 Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1895 loginfo 31080000 Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 892 loginfo 31080000 Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 326 loginfo 31080000 Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1330 loginfo 31080000 Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 816 loginfo 31080000 Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 1066 loginfo 31080000 Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 2042 loginfo 31080000 Dec 12 21:38:42 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 969 loginfo 31080000 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be bf c0 00 00 00 88 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(6). CDB: 08 00 02 38 10 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 46 19 50 38 00 00 00 10 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be c0 a0 00 00 01 00 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be c1 a0 00 00 01 00 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be c2 a0 00 00 01 00 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be c3 a0 00 00 01 00 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be c4 a0 00 00 01 00 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be c5 a0 00 00 01 00 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be c6 a0 00 00 00 c0 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be bd c0 00 00 01 00 00 00 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Info: 0x241bebe58 Dec 12 21:38:42 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 12 21:38:43 knew ZFS[88820]: checksum mismatch, zpool=$system path=$ offset=$4962787282944 size=$4096 Dec 12 21:38:44 knew ZFS[88821]: checksum mismatch, zpool=$system path=$ offset=$4962787917824 size=$4096 Dec 12 21:38:44 knew ZFS[88822]: checksum mismatch, zpool=$system path=$ offset=$4962787942400 size=$4096 Dec 12 21:38:44 knew ZFS[88823]: checksum mismatch, zpool=$system path=$ offset=$4962787930112 size=$4096 Dec 12 21:38:44 knew ZFS[88824]: checksum mismatch, zpool=$system path=$ offset=$4962787946496 size=$8192 Dec 12 21:38:44 knew ZFS[88825]: checksum mismatch, zpool=$system path=$ offset=$4962787934208 size=$8192 Dec 12 21:38:44 knew ZFS[88826]: checksum mismatch, zpool=$system path=$ offset=$4962787921920 size=$8192 Dec 12 21:38:44 knew ZFS[88827]: checksum mismatch, zpool=$system path=$ offset=$4962787831808 size=$4096 Dec 12 21:38:44 knew ZFS[88828]: checksum mismatch, zpool=$system path=$ offset=$4962787819520 size=$4096 Dec 12 21:38:44 knew ZFS[88829]: checksum mismatch, zpool=$system path=$ offset=$4962787807232 size=$4096 Dec 12 21:38:44 knew ZFS[88830]: checksum mismatch, zpool=$system path=$ offset=$4962787835904 size=$8192 Dec 12 21:38:44 knew ZFS[88831]: checksum mismatch, zpool=$system path=$ offset=$4962787823616 size=$8192 Dec 12 21:38:44 knew ZFS[88832]: checksum mismatch, zpool=$system path=$ offset=$4962787844096 size=$4096 Dec 12 21:38:44 knew ZFS[88833]: checksum mismatch, zpool=$system path=$ offset=$4962787811328 size=$8192 Dec 12 21:38:44 knew ZFS[88834]: checksum mismatch, zpool=$system path=$ offset=$4962787794944 size=$4096 Dec 12 21:38:44 knew ZFS[88835]: checksum mismatch, zpool=$system path=$ offset=$4962787758080 size=$4096 Dec 12 21:38:44 knew ZFS[88836]: checksum mismatch, zpool=$system path=$ offset=$4962787799040 size=$8192 Dec 12 21:38:44 knew ZFS[88837]: checksum mismatch, zpool=$system path=$ offset=$4962787745792 size=$4096 Dec 12 21:38:44 knew ZFS[88838]: checksum mismatch, zpool=$system path=$ offset=$4962787737600 size=$8192 Dec 12 21:38:44 knew ZFS[88839]: checksum mismatch, zpool=$system path=$ offset=$4962787708928 size=$4096 Dec 12 21:38:44 knew ZFS[88840]: checksum mismatch, zpool=$system path=$ offset=$4962787786752 size=$8192 Dec 12 21:38:44 knew ZFS[88841]: checksum mismatch, zpool=$system path=$ offset=$4962787770368 size=$4096 Dec 12 21:38:44 knew ZFS[88842]: checksum mismatch, zpool=$system path=$ offset=$4962787635200 size=$4096 Dec 12 21:38:44 knew ZFS[88843]: checksum mismatch, zpool=$system path=$ offset=$4962787676160 size=$8192 Dec 12 21:38:44 knew ZFS[88844]: checksum mismatch, zpool=$system path=$ offset=$4962787688448 size=$8192 Dec 12 21:38:44 knew ZFS[88845]: checksum mismatch, zpool=$system path=$ offset=$4962787782656 size=$4096 Dec 12 21:38:44 knew ZFS[88846]: checksum mismatch, zpool=$system path=$ offset=$4962787774464 size=$8192 Dec 12 21:38:44 knew ZFS[88847]: checksum mismatch, zpool=$system path=$ offset=$4962787684352 size=$4096 Dec 12 21:38:44 knew ZFS[88848]: checksum mismatch, zpool=$system path=$ offset=$4962787672064 size=$4096 Dec 12 21:38:44 knew ZFS[88849]: checksum mismatch, zpool=$system path=$ offset=$4962787598336 size=$4096 Dec 12 21:38:44 knew ZFS[88850]: checksum mismatch, zpool=$system path=$ offset=$4962787721216 size=$4096 Dec 12 21:38:44 knew ZFS[88851]: checksum mismatch, zpool=$system path=$ offset=$4962787586048 size=$4096 Dec 12 21:38:44 knew ZFS[88852]: checksum mismatch, zpool=$system path=$ offset=$4962787762176 size=$8192 Dec 12 21:38:44 knew ZFS[88853]: checksum mismatch, zpool=$system path=$ offset=$4962787725312 size=$8192 Dec 12 21:38:44 knew ZFS[88854]: checksum mismatch, zpool=$system path=$ offset=$4962787749888 size=$8192 Dec 12 21:38:44 knew ZFS[88855]: checksum mismatch, zpool=$system path=$ offset=$4962787733504 size=$4096 Dec 12 21:38:44 knew ZFS[88856]: checksum mismatch, zpool=$system path=$ offset=$4962787659776 size=$4096 Dec 12 21:38:44 knew ZFS[88857]: checksum mismatch, zpool=$system path=$ offset=$4962787639296 size=$8192 Dec 12 21:38:44 knew ZFS[88858]: checksum mismatch, zpool=$system path=$ offset=$4962787647488 size=$4096 Dec 12 21:38:44 knew ZFS[88859]: checksum mismatch, zpool=$system path=$ offset=$4962787713024 size=$8192 Dec 12 21:38:44 knew ZFS[88860]: checksum mismatch, zpool=$system path=$ offset=$4962787700736 size=$8192 Dec 12 21:38:44 knew ZFS[88861]: checksum mismatch, zpool=$system path=$ offset=$4962787627008 size=$8192 Dec 12 21:38:44 knew ZFS[88862]: checksum mismatch, zpool=$system path=$ offset=$4962787696640 size=$4096 Dec 12 21:38:44 knew ZFS[88863]: checksum mismatch, zpool=$system path=$ offset=$4962787622912 size=$4096 Dec 12 21:38:44 knew ZFS[88864]: checksum mismatch, zpool=$system path=$ offset=$4962787610624 size=$4096 Dec 12 21:38:44 knew ZFS[88865]: checksum mismatch, zpool=$system path=$ offset=$4962787573760 size=$4096 Dec 12 21:38:44 knew ZFS[88866]: checksum mismatch, zpool=$system path=$ offset=$4962787651584 size=$8192 Dec 12 21:38:44 knew ZFS[88867]: checksum mismatch, zpool=$system path=$ offset=$4962787663872 size=$8192 Dec 12 21:38:44 knew ZFS[88868]: checksum mismatch, zpool=$system path=$ offset=$4962787602432 size=$8192 Dec 12 21:38:44 knew ZFS[88869]: checksum mismatch, zpool=$system path=$ offset=$4962787614720 size=$8192 Dec 12 21:38:44 knew ZFS[88870]: checksum mismatch, zpool=$system path=$ offset=$4962787577856 size=$8192 Dec 12 21:38:44 knew ZFS[88871]: checksum mismatch, zpool=$system path=$ offset=$4962787565568 size=$8192 Dec 12 21:38:44 knew ZFS[88872]: checksum mismatch, zpool=$system path=$ offset=$4962787590144 size=$8192 Dec 12 21:38:44 knew ZFS[88873]: checksum mismatch, zpool=$system path=$ offset=$4962787561472 size=$4096 Dec 12 21:38:44 knew ZFS[88874]: checksum mismatch, zpool=$system path=$ offset=$4962787954688 size=$61440 Dec 12 21:38:46 knew ZFS[88875]: checksum mismatch, zpool=$system path=$ offset=$4962788167680 size=$4096 Dec 12 21:38:46 knew ZFS[88876]: checksum mismatch, zpool=$system path=$ offset=$4962788171776 size=$4096 Dec 12 21:38:46 knew ZFS[88877]: checksum mismatch, zpool=$system path=$ offset=$4962788130816 size=$4096 Dec 12 21:38:46 knew ZFS[88878]: checksum mismatch, zpool=$system path=$ offset=$4962788155392 size=$4096 Dec 12 21:38:46 knew ZFS[88879]: checksum mismatch, zpool=$system path=$ offset=$4962788143104 size=$4096 Dec 12 21:38:46 knew ZFS[88880]: checksum mismatch, zpool=$system path=$ offset=$4962788147200 size=$8192 Dec 12 21:38:46 knew ZFS[88881]: checksum mismatch, zpool=$system path=$ offset=$4962788134912 size=$8192 Dec 12 21:38:46 knew ZFS[88882]: checksum mismatch, zpool=$system path=$ offset=$4962788159488 size=$8192 Dec 12 21:38:46 knew ZFS[88883]: checksum mismatch, zpool=$system path=$ offset=$4962788106240 size=$4096 Dec 12 21:38:46 knew ZFS[88884]: checksum mismatch, zpool=$system path=$ offset=$4962788110336 size=$8192 Dec 12 21:38:46 knew ZFS[88885]: checksum mismatch, zpool=$system path=$ offset=$4962788069376 size=$4096 Dec 12 21:38:46 knew ZFS[88886]: checksum mismatch, zpool=$system path=$ offset=$4962788122624 size=$8192 Dec 12 21:38:46 knew ZFS[88887]: checksum mismatch, zpool=$system path=$ offset=$4962788057088 size=$4096 Dec 12 21:38:46 knew ZFS[88888]: checksum mismatch, zpool=$system path=$ offset=$4962788085760 size=$8192 Dec 12 21:38:46 knew ZFS[88889]: checksum mismatch, zpool=$system path=$ offset=$4962788098048 size=$8192 Dec 12 21:38:46 knew ZFS[88890]: checksum mismatch, zpool=$system path=$ offset=$4962788036608 size=$8192 Dec 12 21:38:46 knew ZFS[88891]: checksum mismatch, zpool=$system path=$ offset=$4962788081664 size=$4096 Dec 12 21:38:46 knew ZFS[88892]: checksum mismatch, zpool=$system path=$ offset=$4962788118528 size=$4096 Dec 12 21:38:46 knew ZFS[88893]: checksum mismatch, zpool=$system path=$ offset=$4962788044800 size=$4096 Dec 12 21:38:46 knew ZFS[88894]: checksum mismatch, zpool=$system path=$ offset=$4962788048896 size=$8192 Dec 12 21:38:46 knew ZFS[88895]: checksum mismatch, zpool=$system path=$ offset=$4962788032512 size=$4096 Dec 12 21:38:46 knew ZFS[88896]: checksum mismatch, zpool=$system path=$ offset=$4962788093952 size=$4096 Dec 12 21:38:46 knew ZFS[88897]: checksum mismatch, zpool=$system path=$ offset=$4962788073472 size=$8192 Dec 12 21:38:46 knew ZFS[88898]: checksum mismatch, zpool=$system path=$ offset=$4962788061184 size=$8192 Dec 12 21:38:46 knew ZFS[88899]: checksum mismatch, zpool=$system path=$ offset=$4962788020224 size=$4096 Dec 12 21:38:46 knew ZFS[88900]: checksum mismatch, zpool=$system path=$ offset=$4962788024320 size=$8192 Dec 12 21:38:46 knew ZFS[88901]: checksum mismatch, zpool=$system path=$ offset=$4962788179968 size=$4096 Dec 12 21:38:46 knew ZFS[88902]: checksum mismatch, zpool=$system path=$ offset=$4962788200448 size=$4096 Dec 12 21:38:46 knew ZFS[88903]: checksum mismatch, zpool=$system path=$ offset=$4962788184064 size=$8192 Dec 12 21:38:46 knew ZFS[88904]: checksum mismatch, zpool=$system path=$ offset=$4962788265984 size=$4096 Dec 12 21:38:46 knew ZFS[88905]: checksum mismatch, zpool=$system path=$ offset=$4962788192256 size=$8192 Dec 12 21:38:46 knew ZFS[88906]: checksum mismatch, zpool=$system path=$ offset=$4962788270080 size=$8192 Dec 12 21:38:46 knew ZFS[88907]: checksum mismatch, zpool=$system path=$ offset=$4962788257792 size=$8192 Dec 12 21:38:46 knew ZFS[88908]: checksum mismatch, zpool=$system path=$ offset=$4962788564992 size=$4096 Dec 12 21:38:46 knew ZFS[88909]: checksum mismatch, zpool=$system path=$ offset=$4962788577280 size=$4096 Dec 12 21:38:46 knew ZFS[88910]: checksum mismatch, zpool=$system path=$ offset=$4962788540416 size=$4096 Dec 12 21:38:46 knew ZFS[88911]: checksum mismatch, zpool=$system path=$ offset=$4962788552704 size=$4096 Dec 12 21:38:46 knew ZFS[88912]: checksum mismatch, zpool=$system path=$ offset=$4962788569088 size=$8192 Dec 12 21:38:46 knew ZFS[88913]: checksum mismatch, zpool=$system path=$ offset=$4962788544512 size=$8192 Dec 12 21:38:46 knew ZFS[88914]: checksum mismatch, zpool=$system path=$ offset=$4962788556800 size=$8192 Dec 12 21:38:46 knew ZFS[88915]: checksum mismatch, zpool=$system path=$ offset=$4962788581376 size=$8192 Dec 12 21:38:46 knew ZFS[88916]: checksum mismatch, zpool=$system path=$ offset=$4962788491264 size=$4096 Dec 12 21:38:46 knew ZFS[88917]: checksum mismatch, zpool=$system path=$ offset=$4962788532224 size=$8192 Dec 12 21:38:46 knew ZFS[88918]: checksum mismatch, zpool=$system path=$ offset=$4962788503552 size=$4096 Dec 12 21:38:46 knew ZFS[88919]: checksum mismatch, zpool=$system path=$ offset=$4962788343808 size=$4096 Dec 12 21:38:46 knew ZFS[88920]: checksum mismatch, zpool=$system path=$ offset=$4962788368384 size=$4096 Dec 12 21:38:46 knew ZFS[88921]: checksum mismatch, zpool=$system path=$ offset=$4962788519936 size=$8192 Dec 12 21:38:46 knew ZFS[88922]: checksum mismatch, zpool=$system path=$ offset=$4962788478976 size=$4096 Dec 12 21:38:46 knew ZFS[88923]: checksum mismatch, zpool=$system path=$ offset=$4962788515840 size=$4096 Dec 12 21:38:46 knew ZFS[88924]: checksum mismatch, zpool=$system path=$ offset=$4962788470784 size=$8192 Dec 12 21:38:46 knew ZFS[88925]: checksum mismatch, zpool=$system path=$ offset=$4962788356096 size=$4096 Dec 12 21:38:46 knew ZFS[88926]: checksum mismatch, zpool=$system path=$ offset=$4962788421632 size=$8192 Dec 12 21:38:46 knew ZFS[88927]: checksum mismatch, zpool=$system path=$ offset=$4962788483072 size=$8192 Dec 12 21:38:46 knew ZFS[88928]: checksum mismatch, zpool=$system path=$ offset=$4962788319232 size=$4096 Dec 12 21:38:46 knew ZFS[88929]: checksum mismatch, zpool=$system path=$ offset=$4962788507648 size=$8192 Dec 12 21:38:46 knew ZFS[88930]: checksum mismatch, zpool=$system path=$ offset=$4962788495360 size=$8192 Dec 12 21:38:46 knew ZFS[88931]: checksum mismatch, zpool=$system path=$ offset=$4962788458496 size=$8192 Dec 12 21:38:46 knew ZFS[88932]: checksum mismatch, zpool=$system path=$ offset=$4962788306944 size=$4096 Dec 12 21:38:46 knew ZFS[88933]: checksum mismatch, zpool=$system path=$ offset=$4962788298752 size=$8192 Dec 12 21:38:46 knew ZFS[88934]: checksum mismatch, zpool=$system path=$ offset=$4962788466688 size=$4096 Dec 12 21:38:46 knew ZFS[88935]: checksum mismatch, zpool=$system path=$ offset=$4962788372480 size=$8192 Dec 12 21:38:46 knew ZFS[88936]: checksum mismatch, zpool=$system path=$ offset=$4962788528128 size=$4096 Dec 12 21:38:46 knew ZFS[88937]: checksum mismatch, zpool=$system path=$ offset=$4962788446208 size=$8192 Dec 12 21:38:46 knew ZFS[88938]: checksum mismatch, zpool=$system path=$ offset=$4962788347904 size=$8192 Dec 12 21:38:46 knew ZFS[88939]: checksum mismatch, zpool=$system path=$ offset=$4962788323328 size=$8192 Dec 12 21:38:46 knew ZFS[88940]: checksum mismatch, zpool=$system path=$ offset=$4962788442112 size=$4096 Dec 12 21:38:46 knew ZFS[88941]: checksum mismatch, zpool=$system path=$ offset=$4962788454400 size=$4096 Dec 12 21:38:46 knew ZFS[88942]: checksum mismatch, zpool=$system path=$ offset=$4962788417536 size=$4096 Dec 12 21:38:46 knew ZFS[88943]: checksum mismatch, zpool=$system path=$ offset=$4962788409344 size=$8192 Dec 12 21:38:46 knew ZFS[88944]: checksum mismatch, zpool=$system path=$ offset=$4962788311040 size=$8192 Dec 12 21:38:46 knew ZFS[88945]: checksum mismatch, zpool=$system path=$ offset=$4962788294656 size=$4096 Dec 12 21:38:46 knew ZFS[88946]: checksum mismatch, zpool=$system path=$ offset=$4962788397056 size=$8192 Dec 12 21:38:46 knew ZFS[88947]: checksum mismatch, zpool=$system path=$ offset=$4962788433920 size=$8192 Dec 12 21:38:46 knew ZFS[88948]: checksum mismatch, zpool=$system path=$ offset=$4962788429824 size=$4096 Dec 12 21:38:46 knew ZFS[88949]: checksum mismatch, zpool=$system path=$ offset=$4962788405248 size=$4096 Dec 12 21:38:46 knew ZFS[88950]: checksum mismatch, zpool=$system path=$ offset=$4962788392960 size=$4096 Dec 12 21:38:46 knew ZFS[88951]: checksum mismatch, zpool=$system path=$ offset=$4962788380672 size=$4096 Dec 12 21:38:46 knew ZFS[88952]: checksum mismatch, zpool=$system path=$ offset=$4962788384768 size=$8192 Dec 12 21:38:46 knew ZFS[88953]: checksum mismatch, zpool=$system path=$ offset=$4962788335616 size=$8192 Dec 12 21:38:46 knew ZFS[88954]: checksum mismatch, zpool=$system path=$ offset=$4962788331520 size=$4096 Dec 12 21:38:46 knew ZFS[88955]: checksum mismatch, zpool=$system path=$ offset=$4962788360192 size=$8192 Dec 12 21:38:47 knew ZFS[88957]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$5840140110512920130 Dec 12 21:53:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 21:53:04 knew smartd[2124]: Device: /dev/da17 [SAT], ATA error count increased from 393 to 410 Dec 12 22:23:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 23:23:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors
There are some CKSUM errors on the status now:
[dan@knew:~] $ zpool status systemt pool: system state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sat Dec 12 15:35:49 2020 32.4T scanned at 1.33G/s, 28.5T issued at 1.17G/s, 58.5T total 886G resilvered, 48.78% done, 0 days 07:16:21 to go config: NAME STATE READ WRITE CKSUM system ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da3p3 ONLINE 0 0 0 da10p3 ONLINE 0 0 0 da9p3 ONLINE 0 0 0 da2p3 ONLINE 0 0 0 da13p3 ONLINE 0 0 0 da15p3 ONLINE 0 0 0 da11p3 ONLINE 0 0 0 da14p3 ONLINE 0 0 0 da8p3 ONLINE 0 0 0 da7p3 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 da5p1 ONLINE 0 0 0 da6p1 ONLINE 0 0 0 da19p1 ONLINE 0 0 0 da12p1 ONLINE 0 0 0 da4p1 ONLINE 0 0 0 da1p1 ONLINE 0 0 0 replacing-6 ONLINE 0 0 492 da17p1 ONLINE 0 0 0 da22p1 ONLINE 0 0 0 da16p1 ONLINE 0 0 0 da0p1 ONLINE 0 0 0 da18p1 ONLINE 0 0 0 errors: No known data errors [dan@knew:~] $
The next morning
The next morning, I noticed the resilver had completed. I spent the rest of the day working on FreshPorts. Later, that night, I looked in the log files and saw the following.
EDIT: This list was updated to include mps0 output after the original publication.
[dan@knew:~] $ egrep -e 'ZFS|smart|mps0' /var/log/messages.0 Dec 13 00:23:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 13 01:23:05 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 13 03:00:23 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 686 loginfo 31080000 Dec 13 03:00:23 knew kernel: mps0: Controller reported scsi ioc terminated tgt 27 SMID 532 loginfo 31080000 Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 25 d0 00 00 00 10 00 00 Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): READ(10). CDB: 28 00 77 a2 a6 88 00 01 00 00 Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): CAM status: CCB request completed with an error Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): Retrying command, 3 more tries remain Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): READ(16). CDB: 88 00 00 00 00 02 41 be 24 48 00 00 00 10 00 00 Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): CAM status: SCSI Status Error Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): SCSI status: Check Condition Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): Info: 0x241be2448 Dec 13 03:00:23 knew kernel: (da17:mps0:0:27:0): Error 5, Unretryable error Dec 13 03:00:23 knew ZFS[24985]: checksum mismatch, zpool=$system path=$ offset=$4962768142336 size=$8192 Dec 13 03:00:27 knew ZFS[24989]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$5840140110512920130 Dec 13 03:23:05 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 13 03:23:05 knew smartd[2124]: Device: /dev/da17 [SAT], ATA error count increased from 410 to 411 Dec 13 03:53:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 13 05:23:05 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 13 05:28:01 knew ZFS[55296]: vdev state changed, pool_guid=$15879539137961201777 vdev_guid=$10611488957753538131 Dec 13 05:28:01 knew ZFS[55297]: vdev state changed, pool_guid=$15879539137961201777 vdev_guid=$8845306092815458666 Dec 13 05:53:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 13 07:23:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 13 07:51:42 knew ZFS[95169]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$5892227802261634203 Dec 13 07:51:42 knew ZFS[95170]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$9332658639709199239 Dec 13 07:51:42 knew ZFS[95171]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$250004220145174872 Dec 13 07:51:42 knew ZFS[95172]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$6216472763074854678 Dec 13 07:51:42 knew ZFS[95173]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$12795310201775582855 Dec 13 07:51:42 knew ZFS[95174]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$13315402097660581553 Dec 13 07:51:42 knew ZFS[95175]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$18428760864140250121 Dec 13 07:51:42 knew ZFS[95176]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$13603535286907309607 Dec 13 07:51:42 knew ZFS[95178]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$4677401754715191854 Dec 13 07:51:42 knew ZFS[95179]: vdev state changed, pool_guid=$15378250086669402288 vdev_guid=$1933292688604201684 Dec 13 07:53:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 13 09:23:04 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors
da17 is still throwing errors, but we don’t care because it is no longer part of a zpool.
I may try tossing data at that dirve and see if that changes the status.
For interest, here is the resilver presented by LibreNMS.
