For a few days now, this drive in x8dtu has been having a hard time. I think it’s time to buy a replacement. Or two.
[16:37 x8dtu dvl ~] % grep ada3 /var/log/messages Jul 4 15:19:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 16:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 17:19:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 18:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 19:20:00 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 19:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 20:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 21:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 21:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 22:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 23:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 4 23:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 00:19:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 01:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 02:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 02:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 03:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 04:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 04:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 05:50:00 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 06:50:05 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 07:19:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 07:50:00 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 08:49:57 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 09:49:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 10:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 10:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 11:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 12:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 13:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 13:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 14:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 15:49:57 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 16:19:57 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 17:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 18:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 19:19:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 20:19:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 21:20:00 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 22:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 22:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 5 23:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 00:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 01:19:57 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 02:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 03:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 04:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 04:49:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 05:49:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 06:49:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 07:19:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 07:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 08:50:02 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 09:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 10:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 10:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 11:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 12:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 13:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 13:49:57 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 14:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 15:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 16:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 16:50:00 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 17:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 17:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 18:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 19:19:57 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 20:20:02 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 21:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 22:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 6 23:19:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 00:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 01:19:57 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 02:19:57 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 03:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 04:20:00 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 05:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 06:19:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 06:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 07:20:01 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 07:49:59 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 08:50:01 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 09:49:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 10:19:58 x8dtu smartd[8217]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 fd e2 ed d2 40 47 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 ff df ee d2 40 47 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 fd de ef d2 40 47 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 fb db f0 d2 40 47 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 48 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 ff d6 f1 d2 40 47 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 88 ff bb 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 c8 ff bb 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 ff d5 f2 d2 40 47 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu ZFS[38268]: vdev I/O failure, zpool=main_tank path=/dev/ada3p1 offset=270336 size=8192 error=5 Jul 7 10:40:08 x8dtu ZFS[38272]: vdev I/O failure, zpool=main_tank path=/dev/ada3p1 offset=4998267674624 size=8192 error=5 Jul 7 10:40:08 x8dtu ZFS[38276]: vdev I/O failure, zpool=main_tank path=/dev/ada3p1 offset=4998267936768 size=8192 error=5 Jul 7 10:40:08 x8dtu ZFS[38280]: vdev probe failure, zpool=main_tank path=/dev/ada3p1 Jul 7 10:40:08 x8dtu kernel: vdev_geom_close_locked:352[1]: Closing access to ada3p1. Jul 7 10:40:08 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada3p1. Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT ) Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): RES: 41 04 00 00 00 40 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Retrying command, 0 more tries remain Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT ) Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): RES: 41 04 00 00 00 40 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Retries exhausted Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Synchronize cache failed Jul 7 10:40:08 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada3p1. Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 04 f7 1a c6 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 34 1b c6 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 34 1b c6 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 10 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 02 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 00 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 40 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 10 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 02 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 00 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 40 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 08 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 00 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 02 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 10 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 00 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 04 c7 ff bb 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 04 00 bc 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 04 00 bc 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 07 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 06 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 16 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 08 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 06 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 46 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 16 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 08 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 06 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 46 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 0e 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 06 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 08 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 16 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 06 00 00 40 00 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 10:40:08 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 10:49:58 x8dtu smartd[8217]: Device: /dev/ada3, failed to read SMART Attribute Data Jul 7 11:49:58 x8dtu smartd[8217]: Device: /dev/ada3, failed to read SMART Attribute Data Jul 7 12:49:58 x8dtu smartd[8217]: Device: /dev/ada3, failed to read SMART Attribute Data Jul 7 13:19:58 x8dtu smartd[8217]: Device: /dev/ada3, failed to read SMART Attribute Data Jul 7 13:49:58 x8dtu smartd[8217]: Device: /dev/ada3, failed to read SMART Attribute Data Jul 7 14:49:58 x8dtu smartd[8217]: Device: /dev/ada3, failed to read SMART Attribute Data Jul 7 15:49:57 x8dtu smartd[8217]: Device: /dev/ada3, failed to read SMART Attribute Data Jul 7 16:19:59 x8dtu smartd[8217]: Device: /dev/ada3, failed to read SMART Attribute Data
And the status:
[16:37 x8dtu dvl ~] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT main_tank 4.53T 1.26T 3.28T - - 25% 27% 1.00x ONLINE - zroot 212G 56.6G 155G - - 50% 26% 1.00x ONLINE - [16:38 x8dtu dvl ~] % zpool status pool: main_tank state: ONLINE status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0B in 06:54:39 with 0 errors on Mon Jul 7 10:41:58 2025 config: NAME STATE READ WRITE CKSUM main_tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada2p1 ONLINE 0 0 0 ada3p1 FAULTED 21 379 254 too many errors errors: No known data errors pool: zroot state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:03:53 with 0 errors on Mon Jul 7 03:51:20 2025 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 errors: No known data errors
Let’s try clearing:
[16:38 x8dtu dvl ~] % sudo zpool clear main_tank [16:38 x8dtu dvl ~] % sudo zpool status main_tank pool: main_tank state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0B in 06:54:39 with 0 errors on Mon Jul 7 10:41:58 2025 config: NAME STATE READ WRITE CKSUM main_tank DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 ada2p1 ONLINE 0 0 0 ada3p1 FAULTED 3 0 0 too many errors errors: No known data errors [16:38 x8dtu dvl ~] % date Mon Jul 7 16:39:26 UTC 2025 [16:38 x8dtu dvl ~] %
HHmm. Let’s try again:
[16:40 x8dtu dvl ~] % sudo zpool clear main_tank [16:40 x8dtu dvl ~] % sudo zpool status main_tank pool: main_tank state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0B in 06:54:39 with 0 errors on Mon Jul 7 10:41:58 2025 config: NAME STATE READ WRITE CKSUM main_tank DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 ada2p1 ONLINE 0 0 0 ada3p1 FAULTED 3 0 0 too many errors errors: No known data errors [16:40 x8dtu dvl ~] %
Well, look at this:
Jul 7 16:38:46 x8dtu kernel: vdev_geom_open_by_path:799[1]: Found provider by name /dev/ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada3p1... Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 0a 00 00 40 00 00 00 00 00 00 Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 4a 00 00 40 00 00 00 00 00 00 Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 8a ff bb 40 48 00 00 00 00 00 Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c ca ff bb 40 48 00 00 00 00 00 Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_attach_ok:667[1]: Unable to read config from ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_open_by_guids:766[1]: Searching by guids [376274468418206170:15764092042190275056]. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada2p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:268[1]: Found consumer for ada2p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:296[1]: Used existing consumer for ada2p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada2p1... Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada2p1. Jul 7 16:38:46 x8dtu kernel: vdev_attach_ok:703[1]: vdev guid mismatch for provider ada2p1: 15764092042190275056 != 15252581311252777810. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada3p1... Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 0a 00 00 40 00 00 00 00 00 00 Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 4a 00 00 40 00 00 00 00 00 00 Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 8a ff bb 40 48 00 00 00 00 00 Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c ca ff bb 40 48 00 00 00 00 00 Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:46 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_attach_ok:667[1]: Unable to read config from ada3p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada1p3. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:268[1]: Found consumer for ada1p3. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:296[1]: Used existing consumer for ada1p3. Jul 7 16:38:46 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada1p3... Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada1p3. Jul 7 16:38:46 x8dtu kernel: vdev_attach_ok:686[1]: pool guid mismatch for provider ada1p3: 376274468418206170 != 18320603570228782289. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada1p2. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:284[1]: vdev_geom_attach(284): g_access failed: 1 Jul 7 16:38:46 x8dtu kernel: Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada1p2. Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada1p2. Jul 7 16:38:46 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada1p2. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada1p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:230[1]: Failing attach of ada1p1. Incompatible mediasize 524288 Jul 7 16:38:46 x8dtu kernel: Jul 7 16:38:46 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada1p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada0p3. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:268[1]: Found consumer for ada0p3. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:296[1]: Used existing consumer for ada0p3. Jul 7 16:38:46 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada0p3... Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada0p3. Jul 7 16:38:46 x8dtu kernel: vdev_attach_ok:686[1]: pool guid mismatch for provider ada0p3: 376274468418206170 != 18320603570228782289. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada0p2. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:284[1]: vdev_geom_attach(284): g_access failed: 1 Jul 7 16:38:46 x8dtu kernel: Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada0p2. Jul 7 16:38:46 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada0p2. Jul 7 16:38:46 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada0p2. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada0p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:230[1]: Failing attach of ada0p1. Incompatible mediasize 524288 Jul 7 16:38:46 x8dtu kernel: Jul 7 16:38:46 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada0p1. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to gpt/gptboot0. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:230[1]: Failing attach of gpt/gptboot0. Incompatible mediasize 524288 Jul 7 16:38:46 x8dtu kernel: Jul 7 16:38:46 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to gpt/gptboot0. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to iso9660/11_0_RELEASE_P1_AMD64_CD. Jul 7 16:38:46 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for iso9660/11_0_RELEASE_P1_AMD64_CD. Jul 7 16:38:46 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from iso9660/11_0_RELEASE_P1_AMD64_CD... Jul 7 16:38:48 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from iso9660/11_0_RELEASE_P1_AMD64_CD. Jul 7 16:38:48 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for iso9660/11_0_RELEASE_P1_AMD64_CD. Jul 7 16:38:48 x8dtu kernel: vdev_attach_ok:667[1]: Unable to read config from iso9660/11_0_RELEASE_P1_AMD64_CD. Jul 7 16:38:48 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to gpt/gptboot1. Jul 7 16:38:48 x8dtu kernel: vdev_geom_attach:230[1]: Failing attach of gpt/gptboot1. Incompatible mediasize 524288 Jul 7 16:38:48 x8dtu kernel: Jul 7 16:38:48 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to gpt/gptboot1. Jul 7 16:38:48 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to cd0. Jul 7 16:38:48 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for cd0. Jul 7 16:38:48 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from cd0... Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from cd0. Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for cd0. Jul 7 16:38:49 x8dtu kernel: vdev_attach_ok:667[1]: Unable to read config from cd0. Jul 7 16:38:49 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada2. Jul 7 16:38:49 x8dtu kernel: vdev_geom_attach:284[1]: vdev_geom_attach(284): g_access failed: 1 Jul 7 16:38:49 x8dtu kernel: Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada2. Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada2. Jul 7 16:38:49 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada2. Jul 7 16:38:49 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada3. Jul 7 16:38:49 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for ada3. Jul 7 16:38:49 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada3... Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 04 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 44 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 84 1a c6 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c c4 1a c6 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada3. Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada3. Jul 7 16:38:49 x8dtu kernel: vdev_attach_ok:667[1]: Unable to read config from ada3. Jul 7 16:38:49 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada1. Jul 7 16:38:49 x8dtu kernel: vdev_geom_attach:284[1]: vdev_geom_attach(284): g_access failed: 1 Jul 7 16:38:49 x8dtu kernel: Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada1. Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada1. Jul 7 16:38:49 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada1. Jul 7 16:38:49 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada0. Jul 7 16:38:49 x8dtu kernel: vdev_geom_attach:284[1]: vdev_geom_attach(284): g_access failed: 1 Jul 7 16:38:49 x8dtu kernel: Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada0. Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada0. Jul 7 16:38:49 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada0. Jul 7 16:38:49 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada3p1. Jul 7 16:38:49 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for ada3p1. Jul 7 16:38:49 x8dtu kernel: vdev_geom_open_by_guids:778[1]: Attach by guid [376274468418206170:15764092042190275056] succeeded, provider ada3p1. Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 88 ff bb 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 c8 ff bb 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 48 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: vdev_geom_close_locked:352[1]: Closing access to ada3p1. Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada3p1. Jul 7 16:38:49 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada3p1. Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 04 f7 1a c6 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 34 1b c6 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 34 1b c6 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 10 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 02 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 00 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 40 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 10 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 02 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 00 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 40 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 08 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 00 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 02 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 10 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 00 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 04 c7 ff bb 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu ZFS[1318]: vdev state changed, pool_guid=376274468418206170 vdev_guid=15764092042190275056 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 04 00 bc 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 04 00 bc 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 07 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 06 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 16 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 08 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 06 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 46 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 16 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 08 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 06 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 46 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 0e 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 06 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 08 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 16 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 06 00 00 40 00 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:38:49 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:38:49 x8dtu ZFS[1322]: vdev I/O failure, zpool=main_tank path=/dev/ada3p1 offset=4998267674624 size=8192 error=5 Jul 7 16:38:49 x8dtu ZFS[1326]: vdev I/O failure, zpool=main_tank path=/dev/ada3p1 offset=4998267936768 size=8192 error=5 Jul 7 16:38:49 x8dtu ZFS[1330]: vdev I/O failure, zpool=main_tank path=/dev/ada3p1 offset=270336 size=8192 error=5 Jul 7 16:38:49 x8dtu ZFS[1334]: vdev probe failure, zpool=main_tank path=/dev/ada3p1 Jul 7 16:38:49 x8dtu ZFS[1340]: vdev state changed, pool_guid=376274468418206170 vdev_guid=15764092042190275056 Jul 7 16:40:32 x8dtu kernel: vdev_geom_open_by_path:799[1]: Found provider by name /dev/ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada3p1... Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 4a 00 00 40 00 00 00 00 00 00 Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 8a ff bb 40 48 00 00 00 00 00 Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c ca ff bb 40 48 00 00 00 00 00 Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 0a 00 00 40 00 00 00 00 00 00 Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_attach_ok:667[1]: Unable to read config from ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_open_by_guids:766[1]: Searching by guids [376274468418206170:15764092042190275056]. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada2p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:268[1]: Found consumer for ada2p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:296[1]: Used existing consumer for ada2p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada2p1... Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada2p1. Jul 7 16:40:32 x8dtu kernel: vdev_attach_ok:703[1]: vdev guid mismatch for provider ada2p1: 15764092042190275056 != 15252581311252777810. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada3p1... Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 0a 00 00 40 00 00 00 00 00 00 Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 4a 00 00 40 00 00 00 00 00 00 Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 8a ff bb 40 48 00 00 00 00 00 Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c ca ff bb 40 48 00 00 00 00 00 Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:32 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_attach_ok:667[1]: Unable to read config from ada3p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada1p3. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:268[1]: Found consumer for ada1p3. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:296[1]: Used existing consumer for ada1p3. Jul 7 16:40:32 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada1p3... Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada1p3. Jul 7 16:40:32 x8dtu kernel: vdev_attach_ok:686[1]: pool guid mismatch for provider ada1p3: 376274468418206170 != 18320603570228782289. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada1p2. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:284[1]: vdev_geom_attach(284): g_access failed: 1 Jul 7 16:40:32 x8dtu kernel: Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada1p2. Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada1p2. Jul 7 16:40:32 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada1p2. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada1p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:230[1]: Failing attach of ada1p1. Incompatible mediasize 524288 Jul 7 16:40:32 x8dtu kernel: Jul 7 16:40:32 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada1p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada0p3. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:268[1]: Found consumer for ada0p3. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:296[1]: Used existing consumer for ada0p3. Jul 7 16:40:32 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada0p3... Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada0p3. Jul 7 16:40:32 x8dtu kernel: vdev_attach_ok:686[1]: pool guid mismatch for provider ada0p3: 376274468418206170 != 18320603570228782289. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada0p2. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:284[1]: vdev_geom_attach(284): g_access failed: 1 Jul 7 16:40:32 x8dtu kernel: Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada0p2. Jul 7 16:40:32 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada0p2. Jul 7 16:40:32 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada0p2. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada0p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:230[1]: Failing attach of ada0p1. Incompatible mediasize 524288 Jul 7 16:40:32 x8dtu kernel: Jul 7 16:40:32 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada0p1. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to gpt/gptboot0. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:230[1]: Failing attach of gpt/gptboot0. Incompatible mediasize 524288 Jul 7 16:40:32 x8dtu kernel: Jul 7 16:40:32 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to gpt/gptboot0. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to iso9660/11_0_RELEASE_P1_AMD64_CD. Jul 7 16:40:32 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for iso9660/11_0_RELEASE_P1_AMD64_CD. Jul 7 16:40:32 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from iso9660/11_0_RELEASE_P1_AMD64_CD... Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from iso9660/11_0_RELEASE_P1_AMD64_CD. Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for iso9660/11_0_RELEASE_P1_AMD64_CD. Jul 7 16:40:35 x8dtu kernel: vdev_attach_ok:667[1]: Unable to read config from iso9660/11_0_RELEASE_P1_AMD64_CD. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to gpt/gptboot1. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:230[1]: Failing attach of gpt/gptboot1. Incompatible mediasize 524288 Jul 7 16:40:35 x8dtu kernel: Jul 7 16:40:35 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to gpt/gptboot1. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to cd0. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for cd0. Jul 7 16:40:35 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from cd0... Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from cd0. Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for cd0. Jul 7 16:40:35 x8dtu kernel: vdev_attach_ok:667[1]: Unable to read config from cd0. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada2. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:284[1]: vdev_geom_attach(284): g_access failed: 1 Jul 7 16:40:35 x8dtu kernel: Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada2. Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada2. Jul 7 16:40:35 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada2. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada3. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for ada3. Jul 7 16:40:35 x8dtu kernel: vdev_geom_read_config:457[1]: Reading config from ada3... Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 04 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 44 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c 84 1a c6 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 1c c4 1a c6 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada3. Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada3. Jul 7 16:40:35 x8dtu kernel: vdev_attach_ok:667[1]: Unable to read config from ada3. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada1. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:284[1]: vdev_geom_attach(284): g_access failed: 1 Jul 7 16:40:35 x8dtu kernel: Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada1. Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada1. Jul 7 16:40:35 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada1. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada0. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:284[1]: vdev_geom_attach(284): g_access failed: 1 Jul 7 16:40:35 x8dtu kernel: Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada0. Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada0. Jul 7 16:40:35 x8dtu kernel: vdev_attach_ok:659[1]: Unable to attach tasting instance to ada0. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:219[1]: Attaching to ada3p1. Jul 7 16:40:35 x8dtu kernel: vdev_geom_attach:288[1]: Created consumer for ada3p1. Jul 7 16:40:35 x8dtu kernel: vdev_geom_open_by_guids:778[1]: Attach by guid [376274468418206170:15764092042190275056] succeeded, provider ada3p1. Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 48 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 88 ff bb 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 c8 ff bb 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: vdev_geom_close_locked:352[1]: Closing access to ada3p1. Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:315[1]: Detaching from ada3p1. Jul 7 16:40:35 x8dtu kernel: vdev_geom_detach:326[1]: Destroying consumer for ada3p1. Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 04 f7 1a c6 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 34 1b c6 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 34 1b c6 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 10 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 02 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 00 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 40 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 10 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 02 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 00 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 40 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 08 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 00 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 02 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 10 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 00 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 35 1b c6 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 04 c7 ff bb 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 04 00 bc 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu ZFS[2205]: vdev state changed, pool_guid=376274468418206170 vdev_guid=15764092042190275056 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 04 00 bc 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 07 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 06 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 16 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 08 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 06 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 46 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 16 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 08 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 06 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 02 46 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 0e 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 06 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 08 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 16 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 06 00 00 40 00 00 00 00 00 00 Jul 7 16:40:35 x8dtu ZFS[2209]: vdev I/O failure, zpool=main_tank path=/dev/ada3p1 offset=270336 size=8192 error=5 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 05 00 bc 40 48 00 00 00 00 00 Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): CAM status: Auto-Sense Retrieval Failed Jul 7 16:40:35 x8dtu kernel: (ada3:ahcich3:0:0:0): Error 5, Unretryable error Jul 7 16:40:35 x8dtu ZFS[2214]: vdev I/O failure, zpool=main_tank path=/dev/ada3p1 offset=4998267674624 size=8192 error=5 Jul 7 16:40:35 x8dtu ZFS[2218]: vdev I/O failure, zpool=main_tank path=/dev/ada3p1 offset=4998267936768 size=8192 error=5 Jul 7 16:40:35 x8dtu ZFS[2222]: vdev probe failure, zpool=main_tank path=/dev/ada3p1 Jul 7 16:40:35 x8dtu ZFS[2230]: vdev state changed, pool_guid=376274468418206170 vdev_guid=15764092042190275056
I suspect I should replace that drive.
[16:43 x8dtu dvl ~] % sudo smartctl -a /dev/ada3 smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.2-RELEASE-p1 amd64] (local build) Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org Read Device Identity failed: Input/output error If this is a USB connected device, look at the various --device=TYPE variants A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. [16:43 x8dtu dvl ~] %
