This morning I discovered this in my logs:
Feb 14 04:56:39 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 b8 a8 f0 40 4b 00 00 00 e8 00 Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00 Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): Retrying command Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 b8 a8 f0 40 4b 00 00 00 e8 00 Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00 Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): Retrying command Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 b8 a8 f0 40 4b 00 00 00 e8 00 Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00 Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): Retrying command Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 b8 a8 f0 40 4b 00 00 00 e8 00 Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00 Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): Retrying command Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 b8 a8 f0 40 4b 00 00 00 e8 00 Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00 Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): Error 5, Retries exhausted Feb 14 05:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 05:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 06:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 06:52:49 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 07:22:49 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 07:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 08:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 08:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 09:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 09:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 10:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 10:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 11:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 11:52:47 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 12:22:47 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 12:52:47 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 13:22:47 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 13:28:16 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 18 a9 f0 40 4b 00 00 00 88 00 Feb 14 13:28:16 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error Feb 14 13:28:16 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Feb 14 13:28:16 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00 Feb 14 13:28:16 knew kernel: (ada0:ata0:0:0:0): Retrying command Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 18 a9 f0 40 4b 00 00 00 88 00 Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00 Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): Retrying command Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 18 a9 f0 40 4b 00 00 00 88 00 Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00 Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): Retrying command Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 18 a9 f0 40 4b 00 00 00 88 00 Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00 Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): Retrying command Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 18 a9 f0 40 4b 00 00 00 88 00 Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00 Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): Error 5, Retries exhausted Feb 14 13:52:49 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 14:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 14:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 15:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 15:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors Feb 14 16:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Let’s try a status:
$ sudo smartctl -a /dev/ada0
smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" DT01ACA... Desktop HDD
Device Model: TOSHIBA DT01ACA300
Serial Number: Z2T4KGYAS
LU WWN Device Id: 5 000039 ff4c2131c
Firmware Version: MX6OABB0
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sun Feb 14 16:46:43 2016 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (25234) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 421) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 138 138 054 Pre-fail Offline - 76
3 Spin_Up_Time 0x0007 147 147 024 Pre-fail Always - 422 (Average 359)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 127
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 124 124 020 Pre-fail Offline - 33
9 Power_On_Hours 0x0012 097 097 000 Old_age Always - 22457
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 42
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 134
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 134
194 Temperature_Celsius 0x0002 176 176 000 Old_age Always - 34 (Min/Max 18/46)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 8
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 126076
SMART Error Log Version: 1
Warning: ATA error count 65535 inconsistent with error log pointer 3
ATA Error Count: 65535 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 65535 occurred at disk power-on lifetime: 22454 hours (935 days + 14 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 48 58 a9 f0 0b Error: UNC 72 sectors at LBA = 0x0bf0a958 = 200321368
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 88 18 a9 f0 40 00 2d+04:41:12.561 READ DMA EXT
25 00 88 18 a9 f0 40 00 2d+04:41:08.684 READ DMA EXT
25 00 88 18 a9 f0 40 00 2d+04:41:04.933 READ DMA EXT
25 00 88 18 a9 f0 40 00 2d+04:41:01.082 READ DMA EXT
25 00 88 18 a9 f0 40 00 2d+04:40:57.258 READ DMA EXT
Error 65534 occurred at disk power-on lifetime: 22454 hours (935 days + 14 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 48 58 a9 f0 0b Error: UNC 72 sectors at LBA = 0x0bf0a958 = 200321368
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 88 18 a9 f0 40 00 2d+04:41:08.684 READ DMA EXT
25 00 88 18 a9 f0 40 00 2d+04:41:04.933 READ DMA EXT
25 00 88 18 a9 f0 40 00 2d+04:41:01.082 READ DMA EXT
25 00 88 18 a9 f0 40 00 2d+04:40:57.258 READ DMA EXT
25 00 20 f8 a8 f0 40 00 2d+04:40:57.258 READ DMA EXT
Error 65533 occurred at disk power-on lifetime: 22454 hours (935 days + 14 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 48 58 a9 f0 0b Error: UNC 72 sectors at LBA = 0x0bf0a958 = 200321368
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 88 18 a9 f0 40 00 2d+04:41:04.933 READ DMA EXT
25 00 88 18 a9 f0 40 00 2d+04:41:01.082 READ DMA EXT
25 00 88 18 a9 f0 40 00 2d+04:40:57.258 READ DMA EXT
25 00 20 f8 a8 f0 40 00 2d+04:40:57.258 READ DMA EXT
25 00 20 d8 a8 f0 40 00 2d+04:40:57.258 READ DMA EXT
Error 65532 occurred at disk power-on lifetime: 22454 hours (935 days + 14 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 48 58 a9 f0 0b Error: UNC 72 sectors at LBA = 0x0bf0a958 = 200321368
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 88 18 a9 f0 40 00 2d+04:41:01.082 READ DMA EXT
25 00 88 18 a9 f0 40 00 2d+04:40:57.258 READ DMA EXT
25 00 20 f8 a8 f0 40 00 2d+04:40:57.258 READ DMA EXT
25 00 20 d8 a8 f0 40 00 2d+04:40:57.258 READ DMA EXT
25 00 20 b8 a8 f0 40 00 2d+04:40:57.257 READ DMA EXT
Error 65531 occurred at disk power-on lifetime: 22454 hours (935 days + 14 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 48 58 a9 f0 0b Error: UNC 72 sectors at LBA = 0x0bf0a958 = 200321368
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 88 18 a9 f0 40 00 2d+04:40:57.258 READ DMA EXT
25 00 20 f8 a8 f0 40 00 2d+04:40:57.258 READ DMA EXT
25 00 20 d8 a8 f0 40 00 2d+04:40:57.258 READ DMA EXT
25 00 20 b8 a8 f0 40 00 2d+04:40:57.257 READ DMA EXT
25 00 20 70 a8 f0 40 00 2d+04:40:57.257 READ DMA EXT
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 17877 -
# 2 Extended offline Completed without error 00% 17707 -
# 3 Extended offline Aborted by host 90% 17694 -
# 4 Extended offline Interrupted (host reset) 20% 17651 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Let’s try a short test:
$ sudo smartctl -t short /dev/ada0 smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.2-RELEASE-p9 amd64] (local build) Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION === Sending command: "Execute SMART Short self-test routine immediately in off-line mode". Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful. Testing has begun. Please wait 1 minutes for test to complete. Test will complete after Sun Feb 14 16:52:31 2016 Use smartctl -X to abort test. $
I ran smartctl -a /dev/ada0 again and saw this:
Self-test execution status: ( 249) Self-test routine in progress... 90% of test remaining.
Waiting a bit longer, and I saw:
Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run.
Then I ran the long test:
$ sudo smartctl -t long /dev/ada0 smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.2-RELEASE-p9 amd64] (local build) Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION === Sending command: "Execute SMART Extended self-test routine immediately in off-line mode". Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful. Testing has begun. Please wait 421 minutes for test to complete. Test will complete after Sun Feb 14 23:59:08 2016 Use smartctl -X to abort test.
I think my next blog update is at least 7 hours away.
