I noticed these messages this morning, on the new server:
$ grep smartd /var/log/messages Jul 25 11:35:29 varm smartd[813]: Device: /dev/da15 [SAT], 8 Currently unreadable (pending) sectors Jul 25 11:35:29 varm smartd[813]: Device: /dev/da15 [SAT], 10 Offline uncorrectable sectors Jul 25 12:05:28 varm smartd[813]: Device: /dev/da15 [SAT], 8 Currently unreadable (pending) sectors Jul 25 12:35:29 varm smartd[813]: Device: /dev/da15 [SAT], 8 Currently unreadable (pending) sectors Jul 25 13:05:29 varm smartd[813]: Device: /dev/da15 [SAT], 8 Currently unreadable (pending) sectors Jul 25 13:35:28 varm smartd[813]: Device: /dev/da15 [SAT], 8 Currently unreadable (pending) sectors Jul 25 14:05:29 varm smartd[813]: Device: /dev/da15 [SAT], 8 Currently unreadable (pending) sectors Jul 25 14:35:29 varm smartd[813]: Device: /dev/da15 [SAT], 8 Currently unreadable (pending) sectors
The original smartctl output
Let’s see what smartctl says:
$ sudo smartctl -a /dev/da15
smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.2-BETA2-p1 amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" DT01ACA... Desktop HDD
Device Model: TOSHIBA DT01ACA300
Serial Number: 255BTBPGS
LU WWN Device Id: 5 000039 ff4efa21a
Firmware Version: MX6OABB0
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jul 25 14:35:59 2015 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (22795) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 380) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 139 139 054 Pre-fail Offline - 70
3 Spin_Up_Time 0x0007 136 136 024 Pre-fail Always - 418 (Average 425)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 40
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 124 124 020 Pre-fail Offline - 33
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 96
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 40
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 41
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 41
194 Temperature_Celsius 0x0002 171 171 000 Old_age Always - 35 (Min/Max 25/37)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 8
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
zpool status
The status of the ZFS pools are fine:
$ zpool status pool: benchmarking state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM benchmarking ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 gpt/disk_Z2T2UJJAS ONLINE 0 0 0 gpt/disk_13Q8U6GYS ONLINE 0 0 0 gpt/disk_256BWVLGS ONLINE 0 0 0 gpt/disk_256BY66GS ONLINE 0 0 0 gpt/disk_255BV69GS ONLINE 0 0 0 gpt/disk_255BS4NGS ONLINE 0 0 0 gpt/disk_255BUT1GS ONLINE 0 0 0 gpt/disk_255BTBPGS ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/zfs0 ONLINE 0 0 0 gpt/zfs1 ONLINE 0 0 0 errors: No known data errors
zpool scrub
Let’s try a scrub, just because:
[dan@varm:~] $ sudo zpool scrub benchmarking [dan@varm:~] $
When I looked at zpool status again, I saw this at the top of the output:
$ zpool status
pool: benchmarking
state: ONLINE
scan: scrub in progress since Sat Jul 25 14:42:10 2015
87.6G scanned out of 1.71T at 915M/s, 0h31m to go
0 repaired, 4.99% done
...
Twenty minutes later, no ZFS scrub errors found:
$ zpool status benchmarking pool: benchmarking state: ONLINE scan: scrub repaired 0 in 0h34m with 0 errors on Sat Jul 25 15:16:19 2015
The first long test
Let’s try a long test:
$ sudo smartctl -t long /dev/da15 smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.2-BETA2-p1 amd64] (local build) Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION === Sending command: "Execute SMART Extended self-test routine immediately in off-line mode". Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful. Testing has begun. Please wait 380 minutes for test to complete. Test will complete after Sat Jul 25 21:38:04 2015 Use smartctl -X to abort test.
First long test failure
But it seems the test was aborted:
$ sudo smartctl -a /dev/da15
smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.2-BETA2-p1 amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" DT01ACA... Desktop HDD
Device Model: TOSHIBA DT01ACA300
Serial Number: 255BTBPGS
LU WWN Device Id: 5 000039 ff4efa21a
Firmware Version: MX6OABB0
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jul 25 16:10:08 2015 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x85) Offline data collection activity
was aborted by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 121) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: (22795) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 380) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 139 139 054 Pre-fail Offline - 70
3 Spin_Up_Time 0x0007 136 136 024 Pre-fail Always - 418 (Average 425)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 40
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 124 124 020 Pre-fail Offline - 33
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 98
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 40
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 42
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 42
194 Temperature_Celsius 0x0002 171 171 000 Old_age Always - 35 (Min/Max 25/37)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 8
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 97 626066496
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
In the meantime, I went to Toshiba’s webpage to check warranty details. Every drive I checked gave Warranty could not be determined for this serial number. Eventually, I figured out: the serial number supplied by smartctl is wrong. I wrote about this before. Sadly, Toshiba has yet to reply to my two-year-old tweet about this. Perhaps I need to write them a sternly worded letter.
After running another long and a short test, it still failed:
A short test and another long test
I ran a short test, and then another long test. Still no joy, as shown in this output
$ sudo smartctl -a /dev/da15
smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.2-BETA2-p1 amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" DT01ACA... Desktop HDD
Device Model: TOSHIBA DT01ACA300
Serial Number: 255BTBPGS
LU WWN Device Id: 5 000039 ff4efa21a
Firmware Version: MX6OABB0
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jul 25 18:06:04 2015 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x85) Offline data collection activity
was aborted by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 121) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: (22795) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 380) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 139 139 054 Pre-fail Offline - 70
3 Spin_Up_Time 0x0007 136 136 024 Pre-fail Always - 418 (Average 425)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 40
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 124 124 020 Pre-fail Offline - 33
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 100
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 40
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 42
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 42
194 Temperature_Celsius 0x0002 166 166 000 Old_age Always - 36 (Min/Max 25/37)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 8
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 99 626066496
# 2 Short offline Completed without error 00% 98 -
# 3 Extended offline Completed: read failure 90% 97 626066496
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
I’m leaving this for now. Hopefully I can find a better solution.
