Feb 142016
 

This morning I discovered this in my logs:

Feb 14 04:56:39 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 b8 a8 f0 40 4b 00 00 00 e8 00
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): Retrying command
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 b8 a8 f0 40 4b 00 00 00 e8 00
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): Retrying command
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 b8 a8 f0 40 4b 00 00 00 e8 00
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): Retrying command
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 b8 a8 f0 40 4b 00 00 00 e8 00
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): Retrying command
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 b8 a8 f0 40 4b 00 00 00 e8 00
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00
Feb 14 04:56:54 knew kernel: (ada0:ata0:0:0:0): Error 5, Retries exhausted
Feb 14 05:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 05:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 06:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 06:52:49 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 07:22:49 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 07:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 08:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 08:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 09:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 09:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 10:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 10:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 11:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 11:52:47 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 12:22:47 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 12:52:47 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 13:22:47 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 13:28:16 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 18 a9 f0 40 4b 00 00 00 88 00
Feb 14 13:28:16 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Feb 14 13:28:16 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Feb 14 13:28:16 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00
Feb 14 13:28:16 knew kernel: (ada0:ata0:0:0:0): Retrying command
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 18 a9 f0 40 4b 00 00 00 88 00
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): Retrying command
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 18 a9 f0 40 4b 00 00 00 88 00
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): Retrying command
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 18 a9 f0 40 4b 00 00 00 88 00
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): Retrying command
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): READ_DMA48. ACB: 25 00 18 a9 f0 40 4b 00 00 00 88 00
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): RES: 51 40 58 a9 f0 4b 4b 00 00 48 00
Feb 14 13:28:32 knew kernel: (ada0:ata0:0:0:0): Error 5, Retries exhausted
Feb 14 13:52:49 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 14:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 14:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 15:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 15:52:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors
Feb 14 16:22:48 knew smartd[861]: Device: /dev/ada0, 8 Currently unreadable (pending) sectors

Let’s try a status:

$ sudo smartctl -a /dev/ada0
smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba 3.5" DT01ACA... Desktop HDD
Device Model:     TOSHIBA DT01ACA300
Serial Number:    Z2T4KGYAS
LU WWN Device Id: 5 000039 ff4c2131c
Firmware Version: MX6OABB0
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Feb 14 16:46:43 2016 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)	Offline data collection activity
					was suspended by an interrupting command from host.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(25234) seconds.
Offline data collection
capabilities: 			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 421) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   138   138   054    Pre-fail  Offline      -       76
  3 Spin_Up_Time            0x0007   147   147   024    Pre-fail  Always       -       422 (Average 359)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       127
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   124   124   020    Pre-fail  Offline      -       33
  9 Power_On_Hours          0x0012   097   097   000    Old_age   Always       -       22457
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       42
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       134
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       134
194 Temperature_Celsius     0x0002   176   176   000    Old_age   Always       -       34 (Min/Max 18/46)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       8
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       126076

SMART Error Log Version: 1
Warning: ATA error count 65535 inconsistent with error log pointer 3

ATA Error Count: 65535 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 65535 occurred at disk power-on lifetime: 22454 hours (935 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 48 58 a9 f0 0b  Error: UNC 72 sectors at LBA = 0x0bf0a958 = 200321368

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 88 18 a9 f0 40 00   2d+04:41:12.561  READ DMA EXT
  25 00 88 18 a9 f0 40 00   2d+04:41:08.684  READ DMA EXT
  25 00 88 18 a9 f0 40 00   2d+04:41:04.933  READ DMA EXT
  25 00 88 18 a9 f0 40 00   2d+04:41:01.082  READ DMA EXT
  25 00 88 18 a9 f0 40 00   2d+04:40:57.258  READ DMA EXT

Error 65534 occurred at disk power-on lifetime: 22454 hours (935 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 48 58 a9 f0 0b  Error: UNC 72 sectors at LBA = 0x0bf0a958 = 200321368

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 88 18 a9 f0 40 00   2d+04:41:08.684  READ DMA EXT
  25 00 88 18 a9 f0 40 00   2d+04:41:04.933  READ DMA EXT
  25 00 88 18 a9 f0 40 00   2d+04:41:01.082  READ DMA EXT
  25 00 88 18 a9 f0 40 00   2d+04:40:57.258  READ DMA EXT
  25 00 20 f8 a8 f0 40 00   2d+04:40:57.258  READ DMA EXT

Error 65533 occurred at disk power-on lifetime: 22454 hours (935 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 48 58 a9 f0 0b  Error: UNC 72 sectors at LBA = 0x0bf0a958 = 200321368

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 88 18 a9 f0 40 00   2d+04:41:04.933  READ DMA EXT
  25 00 88 18 a9 f0 40 00   2d+04:41:01.082  READ DMA EXT
  25 00 88 18 a9 f0 40 00   2d+04:40:57.258  READ DMA EXT
  25 00 20 f8 a8 f0 40 00   2d+04:40:57.258  READ DMA EXT
  25 00 20 d8 a8 f0 40 00   2d+04:40:57.258  READ DMA EXT

Error 65532 occurred at disk power-on lifetime: 22454 hours (935 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 48 58 a9 f0 0b  Error: UNC 72 sectors at LBA = 0x0bf0a958 = 200321368

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 88 18 a9 f0 40 00   2d+04:41:01.082  READ DMA EXT
  25 00 88 18 a9 f0 40 00   2d+04:40:57.258  READ DMA EXT
  25 00 20 f8 a8 f0 40 00   2d+04:40:57.258  READ DMA EXT
  25 00 20 d8 a8 f0 40 00   2d+04:40:57.258  READ DMA EXT
  25 00 20 b8 a8 f0 40 00   2d+04:40:57.257  READ DMA EXT

Error 65531 occurred at disk power-on lifetime: 22454 hours (935 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 48 58 a9 f0 0b  Error: UNC 72 sectors at LBA = 0x0bf0a958 = 200321368

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 88 18 a9 f0 40 00   2d+04:40:57.258  READ DMA EXT
  25 00 20 f8 a8 f0 40 00   2d+04:40:57.258  READ DMA EXT
  25 00 20 d8 a8 f0 40 00   2d+04:40:57.258  READ DMA EXT
  25 00 20 b8 a8 f0 40 00   2d+04:40:57.257  READ DMA EXT
  25 00 20 70 a8 f0 40 00   2d+04:40:57.257  READ DMA EXT

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     17877         -
# 2  Extended offline    Completed without error       00%     17707         -
# 3  Extended offline    Aborted by host               90%     17694         -
# 4  Extended offline    Interrupted (host reset)      20%     17651         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Let’s try a short test:

$ sudo smartctl -t short /dev/ada0
smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 1 minutes for test to complete.
Test will complete after Sun Feb 14 16:52:31 2016

Use smartctl -X to abort test.
$ 

I ran smartctl -a /dev/ada0 again and saw this:

Self-test execution status:      ( 249)	Self-test routine in progress...
					90% of test remaining.

Waiting a bit longer, and I saw:

Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.

Then I ran the long test:

$ sudo smartctl -t long /dev/ada0
smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 421 minutes for test to complete.
Test will complete after Sun Feb 14 23:59:08 2016

Use smartctl -X to abort test.

I think my next blog update is at least 7 hours away.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive