When the time comes to replace a drive, it is very nice to know which drives is missing. I created this drive map to help me figure out which drive disappeared.
I created this drive-bay map using a combination of:
- zpool status
- sesutil map
- lsblk
- camcontrol
- /var/run/dmesg.boot
I have not included /var/run/dmesg.boot here.
If you click on this image, you’ll see a larger version which is easier to read.
I went through sesutil output and mapped each device to the drive bay, including the serial number and glabel. Eventually, I found that Element 17 (lines 72-74 below) had nothing listed. This differs from Element 18 which contained no drive. The system was powered off, mostly so I could easily pull drives without interfering with the systems.
This is drive bay #16 in the diagram above. This drive has serial number 57NGK1ZGF57D.
After recovering the zpool, I looked back at an older snapshot on this system. The serial number is not found in the dmesg.boot output:
[dan@knew:/.zfs/snapshot/autosnap_2019-11-25_00:00:00_daily/var/run] $ grep 57NGK1ZGF57D dmesg.boot [dan@knew:/.zfs/snapshot/autosnap_2019-11-25_00:00:00_daily/var/run] $
But is is found in the most recent version of that file:
$ grep 57NGK1ZGF57D /var/run/dmesg.boot da18: Serial Number 57NGK1ZGF57D
zpool status
The problem is shows on line 34. The issue was: which drive is that? The problem wasn’t that a drive disappeared in a running system. The issue was it didn’t show up on boot. da18 wasn’t the drive in question.
[dan@knew:~] $ zpool status system pool: system state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-2Q scan: scrub repaired 0 in 0 days 16:23:38 with 0 errors on Tue Nov 19 20:47:29 2019 config: NAME STATE READ WRITE CKSUM system DEGRADED 0 0 0 raidz2-0 ONLINE 0 0 0 da3p3 ONLINE 0 0 0 da10p3 ONLINE 0 0 0 da9p3 ONLINE 0 0 0 da2p3 ONLINE 0 0 0 da13p3 ONLINE 0 0 0 da15p3 ONLINE 0 0 0 da11p3 ONLINE 0 0 0 da14p3 ONLINE 0 0 0 da8p3 ONLINE 0 0 0 da7p3 ONLINE 0 0 0 raidz2-1 DEGRADED 0 0 0 da5p1 ONLINE 0 0 0 da6p1 ONLINE 0 0 0 da18p1 ONLINE 0 0 0 da12p1 ONLINE 0 0 0 da4p1 ONLINE 0 0 0 da1p1 ONLINE 0 0 0 da17p1 ONLINE 0 0 0 da16p1 ONLINE 0 0 0 da0p1 ONLINE 0 0 0 1933292688604201684 UNAVAIL 0 0 0 was /dev/da18p1 errors: No known data errors pool: tank_fast01 state: ONLINE scan: scrub repaired 0 in 0 days 02:07:17 with 0 errors on Sun Nov 24 07:10:04 2019 config: NAME STATE READ WRITE CKSUM tank_fast01 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/S3Z8NB0KB11776R.Slot.11 ONLINE 0 0 0 gpt/S3Z8NB0KB11784L.Slot.05 ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: scrub repaired 0 in 0 days 00:00:40 with 0 errors on Sun Nov 24 05:03:29 2019 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 errors: No known data errors
sesutil map -u /dev/ses0
Element 1 is described as slot 01 which is the first drive at the bottom left of the front of the chassis. This drive cage is physically labelled as 0.
Element 2 is directly above that.
It is similarly interesting that da0 is bottom left, da1, is to it’s right, then da2, etc. But that’s not strictly true. da8 is above da9.
[dan@knew:~] $ sudo sesutil map -u /dev/ses0 ses0: Enclosure Name: LSI SAS2X36 0e12 Enclosure ID: 50030480014e78bf Element 0, Type: Array Device Slot Status: Unsupported (0x00 0x00 0x00 0x00) Description: Drive Slots Element 1, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 01 Device Names: da0,pass1 Element 2, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 02 Device Names: da4,pass5 Element 3, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 03 Device Names: da9,pass10 Element 4, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 04 Device Names: da8,pass9 Element 5, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 05 Device Names: da16,pass20 Element 6, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 06 Device Names: da19,pass23 Element 7, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 07 Device Names: da1,pass2 Element 8, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 08 Device Names: da5,pass6 Element 9, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 09 Device Names: da10,pass12 Element 10, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 10 Device Names: da13,pass15 Element 11, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 11 Device Names: da17,pass21 Element 12, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 12 Device Names: da20,pass24 Element 13, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 13 Device Names: da2,pass3 Element 14, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 14 Device Names: da6,pass7 Element 15, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 15 Device Names: da11,pass13 Element 16, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 16 Device Names: da14,pass17 Element 17, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 17 Element 18, Type: Array Device Slot Status: Not Installed (0x05 0x00 0x00 0x00) Description: Slot 18 Element 19, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 19 Device Names: da3,pass4 Element 20, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 20 Device Names: da7,pass8 Element 21, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 21 Device Names: da12,pass14 Element 22, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 22 Device Names: da15,pass19 Element 23, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 23 Device Names: da21,pass25 Element 24, Type: Array Device Slot Status: OK (0x01 0x00 0x00 0x00) Description: Slot 24 Device Names: da18,pass22 Element 25, Type: Temperature Sensors Status: Unsupported (0x00 0x00 0x00 0x00) Description: Temperature Sensors Extra status: - Temperature: -reserved- Element 26, Type: Temperature Sensors Status: OK (0x01 0x00 0x33 0x00) Description: Temperature Extra status: - Temperature: 31 C Element 27, Type: Cooling Status: Unsupported (0x00 0x00 0x00 0x00) Description: Fans Extra status: - Speed: 0 rpm Element 28, Type: Cooling Status: Not Installed (0x05 0x00 0x00 0x50) Description: Fan1 Extra status: - Speed: 0 rpm Element 29, Type: Cooling Status: Not Installed (0x05 0x00 0x00 0x50) Description: Fan2 Extra status: - Speed: 0 rpm Element 30, Type: Cooling Status: Not Installed (0x05 0x00 0x00 0x50) Description: Fan3 Extra status: - Speed: 0 rpm Element 31, Type: Cooling Status: Not Available (0x07 0x00 0x00 0x10) Description: JBOD_Fan1 Extra status: - Speed: 0 rpm Element 32, Type: Cooling Status: Not Available (0x07 0x00 0x00 0x10) Description: JBOD_Fan2 Extra status: - Speed: 0 rpm Element 33, Type: Voltage Sensor Status: Unsupported (0x00 0x00 0x00 0x00) Description: Voltage Sensors Extra status: - Voltage: 0.00 V Element 34, Type: Voltage Sensor Status: OK (0x01 0x00 0x01 0xf6) Description: 5V Extra status: - Voltage: 5.02 V Element 35, Type: Voltage Sensor Status: OK (0x01 0x00 0x04 0xb6) Description: 12V Extra status: - Voltage: 12.06 V Element 36, Type: Power Supply Status: Unsupported (0x00 0x00 0x00 0x00) Description: Power Supplies Element 37, Type: Power Supply Status: Not Available (0x47 0x80 0x00 0x20) Description: Power Supply 1 Extra status: - Predicted Failure Element 38, Type: Power Supply Status: Not Available (0x47 0x80 0x00 0x20) Description: Power Supply 2 Extra status: - Predicted Failure Element 39, Type: Enclosure Status: Unsupported (0x00 0x00 0x00 0x00) Description: Enclosure Element 40, Type: Enclosure Status: OK (0x01 0x00 0x00 0x00) Description: Enclosure Element 41, Type: SAS Expander Status: Unsupported (0x00 0x00 0x00 0x00) Description: SAS Expanders Element 42, Type: SAS Expander Status: OK (0x01 0x00 0x00 0x00) Description: Primary Expander Element 43, Type: SAS Connector Status: Unsupported (0x00 0x00 0x00 0x00) Description: SAS Connectors Element 44, Type: SAS Connector Status: OK (0x01 0x11 0xff 0x00) Description: Upstream Connector (Primary) Element 45, Type: SAS Connector Status: OK (0x01 0x11 0xff 0x00) Description: Downstream Connector 1 (Primary) Element 46, Type: SAS Connector Status: OK (0x01 0x11 0xff 0x00) Description: Downstream Connector 2 (Primary) Element 47, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 00 Element 48, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 01 Element 49, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 02 Element 50, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 03 Element 51, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 04 Element 52, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 05 Element 53, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 06 Element 54, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 07 Element 55, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 08 Element 56, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 09 Element 57, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 10 Element 58, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 11 Element 59, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 12 Element 60, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 13 Element 61, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 14 Element 62, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 15 Element 63, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 16 Element 64, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 17 Element 65, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 18 Element 66, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 19 Element 67, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 20 Element 68, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 21 Element 69, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 22 Element 70, Type: SAS Connector Status: OK (0x01 0x20 0x00 0x00) Description: Drive Connector 23 Element 71, Type: Communication Port Status: Unsupported (0x00 0x00 0x00 0x00) Description: Ethernet ports Element 72, Type: Communication Port Status: Unsupported (0x00 0x00 0x00 0x00) Description: Ethernet_port_1 Element 73, Type: Communication Port Status: Unsupported (0x00 0x00 0x00 0x00) Description: Ethernet_port_2 [dan@knew:~] $
lsblk
I used this information to grab the label for a given drive.
[dan@knew:~] $ lsblk DEVICE MAJ:MIN SIZE TYPE LABEL MOUNT ada0 0:115 224G GPT - - <FREE> -:- 1.0M - - - ada0p1 0:116 512K freebsd-boot gpt/BTJR5090037J240AGN.boot.internal - <FREE> -:- 512K - - - ada0p2 0:117 4.0G freebsd-swap - - ada0p3 0:118 20G freebsd-zfs - <ZFS> <FREE> -:- 200G - - - ada1 0:147 224G GPT - - <FREE> -:- 1.0M - - - ada1p1 0:151 512K freebsd-boot gpt/BTJR509003N7240AGN.boot.internal - <FREE> -:- 512K - - - ada1p2 0:153 4.0G freebsd-swap - - ada1p3 0:154 20G freebsd-zfs - <ZFS> <FREE> -:- 200G - - - da0 0:158 4.5T GPT - - <FREE> -:- 3.0K - - - da0p1 1:44 4.5T freebsd-zfs gpt/BTJR5090037J240AGN.boot.internal <ZFS> <FREE> -:- 752M - - - da1 0:251 4.5T GPT - - <FREE> -:- 3.0K - - - da1p1 1:46 4.5T freebsd-zfs gpt/BTJR509003N7240AGN.boot.internal <ZFS> <FREE> -:- 752M - - - da10 1:19 4.5T GPT - - <FREE> -:- 1.0M - - - da10p1 1:84 512K freebsd-boot - - <FREE> -:- 512K - - - da10p2 1:86 4.0G freebsd-swap - - da10p3 1:88 4.5T freebsd-zfs gpt/4728K24SF57D.Slot.08 <ZFS> <FREE> -:- 9.2M - - - da11 1:21 4.5T GPT - - <FREE> -:- 1.0M - - - da11p1 1:90 512K freebsd-boot - - <FREE> -:- 512K - - - da11p2 1:92 4.0G freebsd-swap - - da11p3 1:94 4.5T freebsd-zfs gpt/579HKDZYF57D.Slot.14 <ZFS> <FREE> -:- 9.2M - - - da12 1:23 4.5T GPT - - <FREE> -:- 3.0K - - - da12p1 1:96 4.5T freebsd-zfs gpt/653DK7WPFS9A.Slot.20 <ZFS> <FREE> -:- 752M - - - da13 1:25 4.5T GPT - - <FREE> -:- 1.0M - - - da13p1 1:98 512K freebsd-boot - - <FREE> -:- 512K - - - da13p2 1:100 4.0G freebsd-swap - - da13p3 1:102 4.5T freebsd-zfs gpt/5782KL6VF57D.Slot.09 <ZFS> <FREE> -:- 9.2M - - - da14 1:27 4.5T GPT - - <FREE> -:- 1.0M - - - da14p1 1:104 512K freebsd-boot - - <FREE> -:- 512K - - - da14p2 1:106 4.0G freebsd-swap - - da14p3 1:108 4.5T freebsd-zfs gpt/579HKDZXF57D.Slot.15 <ZFS> <FREE> -:- 9.2M - - - da15 1:29 4.5T GPT - - <FREE> -:- 1.0M - - - da15p1 1:110 512K freebsd-boot - - <FREE> -:- 512K - - - da15p2 1:112 4.0G freebsd-swap - - da15p3 1:114 4.5T freebsd-zfs gpt/6525K2DGFS9A.Slot.21 <ZFS> <FREE> -:- 9.2M - - - da16 1:31 4.5T GPT - - <FREE> -:- 3.0K - - - da16p1 1:116 4.5T freebsd-zfs gpt/6539K3OJFS9A.Slot.04 <ZFS> <FREE> -:- 752M - - - da17 1:33 4.5T GPT - - <FREE> -:- 3.0K - - - da17p1 1:118 4.5T freebsd-zfs gpt/653AK2MXFS9A.Slot.10 <ZFS> <FREE> -:- 752M - - - da18 1:35 4.5T GPT - - <FREE> -:- 3.0K - - - da18p1 1:120 4.5T freebsd-zfs gpt/653EK93PFS9A.Slot.23 <ZFS> <FREE> -:- 752M - - - da19 1:37 932G GPT - - da19p1 1:122 932G freebsd-zfs gpt/S3Z8NB0KB11784L.Slot.05 <ZFS> da2 1:3 4.5T GPT - - <FREE> -:- 1.0M - - - da2p1 1:48 512K freebsd-boot - - <FREE> -:- 512K - - - da2p2 1:50 4.0G freebsd-swap - - da2p3 1:52 4.5T freebsd-zfs gpt/37D4KBJPF57D.Slot.12 <ZFS> <FREE> -:- 9.2M - - - da20 1:39 932G GPT - - da20p1 1:124 932G freebsd-zfs gpt/S3Z8NB0KB11776R.Slot.11 <ZFS> da21 1:41 - - - - da3 1:5 4.5T GPT - - <FREE> -:- 1.0M - - - da3p1 1:54 512K freebsd-boot - - <FREE> -:- 512K - - - da3p2 1:56 4.0G freebsd-swap - - da3p3 1:58 4.5T freebsd-zfs gpt/X643KHBFF57D.Slot.18 <ZFS> <FREE> -:- 9.2M - - - da4 1:7 4.5T GPT - - <FREE> -:- 3.0K - - - da4p1 1:60 4.5T freebsd-zfs gpt/653DK7WCFS9A.Slot.01 <ZFS> <FREE> -:- 752M - - - da5 1:9 4.5T GPT - - <FREE> -:- 3.0K - - - da5p1 1:62 4.5T freebsd-zfs gpt/653BK12JFS9A.Slot.07 <ZFS> <FREE> -:- 752M - - - da6 1:11 4.5T GPT - - da6p1 1:64 4.5T freebsd-zfs gpt/579IK5RMF57D.Slot.10 <ZFS> <FREE> -:- 752M - - - da7 1:13 4.5T GPT - - <FREE> -:- 1.0M - - - da7p1 1:66 512K freebsd-boot - - <FREE> -:- 512K - - - da7p2 1:68 4.0G freebsd-swap - - da7p3 1:70 4.5T freebsd-zfs gpt/X6IEKELNF57D.Slot.19 <ZFS> <FREE> -:- 9.2M - - - da8 1:15 4.5T GPT - - <FREE> -:- 1.0M - - - da8p1 1:72 512K freebsd-boot - - <FREE> -:- 512K - - - da8p2 1:74 4.0G freebsd-swap - - da8p3 1:76 4.5T freebsd-zfs gpt/5782KL6MF57D.Slot.03 <ZFS> <FREE> -:- 9.2M - - - da9 1:17 4.5T GPT - - <FREE> -:- 1.0M - - - da9p1 1:78 512K freebsd-boot - - <FREE> -:- 512K - - - da9p2 1:80 4.0G freebsd-swap - - da9p3 1:82 4.5T freebsd-zfs gpt/37KVK1JRF57D.Slot.02 <ZFS> <FREE> -:- 9.2M - - - [dan@knew:~] $ sudo camcontrol devlist
sudo camcontrol devlist
This is include only for completeness.
[dan@knew:~] $ sudo camcontrol devlist <LSI SAS2X36 0e12> at scbus0 target 8 lun 0 (ses0,pass0) <ATA TOSHIBA MD04ACA5 FP2A> at scbus0 target 10 lun 0 (pass1,da0) <ATA TOSHIBA MD04ACA5 FP2A> at scbus0 target 11 lun 0 (pass2,da1) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 12 lun 0 (pass3,da2) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 13 lun 0 (pass4,da3) <ATA TOSHIBA MD04ACA5 FP2A> at scbus0 target 14 lun 0 (pass5,da4) <ATA TOSHIBA MD04ACA5 FP2A> at scbus0 target 15 lun 0 (pass6,da5) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 16 lun 0 (pass7,da6) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 17 lun 0 (pass8,da7) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 18 lun 0 (pass9,da8) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 19 lun 0 (pass10,da9) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 20 lun 0 (pass12,da10) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 21 lun 0 (pass13,da11) <ATA TOSHIBA MD04ACA5 FP2A> at scbus0 target 22 lun 0 (pass14,da12) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 23 lun 0 (pass15,da13) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 24 lun 0 (pass17,da14) <ATA TOSHIBA MD04ACA5 FP2A> at scbus0 target 25 lun 0 (pass19,da15) <ATA TOSHIBA MD04ACA5 FP2A> at scbus0 target 26 lun 0 (pass20,da16) <ATA TOSHIBA MD04ACA5 FP2A> at scbus0 target 27 lun 0 (pass21,da17) <ATA TOSHIBA MD04ACA5 FP2A> at scbus0 target 29 lun 0 (pass22,da18) <ATA Samsung SSD 860 2B6Q> at scbus0 target 30 lun 0 (pass23,da19) <ATA Samsung SSD 860 2B6Q> at scbus0 target 31 lun 0 (pass24,da20) <ATA TOSHIBA HDWE150 FP2A> at scbus0 target 32 lun 0 (pass25,da21) <AHCI SGPIO Enclosure 1.00 0001> at scbus5 target 0 lun 0 (ses1,pass26) <INTEL SSDSC2BP240G4 L2010420> at scbus6 target 0 lun 0 (pass27,ada0) <INTEL SSDSC2BP240G4 L2010420> at scbus8 target 0 lun 0 (pass28,ada1) <AHCI SGPIO Enclosure 1.00 0001> at scbus12 target 0 lun 0 (ses2,pass29) [dan@knew:~] $
