This is about FreeBSD 11.1 on my knew server. I recently replaced the 3TB drives with 5TB drives. Along the way, it didn’t boot, the swap was adjusted, I got concerned about power, and there was a brief CAM status: SCSI Status Error scare. Then today, the planned reboot, I started asking questions on Reddit and on Twitter.
The booting issue
BIOS cannot boot from drives which present only 4096-byte sectors. This is a BIOS limitation. It is not your OS or your hardware. UEFI does not have this issue.
Many modern drives present look like this:
$ sudo smartctl -i /dev/da7 smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-RELEASE-p20 amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: TOSHIBA HDWE150 Serial Number: X643KHBFF57D LU WWN Device Id: 5 000039 75b800701 Firmware Version: FP2A User Capacity: 5,000,981,078,016 bytes [5.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Aug 30 21:31:19 2017 UTC SMART support is: Available - device has SMART capability. SMART support is: Enabled
This drive, which is similar to most of my 5TB drive, physically has 4096 byte sectors, yet it also presents sectors as 512 bytes. This drive can be booted via BIOS.
Compare the above to this drive:
[dan@x8dtu:~] $ sudo smartctl -i /dev/ada0 smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.1-RELEASE-p1 amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: TOSHIBA MG04ACA500A Serial Number: 44E1K00HFK7A LU WWN Device Id: 5 000039 57b701041 Firmware Version: FP1A User Capacity: 5,000,981,078,016 bytes [5.00 TB] Sector Size: 4096 bytes logical/physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Wed Aug 30 21:32:58 2017 UTC SMART support is: Available - device has SMART capability. SMART support is: Enabled [dan@x8dtu:~] $
This drive is 4096 byte sectors only.
Such drives cannot be booted via BIOS. I welcome stories to the contrary.
Fortunately, I learned this the hard way. x8dtu is the new FreshPorts server, still in build-out.
On that server, booting will be done from the two SSD drives.
The first boot
I have a boring video for you, but I also can let you skip through to the end.
The solution, as correctly predicted by Marie Helene, BIOS was set to boot from the wrong drive. Once I selected the correct boot drive in the BIOS, it booted.
Here is the splash screen of the first boot off the new 5TB drives.
Always review your boot messages
I hit scroll-lock and scrolled back. I found the message from powerd++:
powerd++: (ENOFREQ) at least the first CPU core must support frequency updates /etc/rc: WARNING failed to start powerdxx
/etc/rc: WARNING failed to start powerdxx
I installed that package months ago. I never noticed it wasn’t working. Shame on me.
The zpools
Here you go, 20 x 5TB HDD in two zpools, powered off 3 SAS 2008 card:
[dan@knew:~] $ zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT system 45.2T 17.7T 27.6T - 7% 39% 1.00x ONLINE - tank_data 45.2T 35.0T 10.3T - 17% 77% 1.00x ONLINE - [dan@knew:~] $ zpool status pool: system state: ONLINE scan: resilvered 1.69T in 18h56m with 0 errors on Tue Aug 29 17:35:16 2017 config: NAME STATE READ WRITE CKSUM system ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da1p3 ONLINE 0 0 0 da7p3 ONLINE 0 0 0 da2p3 ONLINE 0 0 0 da3p3 ONLINE 0 0 0 da4p3 ONLINE 0 0 0 da0p3 ONLINE 0 0 0 da14p3 ONLINE 0 0 0 da5p3 ONLINE 0 0 0 da6p3 ONLINE 0 0 0 da15p3 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 ada1p1 ONLINE 0 0 0 ada0p1 ONLINE 0 0 0 errors: No known data errors pool: tank_data state: ONLINE scan: scrub repaired 0 in 12h51m with 0 errors on Fri Aug 25 16:23:34 2017 config: NAME STATE READ WRITE CKSUM tank_data ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 da9p1 ONLINE 0 0 0 da10p1 ONLINE 0 0 0 da8p1 ONLINE 0 0 0 da11p1 ONLINE 0 0 0 da12p1 ONLINE 0 0 0 da13p1 ONLINE 0 0 0 da16p1 ONLINE 0 0 0 da17p1 ONLINE 0 0 0 da18p1 ONLINE 0 0 0 da19p1 ONLINE 0 0 0 errors: No known data errors [dan@knew:~] $
