Aug 312017
 

This is about FreeBSD 11.1 on my knew server. I recently replaced the 3TB drives with 5TB drives. Along the way, it didn’t boot, the swap was adjusted, I got concerned about power, and there was a brief CAM status: SCSI Status Error scare. Then today, the planned reboot, I started asking questions on Reddit and on Twitter.

The booting issue

BIOS cannot boot from drives which present only 4096-byte sectors. This is a BIOS limitation. It is not your OS or your hardware. UEFI does not have this issue.

Many modern drives present look like this:

$ sudo smartctl -i /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-RELEASE-p20 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     TOSHIBA HDWE150
Serial Number:    X643KHBFF57D
LU WWN Device Id: 5 000039 75b800701
Firmware Version: FP2A
User Capacity:    5,000,981,078,016 bytes [5.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Aug 30 21:31:19 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

This drive, which is similar to most of my 5TB drive, physically has 4096 byte sectors, yet it also presents sectors as 512 bytes. This drive can be booted via BIOS.

Compare the above to this drive:

[dan@x8dtu:~] $ sudo smartctl -i /dev/ada0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.1-RELEASE-p1 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     TOSHIBA MG04ACA500A
Serial Number:    44E1K00HFK7A
LU WWN Device Id: 5 000039 57b701041
Firmware Version: FP1A
User Capacity:    5,000,981,078,016 bytes [5.00 TB]
Sector Size:      4096 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Aug 30 21:32:58 2017 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

[dan@x8dtu:~] $ 

This drive is 4096 byte sectors only.

Such drives cannot be booted via BIOS. I welcome stories to the contrary.

Fortunately, I learned this the hard way. x8dtu is the new FreshPorts server, still in build-out.

On that server, booting will be done from the two SSD drives.

The first boot

I have a boring video for you, but I also can let you skip through to the end.

The solution, as correctly predicted by Marie Helene, BIOS was set to boot from the wrong drive. Once I selected the correct boot drive in the BIOS, it booted.

BIOS boot drive selection

Selected the right boot drive. It was the Hard Disk:#0400 entry

Here is the splash screen of the first boot off the new 5TB drives.

splash screen

First boot after replacing the 3TB drives with 5TB drives in knew

Always review your boot messages

I hit scroll-lock and scrolled back. I found the message from powerd++:

powerd++: (ENOFREQ) at least the first CPU core must support frequency updates
/etc/rc: WARNING failed to start powerdxx
powerdxx not running

powerd++: (ENOFREQ) at least the first CPU core must support frequency updates
/etc/rc: WARNING failed to start powerdxx

I installed that package months ago. I never noticed it wasn’t working. Shame on me.

The zpools

Here you go, 20 x 5TB HDD in two zpools, powered off 3 SAS 2008 card:

[dan@knew:~] $ zpool list
NAME        SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
system     45.2T  17.7T  27.6T         -     7%    39%  1.00x  ONLINE  -
tank_data  45.2T  35.0T  10.3T         -    17%    77%  1.00x  ONLINE  -


[dan@knew:~] $ zpool status
  pool: system
 state: ONLINE
  scan: resilvered 1.69T in 18h56m with 0 errors on Tue Aug 29 17:35:16 2017
config:

	NAME        STATE     READ WRITE CKSUM
	system      ONLINE       0     0     0
	  raidz2-0  ONLINE       0     0     0
	    da1p3   ONLINE       0     0     0
	    da7p3   ONLINE       0     0     0
	    da2p3   ONLINE       0     0     0
	    da3p3   ONLINE       0     0     0
	    da4p3   ONLINE       0     0     0
	    da0p3   ONLINE       0     0     0
	    da14p3  ONLINE       0     0     0
	    da5p3   ONLINE       0     0     0
	    da6p3   ONLINE       0     0     0
	    da15p3  ONLINE       0     0     0
	logs
	  mirror-1  ONLINE       0     0     0
	    ada1p1  ONLINE       0     0     0
	    ada0p1  ONLINE       0     0     0

errors: No known data errors

  pool: tank_data
 state: ONLINE
  scan: scrub repaired 0 in 12h51m with 0 errors on Fri Aug 25 16:23:34 2017
config:

	NAME        STATE     READ WRITE CKSUM
	tank_data   ONLINE       0     0     0
	  raidz3-0  ONLINE       0     0     0
	    da9p1   ONLINE       0     0     0
	    da10p1  ONLINE       0     0     0
	    da8p1   ONLINE       0     0     0
	    da11p1  ONLINE       0     0     0
	    da12p1  ONLINE       0     0     0
	    da13p1  ONLINE       0     0     0
	    da16p1  ONLINE       0     0     0
	    da17p1  ONLINE       0     0     0
	    da18p1  ONLINE       0     0     0
	    da19p1  ONLINE       0     0     0

errors: No known data errors
[dan@knew:~] $ 
 
Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive