Oct 052019
 

I have obtained a Dell R720 containing 10 x 400GB SSDs. The drives are connected to a RAID controller (H710P) which cannot do JBOD / IT mode. This means the drives are effectively hidden from ZFS, which is never good.

In this post:

  • FreeBSD 12.0
  • ZFS
  • Dell R720
  • SAS 9207-8i

This post describes past work and future plans for this server as I get it ready to be a general purpose server running FreeBSD and providing multiple services, including:

  1. poudriere – for providing a custom package building server for FreeBSD
  2. PostgreSQL – database server for local applications
  3. Bacula – connected to a Dell TL4000 tape library for dumping backups to tape
  4. mqtt instance – Mosquitto instance for local message queuing
  5. Yeah, that’s a lot of stuff and varied workloads.

    Work so far

    Last night I upgraded the iDRAC, and live tweeted it. From racking (6:16 PM) to final firmware upgrade (11:23 PM) was about 5 hours. Some of that was opening and closing the case and waiting for upgrade to fail and then try again.

    Photos from last night

    During the racking and upgrading process, I took photos. There are often long periods of waiting. Much of this I could have done from the comfort of my desk, but instead, I was in the basement because the remote console did not work. It is work now, after the upgrade. Lovely timing.

    USB stick with FreeBSD 12.0 installer

    I burned this installer thumbdrive last night and plugged it into the back of the system.

    The Dell R720 and my main two switches

    The Dell is connected to ports 46 (main connection) and port 48 (iDRAC) on the bottom switch.
    On the top switch, it is connected to port 7. This photo was taken before that connection was made.

    Network card

    R720 network card
    I connected both fiber and a network cable to this system because I wasn’t sure that the fiber optics would work.
    Today I see that both are active.

    Front panel of Dell R720

    The front panel of the Dell R720 does look sharp.

    RAID and ZFS

    RAID cards should not be used with ZFS. Instead, you want an HBA. If you’re looking for an HBA for ZFS, the FreeNAS community is a leading source of information. They have a wealth of knowledge and consulting their mailing list / forum archives is a good place to start. Search for FreeNAS and the name of your controller. They’ll tell you.

    For example, mine is a Perc H710P (that is a PDF link). The search results tell me that is does not do JBOD, which is what you want for ZFS.

    Booting from the FreeBSD install thumbdrive

    I booted the system from the thumbdrive. Accessing the system via the remote console, I looked around via ifconfig and /var/run/dmesg.boot. I could see the NICs were up and working. The SAS controller was found. All looked good.

    I tried to bring up sshd, but that failed because the FreeBSD install drive creates a read-only filesystem. sshd cannot start because it cannot ready the keys it needs.

    This is where I moved to fmsBSD, something I have used many times in the past. With mfBSD, sshd can be easily started and you can login and look around. I find an ssh session to be easier to copy/paste stuff when looking around the system.

    Next Boot

    The Virtual Client Console, or remote console, which the Dell iDRAC provides, has an interesting and useful feature: Next Boot.

    Virtual Console Client screenshot

    Selecting Next Boot for the Dell R720

    The beauty of this is you can select BIOS or LifeCycle Controller. You don’t have to wait and press F2 or F11. You just select, and reboot, and come back at your leisure.

    I set that to be the USB thumbdrive and booted.

    No boot

    After the system finished booting, this is where I was stuck: /boot/config: -D and BIOS Drive C: is disk0

    screen shot of frozen boot

    This is where my boot stopped.

    Things to do

    These are the things I need to do before installing the OS.

    drive cages

    The system holds 16 2.5″ drives (either SAS or SATA). 10 drives are installed and there are 6 slots which need drive cages.

    SAS 9207-8i card

    The Dell R720 came with an H710P mini SAS card, which is not appropriate for ZFS. I found a SAS 9207-8i in my stash and installed it into one of the riser slots.

    SAS 9207-8i card in riser slot

    SAS 9207-8i card in riser slot

    I want to inspect what FreeBSD fins for this card, to ensure it is in IT mode, not IR mode. That’s why I’m trying to get this box booted from mfsBSD.

    back of the SAS 9207-8i

    back of the SAS 9207-8i

    As you can from this photo, I have not connected the cables to the new SAS card. The existing cables plug into a connector which differs from that on the SAS card. The existing connection is in the middle right (look for MB). The SAS connectors are at the top right. The right hand connector is just in front of LSI.

    onboard SAS and the SAS 9207-8i

    onboard SAS and the SAS 9207-8i

    2x SSDs for OS

    I want to use the 8 x 400GB drives only for data, not for the OS and not for SLOG.

    I prefer to boot from a mirrored pair of ZFS drives. I plan to use a pair of SSDS connected via PCIE slots, and avoid the HBA. This leaves all front-facing slots for data drives.

    2x SSDs for ZIL

    I want a separate ZIL. All ZFS uses a ZIL (ZFS Intent Log). By creating a separate ZIL, referred to as a SLOG (Separate ZFS Intent Log)

    I found The ZFS ZIL and SLOG Demystified to be a good read on this topic.

    With a SLOG, I can reduce the wear and tear on the man SSDs. Every synchronous update gets written twice:

    1. to the ZIL
    2. to the main storage

    For most situations, this isn’t an issue. I’m going to do it because I can.

    Combining those four drives

    I can combine the above mentioned four drives into two drives.

    The ZIL can use a relatively small partition on the drives. The OS can have some. The rest would be empty.

    For example, let’s say we get a pair of 500GB SSDs. The OS could have the first 20GB of that. The SLOG could have the next 10GB. The rest of the drive would be unused. This is referred to as over-provisioning. The theory is that the SSD wear levleing algorithm will spread the data over the entire flash memory.

    I have not decided if I’ll use two or four SSDs here.

    Install a new SSL certificate

    The existing SSL certificate on iDRAC needs to be replaced. It is expired and for a different host name. I have tried to create a CSR but the downloaded file is empty.

    fix the System Board Intrusion Chassis is open message

    This message always appears. I think it can be disabled.

    Current plan includes

    I added this section on 2019-10-06.

    After an IRC discussion on #FreeNAS, I saved some notes in a gist.

    Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive