Preparing the Dell R720 for ZFS

I have obtained a Dell R720 containing 10 x 400GB SSDs. The drives are connected to a RAID controller (H710P) which cannot do JBOD / IT mode. This means the drives are effectively hidden from ZFS, which is never good.

In this post:

• FreeBSD 12.0
• ZFS
• Dell R720
• SAS 9207-8i

This post describes past work and future plans for this server as I get it ready to be a general purpose server running FreeBSD and providing multiple services, including:

1. poudriere – for providing a custom package building server for FreeBSD
2. PostgreSQL – database server for local applications
3. Bacula – connected to a Dell TL4000 tape library for dumping backups to tape
4. mqtt instance – Mosquitto instance for local message queuing

Yeah, that’s a lot of stuff and varied workloads.

Work so far

Last night I upgraded the iDRAC, and live tweeted it. From racking (6:16 PM) to final firmware upgrade (11:23 PM) was about 5 hours. Some of that was opening and closing the case and waiting for upgrade to fail and then try again.

Photos from last night

During the racking and upgrading process, I took photos. There are often long periods of waiting. Much of this I could have done from the comfort of my desk, but instead, I was in the basement because the remote console did not work. It is work now, after the upgrade. Lovely timing.









RAID and ZFS

RAID cards should not be used with ZFS. Instead, you want an HBA. If you’re looking for an HBA for ZFS, the FreeNAS community is a leading source of information. They have a wealth of knowledge and consulting their mailing list / forum archives is a good place to start. Search for FreeNAS and the name of your controller. They’ll tell you.

For example, mine is a Perc H710P (that is a PDF link). The search results tell me that is does not do JBOD, which is what you want for ZFS.

Booting from the FreeBSD install thumbdrive

I booted the system from the thumbdrive. Accessing the system via the remote console, I looked around via ifconfig and /var/run/dmesg.boot. I could see the NICs were up and working. The SAS controller was found. All looked good.

I tried to bring up sshd, but that failed because the FreeBSD install drive creates a read-only filesystem. sshd cannot start because it cannot ready the keys it needs.

This is where I moved to fmsBSD, something I have used many times in the past. With mfBSD, sshd can be easily started and you can login and look around. I find an ssh session to be easier to copy/paste stuff when looking around the system.

Next Boot

The Virtual Client Console, or remote console, which the Dell iDRAC provides, has an interesting and useful feature: Next Boot.



The beauty of this is you can select BIOS or LifeCycle Controller. You don’t have to wait and press F2 or F11. You just select, and reboot, and come back at your leisure.

I set that to be the USB thumbdrive and booted.

No boot

After the system finished booting, this is where I was stuck: /boot/config: -D and BIOS Drive C: is disk0



Things to do

These are the things I need to do before installing the OS.

• drive cages
• 8087 cables – 2019-10-06 – Purchased Cancelled this: 2x Internal Mini SAS SFF-8087 Right Angle to Internal Mini SAS SFF-8087 also sold on eBay by the same seller. Instead, I went with brand name Dell cables: M7DP4 – from eBay
• PCIE drive cards – ordered 2x StarTech.com 2.5in SATA Removable Hard Drive Bay for PC Expansion Slot – Storage bay adapter – black – S25SLOTR
• 2x SSDs for OS – purchased 2x Intel 320 Series 80 GB SSDs
• 2x SSDs for ZIL
• Install a new SSL certificate
• fix the System Board Intrusion`Chassis is open message

drive cages

The system holds 16 2.5″ drives (either SAS or SATA). 10 drives are installed and there are 6 slots which need drive cages.

SAS 9207-8i card

The Dell R720 came with an H710P mini SAS card, which is not appropriate for ZFS. I found a SAS 9207-8i in my stash and installed it into one of the riser slots.



I want to inspect what FreeBSD fins for this card, to ensure it is in IT mode, not IR mode. That’s why I’m trying to get this box booted from mfsBSD.



As you can from this photo, I have not connected the cables to the new SAS card. The existing cables plug into a connector which differs from that on the SAS card. The existing connection is in the middle right (look for MB). The SAS connectors are at the top right. The right hand connector is just in front of LSI.



2x SSDs for OS

I want to use the 8 x 400GB drives only for data, not for the OS and not for SLOG.

I prefer to boot from a mirrored pair of ZFS drives. I plan to use a pair of SSDS connected via PCIE slots, and avoid the HBA. This leaves all front-facing slots for data drives.

2x SSDs for ZIL

I want a separate ZIL. All ZFS uses a ZIL (ZFS Intent Log). By creating a separate ZIL, referred to as a SLOG (Separate ZFS Intent Log)

I found The ZFS ZIL and SLOG Demystified to be a good read on this topic.

With a SLOG, I can reduce the wear and tear on the man SSDs. Every synchronous update gets written twice:

1. to the ZIL
2. to the main storage

For most situations, this isn’t an issue. I’m going to do it because I can.

Combining those four drives

I can combine the above mentioned four drives into two drives.

The ZIL can use a relatively small partition on the drives. The OS can have some. The rest would be empty.

For example, let’s say we get a pair of 500GB SSDs. The OS could have the first 20GB of that. The SLOG could have the next 10GB. The rest of the drive would be unused. This is referred to as over-provisioning. The theory is that the SSD wear levleing algorithm will spread the data over the entire flash memory.

I have not decided if I’ll use two or four SSDs here.

Install a new SSL certificate

The existing SSL certificate on iDRAC needs to be replaced. It is expired and for a different host name. I have tried to create a CSR but the downloaded file is empty.

fix the System Board Intrusion Chassis is open message

This message always appears. I think it can be disabled.

Current plan includes

I added this section on 2019-10-06.

• Using TMPFS with poudriere – see USE_TMPFS=all See also tmpfs(5) and tmpfs in fstab w/ zfs root
• 2 x optane for SLOG
• Mounting regular SSDs for the OS, mounted in PCI slots but connecting them to the M/B – e.g. StarTech S25SLOTR 2.5in SATA Removable Hard Drive Bay for PC Expansion Slot
• There are two SATA connectors on the M/B – see p129 on this manual: See p 129 on this manual https://topics-cdn.dell.com/pdf/poweredge-r720_owners-manual_en-us.pdf
• I recall seeing two SATA power connectors inside the case right behind the CD-ROM

After an IRC discussion on #FreeNAS, I saved some notes in a gist.