Adding in new NVMe cards and sticks

Today I removed two existing NVMe cards, each with a 1TB stick on it. Those sticks were added to these cards:

A 1TB and a 4TB NVMe card on each PCIe adaptor.

If you want the same hardware, you can use these links. As an Amazon Associate I earn from qualifying purchases. These are all links to Amazon.

The two NVMe card is a 10Gtek Dual M.2 NVMe SSD Adapter Card – PCIe 3.0 x8 Slot (M-Key)
The 4TB NVMe card is a Samsung 990 EVO Plus SSD 4TB, PCIe Gen 4×4 | Gen 5×2 M.2
The 1TB NVMe card is a SAMSUNG 980 PRO SSD with Heatsink 1TB PCIe Gen 4 NVMe M.2
The bag of NVMe screws, which didn’t come with the top two items, is M2 SSD Screws Kit, PCIe NVMe M.2 SSD Mounting Screws (60pcs)

Today, I opened the chassis of r730-01 (that link needs to be updated now that there is new hardware in there; I’ll post that one soon and page will then have a link to the new page). I removed the two existing NVMe adaptors each with a single 1TB NVMe stick. These sticks were added to the 2x NVMe card and inserted back into the same slots (#2 and #3).

When the host rebooted, I went into the System BIOS (if you want the page sequence to get there, read that page) and adjusted those two slots for 4x4x4x4 bifurcation.

4x4x4x4 Bifurcation set for slots #2 and #3

I rebooted and found the new devices.

Not shown, I reinstalled the cards

Not shown here: I removed the adaptors, and reinstalled the NVMe cards. They were not parallel to the adaptors. It was less than ideal.

Of course, I wrote about it.

After the reboot

After the reboot, what drives do I have now?

[16:40 r730-01 dvl ~] % zpool list data04
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data04  1.81T  5.46M  1.81T        -         -     0%     0%  1.00x    ONLINE  -

[16:40 r730-01 dvl ~] % sysctl kern.disks     
kern.disks: da10 da9 da8 da11 da15 da14 da13 da12 da6 da7 da5 da4 da3 da2 da0 da1 cd0 ada1 ada0 nda3 nda2 nda1 nda0

Let’s look at the nda stuff.

[16:41 r730-01 dvl ~] % grep -E 'nda3|nda2' /var/run/dmesg.boot  
nda2 at nvme2 bus 0 scbus20 target 0 lun 1
nda2: 
nda2: Serial Number S6WSNJ0T207774T
nda2: nvme version 1.3
nda2: 953869MB (1953525168 512 byte sectors)
nda3 at nvme3 bus 0 scbus21 target 0 lun 1
nda3: 
nda3: Serial Number S7U8NJ0Y716801F
nda3: nvme version 2.0
nda3: 3815447MB (7814037168 512 byte sectors)

Well, that’s pretty. It shows the 4TB and 1TB right next to each other, as they physically are. I suppose this shouldn’t be a surprise, however it does please me.

[16:41 r730-01 dvl ~] % grep -E 'nda1|nda0' /var/run/dmesg.boot 
nda0 at nvme0 bus 0 scbus18 target 0 lun 1
nda0: 
nda0: Serial Number S7U8NJ0Y716854P
nda0: nvme version 2.0
nda0: 3815447MB (7814037168 512 byte sectors)
nda1 at nvme1 bus 0 scbus19 target 0 lun 1
nda1: 
nda1: Serial Number S6WSNJ0T208743F
nda1: nvme version 1.3
nda1: 953869MB (1953525168 512 byte sectors)

zpool destruction

I’m going to destroy zpool data04:

[19:15 r730-01 dvl ~] % zpool status data04
  pool: data04
 state: ONLINE
  scan: scrub repaired 0B in 00:00:01 with 0 errors on Thu Sep 25 03:43:37 2025
config:

	NAME        STATE     READ WRITE CKSUM
	data04      ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nda2p1  ONLINE       0     0     0
	    da12p1  ONLINE       0     0     0
	  mirror-1  ONLINE       0     0     0
	    nda1p1  ONLINE       0     0     0
	    da13p1  ONLINE       0     0     0

errors: No known data errors

From the devices above, I searched for more information:

[19:16 r730-01 dvl ~] % grep -E 'nda2|da12|nda1|da13' /var/run/dmesg.boot 
nda1 at nvme1 bus 0 scbus19 target 0 lun 1
nda1: <Samsung SSD 980 PRO with Heatsink 1TB 4B2QGXA7 S6WSNJ0T208743F>
nda1: Serial Number S6WSNJ0T208743F
nda1: nvme version 1.3
nda1: 953869MB (1953525168 512 byte sectors)
nda2 at nvme2 bus 0 scbus20 target 0 lun 1
nda2: <Samsung SSD 980 PRO with Heatsink 1TB 4B2QGXA7 S6WSNJ0T207774T>
nda2: Serial Number S6WSNJ0T207774T
nda2: nvme version 1.3
nda2: 953869MB (1953525168 512 byte sectors)
da12 at mrsas0 bus 1 scbus1 target 12 lun 0
da12: <ATA Samsung SSD 860 2B6Q> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number S3Z8NB0KB11784L
da12: 150.000MB/s transfers
da12: 953869MB (1953525168 512 byte sectors)
da12: quirks=0x8<4K>
da13 at mrsas0 bus 1 scbus1 target 13 lun 0
da13: <ATA Samsung SSD 860 2B6Q> Fixed Direct Access SPC-4 SCSI device
da13: Serial Number S3Z8NB0KB11776R
da13: 150.000MB/s transfers
da13: 953869MB (1953525168 512 byte sectors)
da13: quirks=0x8<4K>

Ignore the declared 150.000MB/s transfers declaration. It’s wrong.

As displayed, those are all 1TB devices.

The two Samsung SSDs are in slots 12 and 13 (e.g. mrsas0 bus 1 scbus1 target 12 lun 0) – that matches up with conclusion in the not-for-public-education but still good to have online

There. I did it.

[17:01 r730-01 dvl ~] % sudo zpool destroy data04
[17:02 r730-01 dvl ~] % zpool list
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data01  5.81T  2.84T  2.97T        -         -     1%    48%  1.00x    ONLINE  -
data02  1.73T   717G  1.03T        -         -    47%    40%  1.00x    ONLINE  -
data03  7.25T  1.28T  5.97T        -         -    32%    17%  1.00x    ONLINE  -
zroot    424G  15.6G   408G        -         -    13%     3%  1.00x    ONLINE  -

This is me replacing the 1TB drives with 4TB drives.

Oct  2 17:05:12 r730-01 kernel: mrsas0: System PD deleted target ID: 0xc
Oct  2 17:05:12 r730-01 kernel: da12 at mrsas0 bus 1 scbus1 target 12 lun 0
Oct  2 17:05:12 r730-01 kernel: da12:   s/n S3Z8NB0KB11784L detached
Oct  2 17:05:12 r730-01 kernel: (da12:mrsas0:1:12:0): Periph destroyed

Oct  2 17:06:05 r730-01 kernel: mrsas0: System PD created target ID: 0xc
Oct  2 17:06:05 r730-01 kernel: da12 at mrsas0 bus 1 scbus1 target 12 lun 0
Oct  2 17:06:05 r730-01 kernel: da12:  Fixed Direct Access SPC-4 SCSI device
Oct  2 17:06:05 r730-01 kernel: da12: Serial Number S757NS0Y700758M
Oct  2 17:06:05 r730-01 kernel: da12: 150.000MB/s transfers
Oct  2 17:06:05 r730-01 kernel: da12: 3815447MB (7814037168 512 byte sectors)
Oct  2 17:06:05 r730-01 kernel: da12: quirks=0x8<4K>

Oct  2 17:06:36 r730-01 kernel: mrsas0: System PD deleted target ID: 0xd
Oct  2 17:06:36 r730-01 kernel: da13 at mrsas0 bus 1 scbus1 target 13 lun 0
Oct  2 17:06:36 r730-01 kernel: da13:   s/n S3Z8NB0KB11776R detached
Oct  2 17:06:36 r730-01 kernel: (da13:mrsas0:1:13:0): Periph destroyed

Oct  2 17:07:17 r730-01 kernel: mrsas0: System PD created target ID: 0xd
Oct  2 17:07:17 r730-01 kernel: da13 at mrsas0 bus 1 scbus1 target 13 lun 0
Oct  2 17:07:17 r730-01 kernel: da13:  Fixed Direct Access SPC-4 SCSI device
Oct  2 17:07:17 r730-01 kernel: da13: Serial Number S757NS0Y700760R
Oct  2 17:07:17 r730-01 kernel: da13: 150.000MB/s transfers
Oct  2 17:07:17 r730-01 kernel: da13: 3815447MB (7814037168 512 byte sectors)
Oct  2 17:07:17 r730-01 kernel: da13: quirks=0x8<4K>

I am especially proud of having attached those SSDs to the drives slides in the correct position and orientation without having an example to go by. Well, I think I looked at some photos I took earlier in the week and guessed.

As of now. all new devices are in the server and I have 2x 1TB 2.5″ SSDs left over.

Next, zpools.