This post is mostly for my benefit, so I can keep track of my plan for adjusting space. It doesn’t contain much of educational value.
For what I actually did, see zfs: setting compression and adding new vdevs
One recent day, I powered off r730-01 and took inventory of the 16 drive bays.
A short overview:
[17:36 r730-01 dvl ~] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data01 5.81T 2.84T 2.97T - - 1% 48% 1.00x ONLINE - data02 1.73T 712G 1.04T - - 46% 40% 1.00x ONLINE - data03 7.25T 1.29T 5.96T - - 32% 17% 1.00x ONLINE - data04 1.81T 5.79M 1.81T - - 0% 0% 1.00x ONLINE - zroot 424G 13.7G 410G - - 13% 3% 1.00x ONLINE -
My ideas at present:
* dismantle data04 – it is empty and unused
* move data03 from SSDs (7.25TB – 17% full) to NVMe (3.62TB – 34% full)
* can the 1x 1TB NVMe be used for poudriere? It is 234G
Possible list of actions:
* Remove data04 – unused
* Remove those drives
* Replace slots 12-13 with new 4TB drives
* Use them to replace the drives already within data02
* Replace existing 1TB NVME drives (part of data04) with new 4TB NVMEs
* Use the new NVMe to enhance data03x
Slots 0-7
Slots 0-7 are used by data01:
[19:07 r730-01 dvl ~] % zpool list data01 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data01 5.81T 2.84T 2.97T - - 1% 48% 1.00x ONLINE - [19:09 r730-01 dvl ~] % zpool status data01 pool: data01 state: ONLINE scan: scrub repaired 0B in 00:14:54 with 0 errors on Thu Sep 18 04:30:26 2025 config: NAME STATE READ WRITE CKSUM data01 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/Y7P0A022TEVE ONLINE 0 0 0 gpt/Y7P0A02ATEVE ONLINE 0 0 0 gpt/Y7P0A02DTEVE ONLINE 0 0 0 gpt/Y7P0A02GTEVE ONLINE 0 0 0 gpt/Y7P0A02LTEVE ONLINE 0 0 0 gpt/Y7P0A02MTEVE ONLINE 0 0 0 gpt/Y7P0A02QTEVE ONLINE 0 0 0 gpt/Y7P0A033TEVE ONLINE 0 0 0 errors: No known data errors [19:09 r730-01 dvl ~] %
That’s 8x 745G SSDs, giving a total space of 5.8TB and using 3TB – We could move that onto 4x 4TB drives. With slightly less resilience.
Slots 8-11
This is data03
- Slot 8 is WD_230151801478, 4TB part of data03
- Slot 9 is WD_230151801284, 4TB part of data03
- Slot 10 is WD_22492H800867, 4TB part of data03
- Slot 11 is WD_230151800473, 4TB part of data03
This pool is used mainly for poudriere, git, subversion, librenms, and acme.sh and PostgreSQL.
[17:16 r730-01 dvl ~] % zpool list data03 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data03 7.25T 1.28T 5.97T - - 32% 17% 1.00x ONLINE - [17:16 r730-01 dvl ~] % zpool status data03 pool: data03 state: ONLINE scan: scrub repaired 0B in 00:50:25 with 0 errors on Thu Sep 18 05:06:22 2025 config: NAME STATE READ WRITE CKSUM data03 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/WD_22492H800867 ONLINE 0 0 0 gpt/WD_230151801284 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/WD_230151801478 ONLINE 0 0 0 gpt/WD_230151800473 ONLINE 0 0 0 errors: No known data errors
Slots 12-13
This is data04.
Slot 12 is S3Z8NB0KB11784L, da12, 2TB part of data04
Slot 13 is S3Z8NB0KB11776R, da13, 2TB part of data04
The other two devices in data04 are NVMe and are not in the front drive cages. They are in PCIe slots at the rear of the host.
This zpool is not in use. I was dubious about it from the start.
[17:17 r730-01 dvl ~] % zpool list data04 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data04 1.81T 5.79M 1.81T - - 0% 0% 1.00x ONLINE - [17:20 r730-01 dvl ~] % zpool status data04 pool: data04 state: ONLINE scan: scrub repaired 0B in 00:00:00 with 0 errors on Thu Sep 18 04:15:58 2025 config: NAME STATE READ WRITE CKSUM data04 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nda0p1 ONLINE 0 0 0 da12p1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 nda1p1 ONLINE 0 0 0 da13p1 ONLINE 0 0 0 errors: No known data errors [17:20 r730-01 dvl ~] %
Slot 14-15
This is data02
Slot 14 is serial number S2NKNCAH501524K, da14, incorrectly labelled as S2NKNCAH501536P and hence the device id is still incorrect, is part of data02
Slot 15 is S2NKNCAH501536P, da15, part of data02
[17:20 r730-01 dvl ~] % zpool list data02 NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data02 1.73T 712G 1.04T - - 46% 40% 1.00x ONLINE - [17:21 r730-01 dvl ~] % zpool status data02 pool: data02 state: ONLINE scan: scrub repaired 0B in 00:33:52 with 0 errors on Thu Sep 18 04:49:36 2025 config: NAME STATE READ WRITE CKSUM data02 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/BTHC536302CT400VGN ONLINE 0 0 0 gpt/S2NKNCAH501536P ONLINE 0 0 0 errors: No known data errors
zpools
This section is an overview of the system.
[19:37 r730-01 dvl ~] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data01 5.81T 2.84T 2.97T - - 1% 48% 1.00x ONLINE - data02 1.73T 711G 1.04T - - 46% 40% 1.00x ONLINE - data03 7.25T 1.28T 5.97T - - 32% 17% 1.00x ONLINE - data04 1.81T 5.79M 1.81T - - 0% 0% 1.00x ONLINE - zroot 424G 13.6G 410G - - 13% 3% 1.00x ONLINE - [19:38 r730-01 dvl ~] % zpool status pool: data01 state: ONLINE scan: scrub repaired 0B in 00:14:54 with 0 errors on Thu Sep 18 04:30:26 2025 config: NAME STATE READ WRITE CKSUM data01 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/Y7P0A022TEVE ONLINE 0 0 0 gpt/Y7P0A02ATEVE ONLINE 0 0 0 gpt/Y7P0A02DTEVE ONLINE 0 0 0 gpt/Y7P0A02GTEVE ONLINE 0 0 0 gpt/Y7P0A02LTEVE ONLINE 0 0 0 gpt/Y7P0A02MTEVE ONLINE 0 0 0 gpt/Y7P0A02QTEVE ONLINE 0 0 0 gpt/Y7P0A033TEVE ONLINE 0 0 0 errors: No known data errors pool: data02 state: ONLINE scan: scrub repaired 0B in 00:33:52 with 0 errors on Thu Sep 18 04:49:36 2025 config: NAME STATE READ WRITE CKSUM data02 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/BTHC536302CT400VGN ONLINE 0 0 0 gpt/S2NKNCAH501536P ONLINE 0 0 0 errors: No known data errors pool: data03 state: ONLINE scan: scrub repaired 0B in 00:50:25 with 0 errors on Thu Sep 18 05:06:22 2025 config: NAME STATE READ WRITE CKSUM data03 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/WD_22492H800867 ONLINE 0 0 0 gpt/WD_230151801284 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/WD_230151801478 ONLINE 0 0 0 gpt/WD_230151800473 ONLINE 0 0 0 errors: No known data errors pool: data04 state: ONLINE scan: scrub repaired 0B in 00:00:00 with 0 errors on Thu Sep 18 04:15:58 2025 config: NAME STATE READ WRITE CKSUM data04 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nda0p1 ONLINE 0 0 0 da12p1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 nda1p1 ONLINE 0 0 0 da13p1 ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: scrub repaired 0B in 00:00:55 with 0 errors on Thu Sep 18 04:16:59 2025 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 errors: No known data errors
