Now that the new storage is in place, I thought to test with compression on and off. Perhaps also use both lz4 and zstd.
For reference, this is one of a series of posts regarding compression testing on zpools:
- Experimenting with compression off – you are here
- Experimenting with compression=lz4
- Experimenting with compression=zstd
- Compression results
My plan: create three zpools of two mirrored devices and test. There will be three posts: compression off, compression lz4, and compression zstd,
NOTE: Don’t make plans based on my results. This is a general purpose server and has other tasks. My results may be skewed by other workloads. I’m sure there are valid criticisms of my methods.
The devices
The devices are:
-
nda0 at nvme0 bus 0 scbus18 target 0 lun 1 nda0: <Samsung SSD 990 EVO Plus 4TB 2B2QKXG7 S7U8NJ0Y716854P> nda0: Serial Number S7U8NJ0Y716854P nda0: nvme version 2.0 nda0: 3815447MB (7814037168 512 byte sectors)
-
nda1 at nvme1 bus 0 scbus19 target 0 lun 1 nda1: <Samsung SSD 980 PRO with Heatsink 1TB 4B2QGXA7 S6WSNJ0T208743F> nda1: Serial Number S6WSNJ0T208743F nda1: nvme version 1.3 nda1: 953869MB (1953525168 512 byte sectors)
-
nda2 at nvme2 bus 0 scbus20 target 0 lun 1 nda2: <Samsung SSD 980 PRO with Heatsink 1TB 4B2QGXA7 S6WSNJ0T207774T> nda2: Serial Number S6WSNJ0T207774T nda2: nvme version 1.3 nda2: 953869MB (1953525168 512 byte sectors)
-
nda3 at nvme3 bus 0 scbus21 target 0 lun 1 nda3: <Samsung SSD 990 EVO Plus 4TB 2B2QKXG7 S7U8NJ0Y716801F> nda3: Serial Number S7U8NJ0Y716801F nda3: nvme version 2.0
-
da12 at mrsas0 bus 1 scbus1 target 12 lun 0 da12: <ATA Samsung SSD 870 3B6Q> Fixed Direct Access SPC-4 SCSI device da12: Serial Number S757NS0Y700758M da12: 150.000MB/s transfers da12: 3815447MB (7814037168 512 byte sectors) da12: quirks=0x8<4K>
-
da13 at mrsas0 bus 1 scbus1 target 13 lun 0 da13: <ATA Samsung SSD 870 3B6Q> Fixed Direct Access SPC-4 SCSI device da13: Serial Number S757NS0Y700760R da13: 150.000MB/s transfers da13: 3815447MB (7814037168 512 byte sectors) da13: quirks=0x8<4K>
Samsung SSD 870 – 4TB SSD mirror
I will use whole drive for this, then perhaps, if time and patience permits, with partitions. I usually use partitions.
[16:39 r730-01 dvl ~] % sudo zpool create Samsung-SSD-870-no-compression mirror /dev/da12 /dev/da13 [16:39 r730-01 dvl ~] % sudo zfs set compression=off Samsung-SSD-870-no-compression [16:40 r730-01 dvl ~] % zpool status Samsung-SSD-870-no-compression pool: Samsung-SSD-870-no-compression state: ONLINE config: NAME STATE READ WRITE CKSUM Samsung-SSD-870-no-compression ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 errors: No known data errors [16:40 r730-01 dvl ~] % zpool get all Samsung-SSD-870-no-compression NAME PROPERTY VALUE SOURCE Samsung-SSD-870-no-compression size 3.62T - Samsung-SSD-870-no-compression capacity 0% - Samsung-SSD-870-no-compression altroot - default Samsung-SSD-870-no-compression health ONLINE - Samsung-SSD-870-no-compression guid 9713759034737043729 - Samsung-SSD-870-no-compression version - default Samsung-SSD-870-no-compression bootfs - default Samsung-SSD-870-no-compression delegation on default Samsung-SSD-870-no-compression autoreplace off default Samsung-SSD-870-no-compression cachefile - default Samsung-SSD-870-no-compression failmode wait default Samsung-SSD-870-no-compression listsnapshots off default Samsung-SSD-870-no-compression autoexpand off default Samsung-SSD-870-no-compression dedupratio 1.00x - Samsung-SSD-870-no-compression free 3.62T - Samsung-SSD-870-no-compression allocated 528K - Samsung-SSD-870-no-compression readonly off - Samsung-SSD-870-no-compression ashift 0 default Samsung-SSD-870-no-compression comment - default Samsung-SSD-870-no-compression expandsize - - Samsung-SSD-870-no-compression freeing 0 - Samsung-SSD-870-no-compression fragmentation 0% - Samsung-SSD-870-no-compression leaked 0 - Samsung-SSD-870-no-compression multihost off default Samsung-SSD-870-no-compression checkpoint - - Samsung-SSD-870-no-compression load_guid 186162523614580301 - Samsung-SSD-870-no-compression autotrim off default Samsung-SSD-870-no-compression compatibility off default Samsung-SSD-870-no-compression bcloneused 0 - Samsung-SSD-870-no-compression bclonesaved 0 - Samsung-SSD-870-no-compression bcloneratio 1.00x - Samsung-SSD-870-no-compression feature@async_destroy enabled local Samsung-SSD-870-no-compression feature@empty_bpobj enabled local Samsung-SSD-870-no-compression feature@lz4_compress active local Samsung-SSD-870-no-compression feature@multi_vdev_crash_dump enabled local Samsung-SSD-870-no-compression feature@spacemap_histogram active local Samsung-SSD-870-no-compression feature@enabled_txg active local Samsung-SSD-870-no-compression feature@hole_birth active local Samsung-SSD-870-no-compression feature@extensible_dataset active local Samsung-SSD-870-no-compression feature@embedded_data active local Samsung-SSD-870-no-compression feature@bookmarks enabled local Samsung-SSD-870-no-compression feature@filesystem_limits enabled local Samsung-SSD-870-no-compression feature@large_blocks enabled local Samsung-SSD-870-no-compression feature@large_dnode enabled local Samsung-SSD-870-no-compression feature@sha512 enabled local Samsung-SSD-870-no-compression feature@skein enabled local Samsung-SSD-870-no-compression feature@edonr enabled local Samsung-SSD-870-no-compression feature@userobj_accounting active local Samsung-SSD-870-no-compression feature@encryption enabled local Samsung-SSD-870-no-compression feature@project_quota active local Samsung-SSD-870-no-compression feature@device_removal enabled local Samsung-SSD-870-no-compression feature@obsolete_counts enabled local Samsung-SSD-870-no-compression feature@zpool_checkpoint enabled local Samsung-SSD-870-no-compression feature@spacemap_v2 active local Samsung-SSD-870-no-compression feature@allocation_classes enabled local Samsung-SSD-870-no-compression feature@resilver_defer enabled local Samsung-SSD-870-no-compression feature@bookmark_v2 enabled local Samsung-SSD-870-no-compression feature@redaction_bookmarks enabled local Samsung-SSD-870-no-compression feature@redacted_datasets enabled local Samsung-SSD-870-no-compression feature@bookmark_written enabled local Samsung-SSD-870-no-compression feature@log_spacemap active local Samsung-SSD-870-no-compression feature@livelist enabled local Samsung-SSD-870-no-compression feature@device_rebuild enabled local Samsung-SSD-870-no-compression feature@zstd_compress enabled local Samsung-SSD-870-no-compression feature@draid enabled local Samsung-SSD-870-no-compression feature@zilsaxattr enabled local Samsung-SSD-870-no-compression feature@head_errlog active local Samsung-SSD-870-no-compression feature@blake3 enabled local Samsung-SSD-870-no-compression feature@block_cloning enabled local Samsung-SSD-870-no-compression feature@vdev_zaps_v2 active local
Samsung SSD 980 PRO – 1TB NVMe mirror
Here are the zpool create commands.
[16:29 r730-01 dvl ~] % zpool status Samsung-SSD-980-PRO-no-compression pool: Samsung-SSD-980-PRO-no-compression state: ONLINE config: NAME STATE READ WRITE CKSUM Samsung-SSD-980-PRO-no-compression ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nda1 ONLINE 0 0 0 nda2 ONLINE 0 0 0 errors: No known data errors [16:27 r730-01 dvl ~] % sudo zpool create Samsung-SSD-980-PRO-no-compression mirror /dev/nda1 /dev/nda2 [16:29 r730-01 dvl ~] % sudo zfs set compression=off Samsung-SSD-980-PRO-no-compression [16:29 r730-01 dvl ~] % zfs get all Samsung-SSD-980-PRO-no-compression NAME PROPERTY VALUE SOURCE Samsung-SSD-980-PRO-no-compression type filesystem - Samsung-SSD-980-PRO-no-compression creation Fri Oct 3 16:29 2025 - Samsung-SSD-980-PRO-no-compression used 432K - Samsung-SSD-980-PRO-no-compression available 899G - Samsung-SSD-980-PRO-no-compression referenced 96K - Samsung-SSD-980-PRO-no-compression compressratio 1.00x - Samsung-SSD-980-PRO-no-compression mounted yes - Samsung-SSD-980-PRO-no-compression quota none default Samsung-SSD-980-PRO-no-compression reservation none default Samsung-SSD-980-PRO-no-compression recordsize 128K default Samsung-SSD-980-PRO-no-compression mountpoint /Samsung-SSD-980-PRO-no-compression default Samsung-SSD-980-PRO-no-compression sharenfs off default Samsung-SSD-980-PRO-no-compression checksum on default Samsung-SSD-980-PRO-no-compression compression off local Samsung-SSD-980-PRO-no-compression atime on default Samsung-SSD-980-PRO-no-compression devices on default Samsung-SSD-980-PRO-no-compression exec on default Samsung-SSD-980-PRO-no-compression setuid on default Samsung-SSD-980-PRO-no-compression readonly off default Samsung-SSD-980-PRO-no-compression jailed off default Samsung-SSD-980-PRO-no-compression snapdir hidden default Samsung-SSD-980-PRO-no-compression aclmode discard default Samsung-SSD-980-PRO-no-compression aclinherit restricted default Samsung-SSD-980-PRO-no-compression createtxg 1 - Samsung-SSD-980-PRO-no-compression canmount on default Samsung-SSD-980-PRO-no-compression xattr on default Samsung-SSD-980-PRO-no-compression copies 1 default Samsung-SSD-980-PRO-no-compression version 5 - Samsung-SSD-980-PRO-no-compression utf8only off - Samsung-SSD-980-PRO-no-compression normalization none - Samsung-SSD-980-PRO-no-compression casesensitivity sensitive - Samsung-SSD-980-PRO-no-compression vscan off default Samsung-SSD-980-PRO-no-compression nbmand off default Samsung-SSD-980-PRO-no-compression sharesmb off default Samsung-SSD-980-PRO-no-compression refquota none default Samsung-SSD-980-PRO-no-compression refreservation none default Samsung-SSD-980-PRO-no-compression guid 5904310603945008324 - Samsung-SSD-980-PRO-no-compression primarycache all default Samsung-SSD-980-PRO-no-compression secondarycache all default Samsung-SSD-980-PRO-no-compression usedbysnapshots 0B - Samsung-SSD-980-PRO-no-compression usedbydataset 96K - Samsung-SSD-980-PRO-no-compression usedbychildren 336K - Samsung-SSD-980-PRO-no-compression usedbyrefreservation 0B - Samsung-SSD-980-PRO-no-compression logbias latency default Samsung-SSD-980-PRO-no-compression objsetid 54 - Samsung-SSD-980-PRO-no-compression dedup off default Samsung-SSD-980-PRO-no-compression mlslabel none default Samsung-SSD-980-PRO-no-compression sync standard default Samsung-SSD-980-PRO-no-compression dnodesize legacy default Samsung-SSD-980-PRO-no-compression refcompressratio 1.00x - Samsung-SSD-980-PRO-no-compression written 96K - Samsung-SSD-980-PRO-no-compression logicalused 148K - Samsung-SSD-980-PRO-no-compression logicalreferenced 42.5K - Samsung-SSD-980-PRO-no-compression volmode default default Samsung-SSD-980-PRO-no-compression filesystem_limit none default Samsung-SSD-980-PRO-no-compression snapshot_limit none default Samsung-SSD-980-PRO-no-compression filesystem_count none default Samsung-SSD-980-PRO-no-compression snapshot_count none default Samsung-SSD-980-PRO-no-compression snapdev hidden default Samsung-SSD-980-PRO-no-compression acltype nfsv4 default Samsung-SSD-980-PRO-no-compression context none default Samsung-SSD-980-PRO-no-compression fscontext none default Samsung-SSD-980-PRO-no-compression defcontext none default Samsung-SSD-980-PRO-no-compression rootcontext none default Samsung-SSD-980-PRO-no-compression relatime on default Samsung-SSD-980-PRO-no-compression redundant_metadata all default Samsung-SSD-980-PRO-no-compression overlay on default Samsung-SSD-980-PRO-no-compression encryption off default Samsung-SSD-980-PRO-no-compression keylocation none default Samsung-SSD-980-PRO-no-compression keyformat none default Samsung-SSD-980-PRO-no-compression pbkdf2iters 0 default Samsung-SSD-980-PRO-no-compression special_small_blocks 0 default Samsung-SSD-980-PRO-no-compression prefetch all default
Samsung SSD 990 EVO Plus – 4TB NVMe mirror
[16:33 r730-01 dvl ~] % sudo zpool create Samsung-SSD-990-EVO-no-compression mirror /dev/nda0 /dev/nda3 [16:33 r730-01 dvl ~] % sudo zfs set compression=off Samsung-SSD-990-EVO-no-compression [16:33 r730-01 dvl ~] % zpool status Samsung-SSD-990-EVO-no-compression pool: Samsung-SSD-990-EVO-no-compression state: ONLINE config: NAME STATE READ WRITE CKSUM Samsung-SSD-990-EVO-no-compression ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nda0 ONLINE 0 0 0 nda3 ONLINE 0 0 0 errors: No known data errors [16:33 r730-01 dvl ~] % zfs get all Samsung-SSD-990-EVO-no-compression NAME PROPERTY VALUE SOURCE Samsung-SSD-990-EVO-no-compression type filesystem - Samsung-SSD-990-EVO-no-compression creation Fri Oct 3 16:33 2025 - Samsung-SSD-990-EVO-no-compression used 444K - Samsung-SSD-990-EVO-no-compression available 3.51T - Samsung-SSD-990-EVO-no-compression referenced 96K - Samsung-SSD-990-EVO-no-compression compressratio 1.00x - Samsung-SSD-990-EVO-no-compression mounted yes - Samsung-SSD-990-EVO-no-compression quota none default Samsung-SSD-990-EVO-no-compression reservation none default Samsung-SSD-990-EVO-no-compression recordsize 128K default Samsung-SSD-990-EVO-no-compression mountpoint /Samsung-SSD-990-EVO-no-compression default Samsung-SSD-990-EVO-no-compression sharenfs off default Samsung-SSD-990-EVO-no-compression checksum on default Samsung-SSD-990-EVO-no-compression compression off local Samsung-SSD-990-EVO-no-compression atime on default Samsung-SSD-990-EVO-no-compression devices on default Samsung-SSD-990-EVO-no-compression exec on default Samsung-SSD-990-EVO-no-compression setuid on default Samsung-SSD-990-EVO-no-compression readonly off default Samsung-SSD-990-EVO-no-compression jailed off default Samsung-SSD-990-EVO-no-compression snapdir hidden default Samsung-SSD-990-EVO-no-compression aclmode discard default Samsung-SSD-990-EVO-no-compression aclinherit restricted default Samsung-SSD-990-EVO-no-compression createtxg 1 - Samsung-SSD-990-EVO-no-compression canmount on default Samsung-SSD-990-EVO-no-compression xattr on default Samsung-SSD-990-EVO-no-compression copies 1 default Samsung-SSD-990-EVO-no-compression version 5 - Samsung-SSD-990-EVO-no-compression utf8only off - Samsung-SSD-990-EVO-no-compression normalization none - Samsung-SSD-990-EVO-no-compression casesensitivity sensitive - Samsung-SSD-990-EVO-no-compression vscan off default Samsung-SSD-990-EVO-no-compression nbmand off default Samsung-SSD-990-EVO-no-compression sharesmb off default Samsung-SSD-990-EVO-no-compression refquota none default Samsung-SSD-990-EVO-no-compression refreservation none default Samsung-SSD-990-EVO-no-compression guid 9037015278289255896 - Samsung-SSD-990-EVO-no-compression primarycache all default Samsung-SSD-990-EVO-no-compression secondarycache all default Samsung-SSD-990-EVO-no-compression usedbysnapshots 0B - Samsung-SSD-990-EVO-no-compression usedbydataset 96K - Samsung-SSD-990-EVO-no-compression usedbychildren 348K - Samsung-SSD-990-EVO-no-compression usedbyrefreservation 0B - Samsung-SSD-990-EVO-no-compression logbias latency default Samsung-SSD-990-EVO-no-compression objsetid 54 - Samsung-SSD-990-EVO-no-compression dedup off default Samsung-SSD-990-EVO-no-compression mlslabel none default Samsung-SSD-990-EVO-no-compression sync standard default Samsung-SSD-990-EVO-no-compression dnodesize legacy default Samsung-SSD-990-EVO-no-compression refcompressratio 1.00x - Samsung-SSD-990-EVO-no-compression written 96K - Samsung-SSD-990-EVO-no-compression logicalused 152K - Samsung-SSD-990-EVO-no-compression logicalreferenced 42.5K - Samsung-SSD-990-EVO-no-compression volmode default default Samsung-SSD-990-EVO-no-compression filesystem_limit none default Samsung-SSD-990-EVO-no-compression snapshot_limit none default Samsung-SSD-990-EVO-no-compression filesystem_count none default Samsung-SSD-990-EVO-no-compression snapshot_count none default Samsung-SSD-990-EVO-no-compression snapdev hidden default Samsung-SSD-990-EVO-no-compression acltype nfsv4 default Samsung-SSD-990-EVO-no-compression context none default Samsung-SSD-990-EVO-no-compression fscontext none default Samsung-SSD-990-EVO-no-compression defcontext none default Samsung-SSD-990-EVO-no-compression rootcontext none default Samsung-SSD-990-EVO-no-compression relatime on default Samsung-SSD-990-EVO-no-compression redundant_metadata all default Samsung-SSD-990-EVO-no-compression overlay on default Samsung-SSD-990-EVO-no-compression encryption off default Samsung-SSD-990-EVO-no-compression keylocation none default Samsung-SSD-990-EVO-no-compression keyformat none default Samsung-SSD-990-EVO-no-compression pbkdf2iters 0 default Samsung-SSD-990-EVO-no-compression special_small_blocks 0 default Samsung-SSD-990-EVO-no-compression prefetch all default
Nagios warnings
It was now that I started getting Nagios warnings.
UNKNOWN: The pool 'Samsung-SSD-980-PRO-no-compression' has never had a scrub. UNKNOWN: The pool 'Samsung-SSD-990-EVO-no-compression' has never had a scrub. OK: The last scrub on zpool 'data01' was performed on 2025-10-02.04:28:57. OK: The last scrub on zpool 'data02' was performed on 2025-10-02.04:47:09. OK: The last scrub on zpool 'data03' was performed on 2025-10-02.05:02:45. OK: The last scrub on zpool 'zroot' was performed on 2025-10-02.04:13:47.
No worries, that can be ignored or I can also just scrub…. Let’s scrub.
[16:40 r730-01 dvl ~] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT Samsung-SSD-870-no-compression 3.62T 432K 3.62T - - 0% 0% 1.00x ONLINE - Samsung-SSD-980-PRO-no-compression 928G 432K 928G - - 0% 0% 1.00x ONLINE - Samsung-SSD-990-EVO-no-compression 3.62T 444K 3.62T - - 0% 0% 1.00x ONLINE - data01 5.81T 2.84T 2.97T - - 1% 48% 1.00x ONLINE - data02 1.73T 714G 1.04T - - 47% 40% 1.00x ONLINE - data03 7.25T 1.28T 5.97T - - 32% 17% 1.00x ONLINE - zroot 424G 15.6G 408G - - 13% 3% 1.00x ONLINE - [16:40 r730-01 dvl ~] % sudo zpool scrub Samsung-SSD-870-no-compression Samsung-SSD-980-PRO-no-compression Samsung-SSD-990-EVO-no-compression [16:41 r730-01 dvl ~] % zpool status Samsung-SSD-870-no-compression Samsung-SSD-980-PRO-no-compression Samsung-SSD-990-EVO-no-compression pool: Samsung-SSD-870-no-compression state: ONLINE scan: scrub repaired 0B in 00:00:00 with 0 errors on Fri Oct 3 16:41:21 2025 config: NAME STATE READ WRITE CKSUM Samsung-SSD-870-no-compression ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 errors: No known data errors pool: Samsung-SSD-980-PRO-no-compression state: ONLINE scan: scrub repaired 0B in 00:00:00 with 0 errors on Fri Oct 3 16:41:21 2025 config: NAME STATE READ WRITE CKSUM Samsung-SSD-980-PRO-no-compression ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nda1 ONLINE 0 0 0 nda2 ONLINE 0 0 0 errors: No known data errors pool: Samsung-SSD-990-EVO-no-compression state: ONLINE scan: scrub repaired 0B in 00:00:00 with 0 errors on Fri Oct 3 16:41:21 2025 config: NAME STATE READ WRITE CKSUM Samsung-SSD-990-EVO-no-compression ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nda0 ONLINE 0 0 0 nda3 ONLINE 0 0 0 errors: No known data errors
Good, now onto some quick tests.
But first, some GEOM warnings
I saw these warnings in /var/log/messages:
kernel: GEOM: nda1: the primary GPT table is corrupt or invalid. kernel: GEOM: nda1: using the secondary instead -- recovery strongly advised. kernel: GEOM: nda2: the primary GPT table is corrupt or invalid. kernel: GEOM: nda2: using the secondary instead -- recovery strongly advised.
I think I know why. Those are the 1TB NVMe sticks which were once part of another zpool (see r730-01: the drive replacement plan – which is more personal notes than educational).
I’ll ignore that for now.
Let’s cache some data
In weak attempts to eliminate caching, it’s do this:
[17:24 r730-01 dvl ~/tmp] % mkdir DELETE-ME [17:25 r730-01 dvl ~/tmp/DELETE-ME] % time tar -xzf /var/db/mkjail/releases/amd64/14.3-RELEASE/base.txz tar -xzf /var/db/mkjail/releases/amd64/14.3-RELEASE/base.txz 11.09s user 1.46s system 99% cpu 12.563 total [17:25 r730-01 dvl ~/tmp/DELETE-ME] % cd .. [17:27 r730-01 dvl ~/tmp] % time cp -r DELETE-ME DELETE-ME-2 cp: DELETE-ME/var/db/etcupdate/current/usr/share/nls/POSIX: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/usr/share/nls/en_US.US_ASCII: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/etc/unbound: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/etc/os-release: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/etc/rmt: No such file or directory cp: DELETE-ME/sys: No such file or directory cp: DELETE-ME/etc/os-release: No such file or directory cp -r DELETE-ME DELETE-ME-2 0.12s user 5.03s system 98% cpu 5.216 total [17:27 r730-01 dvl ~/tmp] % ls -l DELETE-ME/var/db/etcupdate/current/usr/share/nls/POSIX lrwxr-xr-x 1 dvl dvl 1 2025.06.06 08:35 DELETE-ME/var/db/etcupdate/current/usr/share/nls/POSIX -> C
Those errors relate to symlinks. I’ll ignore that.
More. These are not the tests.
[17:30 r730-01 dvl ~/tmp] % time cp -r DELETE-ME-2 DELETE-ME-3 cp -r DELETE-ME-2 DELETE-ME-3 0.06s user 6.45s system 99% cpu 6.547 total [17:30 r730-01 dvl ~/tmp] % time cp -r DELETE-ME-2 DELETE-ME-4 cp -r DELETE-ME-2 DELETE-ME-4 0.13s user 6.37s system 90% cpu 7.194 total [17:30 r730-01 dvl ~/tmp] % time cp -r DELETE-ME-2 DELETE-ME-5 cp -r DELETE-ME-2 DELETE-ME-5 0.14s user 6.41s system 83% cpu 7.810 total [17:30 r730-01 dvl ~/tmp] % time cp -r DELETE-ME-2 DELETE-ME-6 cp -r DELETE-ME-2 DELETE-ME-6 0.10s user 6.59s system 81% cpu 8.174 total [17:31 r730-01 dvl ~/tmp] % time cp -r DELETE-ME-2 DELETE-ME-7 cp -r DELETE-ME-2 DELETE-ME-7 0.08s user 6.47s system 71% cpu 9.188 total [17:31 r730-01 dvl ~/tmp] % time cp -r DELETE-ME-2 DELETE-ME-8 cp -r DELETE-ME-2 DELETE-ME-8 0.12s user 6.57s system 66% cpu 10.101 total [17:31 r730-01 dvl ~/tmp] % time cp -r DELETE-ME-2 DELETE-ME-9 cp -r DELETE-ME-2 DELETE-ME-9 0.13s user 6.52s system 59% cpu 11.185 total
Here’s some interesting information, the directories do not all contain the same data. Here’s some output from sysutils/ncdu:
ncdu 1.22 ~ Use the arrow keys to navigate, press ? for help
--- /usr/home/dvl/tmp -----------------------
2.4 GiB [#################################] /DELETE-ME-2
2.4 GiB [################################ ] /DELETE-ME-9
2.4 GiB [################################ ] /DELETE-ME-7
2.4 GiB [################################ ] /DELETE-ME-8
2.4 GiB [################################ ] /DELETE-ME-3
2.4 GiB [################################ ] /DELETE-ME-5
2.4 GiB [################################ ] /DELETE-ME-4
2.4 GiB [################################ ] /DELETE-ME-6
463.1 MiB [###### ] /DELETE-ME
Interesting, but right now, I don’t care. There is a different file count:
[17:39 r730-01 dvl ~/tmp] % find DELETE-ME | wc -l 29929 [17:39 r730-01 dvl ~/tmp] % find DELETE-ME-2 | wc -l 33184 [17:39 r730-01 dvl ~/tmp] % find DELETE-ME-3 | wc -l 33184 [17:39 r730-01 dvl ~/tmp] % find DELETE-ME-4 | wc -l 33184
Onward…
Simple tests – Samsung-SSD-870-no-compression
These are the destinations:
[17:19 r730-01 dvl ~] % zfs list | grep Sam Samsung-SSD-870-no-compression 540K 3.51T 96K /Samsung-SSD-870-no-compression Samsung-SSD-980-PRO-no-compression 528K 899G 96K /Samsung-SSD-980-PRO-no-compression Samsung-SSD-990-EVO-no-compression 552K 3.51T 96K /Samsung-SSD-990-EVO-no-compression
Allow me to write there:
[17:31 r730-01 dvl ~/tmp] % sudo chown dvl:dvl /Samsung-SSD-*
First test is 2:35 seconds, for
[17:34 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/ cp: DELETE-ME/var/db/etcupdate/current/usr/share/nls/POSIX: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/usr/share/nls/en_US.US_ASCII: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/etc/unbound: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/etc/os-release: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/etc/rmt: No such file or directory cp: DELETE-ME/sys: No such file or directory cp: DELETE-ME/etc/os-release: No such file or directory sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/ 0.00s user 0.01s system 0% cpu 2:34.49 total
Second test:
[17:42 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test2 cp: DELETE-ME/var/db/etcupdate/current/usr/share/nls/POSIX: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/usr/share/nls/en_US.US_ASCII: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/etc/unbound: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/etc/os-release: No such file or directory cp: DELETE-ME/var/db/etcupdate/current/etc/rmt: No such file or directory cp: DELETE-ME/sys: No such file or directory cp: DELETE-ME/etc/os-release: No such file or directory sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test2 0.00s user 0.01s system 0% cpu 2:32.70 total
That’s pretty consistent.
It does not stay that way. I’m copy to redact the “No such file or directory” messages in the rest of the test output.
[17:45 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-870-no-compression/test3 [17:46 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test3 sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test3 0.01s user 0.00s system 0% cpu 2:30.22 total [17:49 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-870-no-compression/test4 [17:50 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test4 sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test4 0.01s user 0.00s system 0% cpu 3:52.85 total [17:54 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-870-no-compression/test5 [17:54 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test5 sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test5 0.00s user 0.01s system 0% cpu 4:46.31 total [17:59 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-870-no-compression/test6 [18:01 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test6 sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test6 0.01s user 0.00s system 0% cpu 4:16.23 total [18:08 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-870-no-compression/test7 [18:08 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test7 sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test7 0.00s user 0.01s system 0% cpu 4:15.96 total [18:12 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-870-no-compression/test8 [18:14 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test8 sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test8 0.00s user 0.01s system 0% cpu 4:15.46 total [18:19 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-870-no-compression/test9 [18:19 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test9 sudo cp -r DELETE-ME* /Samsung-SSD-870-no-compression/test9 0.00s user 0.01s system 0% cpu 4:49.01 total
That’s enough of that.
[18:24 r730-01 dvl ~/tmp] % zpool list Samsung-SSD-870-no-compression NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT Samsung-SSD-870-no-compression 3.62T 324G 3.31T - - 0% 8% 1.00x ONLINE -
Simple tests – Samsung-SSD-980-PRO-no-compression
I’ll do the same thing here.
[18:26 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-980-PRO-no-compression/test1 [18:27 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test1 sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test1 0.00s user 0.01s system 0% cpu 1:42.93 total [18:29 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-980-PRO-no-compression/test2 [18:30 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test2 sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test2 0.00s user 0.01s system 0% cpu 1:44.78 total [18:32 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-980-PRO-no-compression/test3 [18:32 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test3 sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test3 0.01s user 0.00s system 0% cpu 1:45.73 total [18:34 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-980-PRO-no-compression/test4 [18:34 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test4 sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test4 0.01s user 0.00s system 0% cpu 1:44.91 total [18:36 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-980-PRO-no-compression/test5 [18:49 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test5 sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test5 0.01s user 0.01s system 0% cpu 1:42.85 total [18:51 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-980-PRO-no-compression/test6 [18:51 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test6 sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test6 0.01s user 0.00s system 0% cpu 1:44.16 total [18:53 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-980-PRO-no-compression/test7 [18:53 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test7 sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test7 0.00s user 0.02s system 0% cpu 1:43.07 total [18:55 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-980-PRO-no-compression/test8 [18:55 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test8 sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test8 0.00s user 0.01s system 0% cpu 1:42.65 total [18:57 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-980-PRO-no-compression/test9 [18:57 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test9 sudo cp -r DELETE-ME* /Samsung-SSD-980-PRO-no-compression/test9 0.00s user 0.01s system 0% cpu 1:48.38 total [18:59 r730-01 dvl ~/tmp] % zpool list Samsung-SSD-980-PRO-no-compression NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT Samsung-SSD-980-PRO-no-compression 928G 324G 604G - - 0% 34% 1.00x ONLINE - [18:59 r730-01 dvl ~/tmp] %
There was no slow down.
Simple tests – Samsung-SSD-990-EVO-no-compression
And one more… i just noticed I’m sometimes using sudo when I don’t need to.
[19:02 r730-01 dvl ~/tmp] % sudo mkdir /Samsung-SSD-990-EVO-no-compression/test1 [19:02 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test1 sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test1 0.01s user 0.01s system 0% cpu 1:46.11 total [19:04 r730-01 dvl ~/tmp] % sudo mkdir /Samsung-SSD-990-EVO-no-compression/test2 [19:05 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test2 sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test2 0.01s user 0.00s system 0% cpu 1:44.20 total [19:07 r730-01 dvl ~/tmp] % sudo mkdir /Samsung-SSD-990-EVO-no-compression/test3 [19:07 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test3 sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test3 0.00s user 0.01s system 0% cpu 1:43.13 total [19:09 r730-01 dvl ~/tmp] % sudo mkdir /Samsung-SSD-990-EVO-no-compression/test4 [19:09 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test4 sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test4 0.01s user 0.01s system 0% cpu 1:43.85 total [19:11 r730-01 dvl ~/tmp] % sudo mkdir /Samsung-SSD-990-EVO-no-compression/test5 [19:11 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test5 sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test5 0.01s user 0.00s system 0% cpu 1:42.75 total [19:13 r730-01 dvl ~/tmp] % sudo mkdir /Samsung-SSD-990-EVO-no-compression/test6 [19:13 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test6 sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test6 0.01s user 0.01s system 0% cpu 1:42.37 total [19:15 r730-01 dvl ~/tmp] % sudo mkdir /Samsung-SSD-990-EVO-no-compression/test7 [19:15 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test7 sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test7 0.01s user 0.00s system 0% cpu 1:43.01 total [19:17 r730-01 dvl ~/tmp] % sudo mkdir /Samsung-SSD-990-EVO-no-compression/test8 [19:17 r730-01 dvl ~/tmp] % time sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test8 sudo cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test8 0.00s user 0.01s system 0% cpu 1:42.78 total [19:19 r730-01 dvl ~/tmp] % mkdir /Samsung-SSD-990-EVO-no-compression/test9 [19:19 r730-01 dvl ~/tmp] % time cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test9 cp -r DELETE-ME* /Samsung-SSD-990-EVO-no-compression/test9 1.13s user 61.61s system 60% cpu 1:43.12 total [19:02 r730-01 dvl ~] % zpool list Samsung-SSD-990-EVO-no-compression NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT Samsung-SSD-990-EVO-no-compression 3.62T 324G 3.31T - - 0% 8% 1.00x ONLINE -
Pretty consistent times.
Let’s try lz4 next.
Here’s the cleanup:
[19:35 r730-01 dvl ~] % sudo rm -rf /Samsung-SSD-*/* [19:40 r730-01 dvl ~] % [19:42 r730-01 dvl ~] % zpool list Samsung-SSD-870-no-compression Samsung-SSD-980-PRO-no-compression Samsung-SSD-990-EVO-no-compression NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT Samsung-SSD-870-no-compression 3.62T 4.03M 3.62T - - 0% 0% 1.00x ONLINE - Samsung-SSD-980-PRO-no-compression 928G 4.59M 928G - - 0% 0% 1.00x ONLINE - Samsung-SSD-990-EVO-no-compression 3.62T 3.55M 3.62T - - 0% 0% 1.00x ONLINE -
Next, lz4 compression.
