Two days ago, I was greeted with these message.
There are two recent issues:
- data04: DEGRADED – 102 days ago
- data04: DEGRADED – oh and have I mentioned WatchGoose? – 30 days ago
The first link mentions the problem device (S7KGNU0Y912955D) as seen below on the highlighted line 129.
Oct 10 05:00:46 r730-01 kernel: nvme2: Resetting controller due to a timeout. Oct 10 05:00:46 r730-01 kernel: nvme2: event="start" Oct 10 05:00:46 r730-01 kernel: nvme2: Waiting for reset to complete Oct 10 05:01:07 r730-01 kernel: nvme2: controller ready did not become 0 within 20500 ms Oct 10 05:01:07 r730-01 kernel: nvme2: event="timed_out" Oct 10 05:01:07 r730-01 kernel: nvme2: failing outstanding i/o Oct 10 05:01:07 r730-01 kernel: nvme2: WRITE (01) sqid:2 cid:123 nsid:1 lba:7747276704 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:2 cid:123 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:2 cid:109 nsid:1 lba:5774429032 len:8 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=cdc60fa0 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:2 cid:109 cdw0:0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: nvme2: failing outstanding i/o Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=582ecb68 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:3 cid:113 nsid:1 lba:6497288248 len:2048 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:3 cid:113 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:3 cid:123 nsid:1 lba:6497290296 len:2048 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:3 cid:123 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:3 cid:111 nsid:1 lba:6497292344 len:1360 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:3 cid:111 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: failing outstanding i/o Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:4 cid:126 nsid:1 lba:4800679536 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:126 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: failing outstanding i/o Oct 10 05:01:07 r730-01 kernel: nvme2: FLUSH (00) sqid:5 cid:123 nsid:1 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:123 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:5 cid:109 nsid:1 lba:6398641528 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:109 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:5 cid:113 nsid:1 lba:6788800920 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:5 cid:113 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: failing outstanding i/o Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:6 cid:107 nsid:1 lba:6497270488 len:1368 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:6 cid:107 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:6 cid:118 nsid:1 lba:4741688848 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:6 cid:118 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: failing outstanding i/o Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:7 cid:124 nsid:1 lba:6775169280 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:7 cid:124 cdw0:0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8344c038 1 7ff 0 0 0 Oct 10 05:01:07 r730-01 kernel: nvme2: failing outstanding i/o Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8344c838 1 7ff 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8344d038 1 54f 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=1e248e70 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): FLUSH (00). NCB: opc=0 fuse=0 nsid=1 prp1=0 prp2=0 cdw=0 0 0 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=7d638578 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:8 cid:123 nsid:1 lba:6497282784 len:2048 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:123 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: WRITE (01) sqid:8 cid:127 nsid:1 lba:7747276696 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:127 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:8 cid:117 nsid:1 lba:6774648024 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:8 cid:117 cdw0:0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: nvme2: failing outstanding i/o Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:9 cid:124 nsid:1 lba:6736847176 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:9 cid:124 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: failing outstanding i/o Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:13 cid:124 nsid:1 lba:6546665816 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:13 cid:124 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:13 cid:119 nsid:1 lba:5717137776 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:13 cid:119 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: failing outstanding i/o Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:15 cid:121 nsid:1 lba:4517574864 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:15 cid:121 cdw0:0 Oct 10 05:01:07 r730-01 kernel: nvme2: READ (02) sqid:15 cid:119 nsid:1 lba:4909080344 len:8 Oct 10 05:01:07 r730-01 kernel: nvme2: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:15 cid:119 cdw0:0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=94a4e198 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=83447ad8 1 557 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=1aa06e10 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=93d4e100 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=8344aae0 1 7ff 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): WRITE (01). NCB: opc=1 fuse=0 nsid=1 prp1=0 prp2=0 cdw=cdc60f98 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=93ccecd8 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=918c2148 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=86363158 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=54c49970 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 5, Retries exhausted Oct 10 05:01:07 r730-01 kernel: nda2 at nvme2 bus 0 scbus18 target 0 lun 1 Oct 10 05:01:07 r730-01 kernel: nda2: <Samsung SSD 990 PRO 4TB 4B2QJXD7 S7KGNU0Y912955D> s/n S7KGNU0Y912955D detached Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=d44b8d0 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 6, Periph was invalidated Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=249a9f18 1 7 0 0 0 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): CAM status: NVME Status Error Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Error 6, Periph was invalidated Oct 10 05:01:07 r730-01 kernel: nvme2: Failed controller, stopping watchdog timeout. Oct 10 05:01:07 r730-01 ZFS[23015]: vdev state changed, pool_guid=1760090623298529734 vdev_guid=13841628067708530877 Oct 10 05:01:07 r730-01 ZFS[23019]: vdev is removed, pool_guid=1760090623298529734 vdev_guid=13841628067708530877 Oct 10 05:01:07 r730-01 kernel: (nda2:nvme2:0:0:1): Periph destroyed Oct 10 05:01:07 r730-01 kernel: nvme2: Failed controller, stopping watchdog timeout.
Fixing it
This time, as in the previous two incidents, a power cycle failed to bring back the missing device.
Today, I pulled the NVMe card, removed the heatsink, and extracted the NVMe stick in question, first identifying it by the serial number shown above.
I cleaned the NVMe contacts using isopropyl alcohol and a cotton swab. I waited some time for it to dry, because it was only 70%. I’m going to buy some 99% from the drug store, next time I’m there.
After the power on
After reassembling the host, including a dust-out with the mini air-blaster, the zpool status showed:
state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Oct 12 21:12:18 2026 4.08T / 10.1T scanned, 11.2G / 6.16T issued at 3.74G/s 1.40G resilvered, 0.18% done, 00:28:02 to go config: NAME STATE READ WRITE CKSUM data04 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/S7KGNU0Y722875X ONLINE 0 0 0 gpt/S7KGNU0Y915666E ONLINE 0 0 0 gpt/S7KGNU0Y912937J ONLINE 0 0 0 gpt/S7KGNU0Y912955D ONLINE 0 0 2 (resilvering) gpt/S7U8NJ0Y716854P ONLINE 0 0 0 gpt/S7U8NJ0Y716801F ONLINE 0 0 0 gpt/S757NS0Y700758M ONLINE 0 0 0 gpt/S757NS0Y700760R ONLINE 0 0 0 errors: No known data errors
Shortly after:
pool: data04 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scan: resilvered 12.7G in 00:00:59 with 0 errors on Mon Oct 12 21:13:17 2026 config: NAME STATE READ WRITE CKSUM data04 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/S7KGNU0Y722875X ONLINE 0 0 0 gpt/S7KGNU0Y915666E ONLINE 0 0 0 gpt/S7KGNU0Y912937J ONLINE 0 0 0 gpt/S7KGNU0Y912955D ONLINE 0 0 2 gpt/S7U8NJ0Y716854P ONLINE 0 0 0 gpt/S7U8NJ0Y716801F ONLINE 0 0 0 gpt/S757NS0Y700758M ONLINE 0 0 0 gpt/S757NS0Y700760R ONLINE 0 0 0 errors: No known data errors
Just 1 minute to fix after being offline for 69 hours.
Notes for me
I should update my config:
Oct 1 18:09:51 r730-01 kernel: WARNING: sysctl vfs.zfs.arc_max is deprecated. Use vfs.zfs.arc.max instead.
And buy that 99% stuff.
