ZFS

da12 in my FreeBSD zfs array disappeared at :02 and came back at :04

Leave a Comment / ZFS / By /

This post is based on a tweet and was created after a followup incident occurred today. The post was created to consolidate the information into a blog post so I could easily find it later. Details about this host (disks, zpool, gpart, etc) are in this post. On March 15 2022, I noticed these messages in /var/log/messages: The zpool array was degraded. I don’t have a record of that message. The tweet also […]

da12 in my FreeBSD zfs array disappeared at :02 and came back at :04 Read More »

mps0: IOC Fault 0x40007e23, Resetting

Leave a Comment / File Systems, FreeBSD, Open Source, ZFS, ZFS / By /

Here I am, sitting on a beach, writing a blog post, and sipping a cool adult beverage. Reading email. I see this: I quickly ssh into the host to check zpool status: Lines 15-17 are relevant. There was a resilver event, which completed at 08:49:48 The vdev state changed event occurred at 08:49:34 That all seems to tie in, time-wise. More info than you want This displays vdev guids: Line 11 shows the

mps0: IOC Fault 0x40007e23, Resetting Read More »

I forgot to enable compression on ZFS

Leave a Comment / ZFS / By /

I forgot to enable compression on this FreshPorts node. I have no idea why/how. After doing this, I went to a directory with a lot of space and copied stuff around. [aws-1 dan /jails/ingress01/var/db/freshports/message-queues/archive] % sudo mv 2021_03.tar.gz 2021_03.tar.gz.tmp 21:01:28 [aws-1 dan /jails/ingress01/var/db/freshports/message-queues/archive] % sudo mv 2021_03.tar.gz.tmp 2021_03.tar.gz 21:01:38 That will free up *some* space. But really, most of this zpool really needs to have zfs send | zfs recv done on each

I forgot to enable compression on ZFS Read More »

Adjusting my ZFS filesystems to conform with standard FreeBSD boot environments

Leave a Comment / beadm, bectl, ZFS / By /

Get FreeBSD 12.2 mfsBSD (because that matches the OS on slocum), burn it to a thumb drive using OSX. NOTE: the following didn’t work. I wound up not having the right values in /lib and /var/db/ I don’t know what I did wrong, but I did manage to fix them up afterwards. I suspect that I still had zroot/usr mounted and did not notice. I will keep my notes here for future reference

Adjusting my ZFS filesystems to conform with standard FreeBSD boot environments Read More »

Moving zroot/usr/local to zroot/usr

Leave a Comment / bectl, FreeBSD, freebsd-update, ZFS / By /

Today I was updating a FreeBSD server from 12.2 to 13.0 – I was using a new approach for my upgrades. This was my second host to upgrade like this. The first went smoothly. This one, not so much. NOTE: this turned out to be insufficient because /usr was mounted: [dan@slocum:~] $ zfs get canmount zroot/usr NAME PROPERTY VALUE SOURCE zroot/usr canmount on received This system was manually converted, poorly, to a BE

Moving zroot/usr/local to zroot/usr Read More »

snapshot ; ls ; destroy; dataset is busy – WTF?

Leave a Comment / FreeBSD, Jails, ZFS / By /

I have found this bizarre situation where a FreeBSD 12.2 snapshot cannot be destroyed. This is the simple example, which works: [dan@pkg01:~] $ sudo zfs snapshot tank_fast/poudriere@snapshot-for-backup [dan@pkg01:~] $ sudo zfs destroy tank_fast/poudriere@snapshot-for-backup [dan@pkg01:~] $ This is the simple example, which fails: [dan@pkg01:~] $ sudo zfs snapshot tank_fast/poudriere@snapshot-for-backup [dan@pkg01:~] $ ls /usr/local/poudriere/.zfs/snapshot/snapshot-for-backup cache data jails ports [dan@pkg01:~] $ sudo zfs destroy tank_fast/poudriere@snapshot-for-backup cannot destroy snapshot tank_fast/poudriere@snapshot-for-backup: dataset is busy [dan@pkg01:~] $ EDIT: we

snapshot ; ls ; destroy; dataset is busy – WTF? Read More »

Moving some ZFS filesystems to the ‘trash’ and removing all their snapshots – sanoid

Leave a Comment / General, sanoid, snapshots, ZFS / By /

I recently discovered that you can delete all snapshot from a ZFS filesystem with a single command. It came to me via fortune: You can delete a range of ZFS snapshots (a-z) in multiple ways. The following will delete d and all earlier snapshots: zfs destroy mypool/data@%d To delete d and all later snapshots: zfs destroy mypool/data@d% To delete all dataset snapshots: zfs destroy mypool/data@% Make sure to let ZFS perform a dry

Moving some ZFS filesystems to the ‘trash’ and removing all their snapshots – sanoid Read More »

The replacement – a followup

Leave a Comment / FreeBSD, ZFS / By /

Yesterday, I started a zpool replace. It finished overnight, and dropped the suspect drive out of the vdev. The resilver finished in the middle of the night: As you can see, da22p1 has taken the place of da17p1. The resilver took As the new drive is being resilvered, I started looking at metrics. Here is gstat. Look at how da22, the replacement drive is getting all the writes, compared to the others. LibreNMS

The replacement – a followup Read More »

Replacing a failing drive in a ZFS zpool

Leave a Comment / Disks, hardware, ZFS / By /

In this post I will replace a working, but suspect, drive with another drive. No down time. The server is knew. In this post: FreeBSD 12.2 ZFS TOSHIBA MD04ACA500 5TB drive – the suspect drive: da17 TOSHIBA HDWE150 5TB drive – the replacement: da22 None of these drives are under warranty What drives are in this server? I have had good luck with Toshiba DT01ACA300 3TB drives (presenting as Hitachi HDS723030BLE640), starting a

Replacing a failing drive in a ZFS zpool Read More »

smartctl output TOSHIBA MD04ACA500 653AK2MXFS9A

Leave a Comment / FreeBSD, Open Source, ZFS / By /

It is time to replace /dev/da17 in knew. The replacement drive is also documented. The replacement procedure has also been documented. An upcoming blog post will document the replacement. I’m seeing these messages: Dec 12 09:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 09:53:04 knew syslogd: last message repeated 1 times Dec 12 10:23:03 knew syslogd: last message repeated 1 times Dec 12 10:53:04 knew syslogd: last message

smartctl output TOSHIBA MD04ACA500 653AK2MXFS9A Read More »

Scroll to Top