This post is based on a tweet and was created after a followup incident occurred today. The post was created to consolidate the information into a blog post so I could easily find it later. Details about this host (disks, zpool, gpart, etc) are in this post. On March 15 2022, I noticed these messages […]
da12 in my FreeBSD zfs array disappeared at :02 and came back at :04 Read More »
Here I am, sitting on a beach, writing a blog post, and sipping a cool adult beverage. Reading email. I see this: I quickly ssh into the host to check zpool status: Lines 15-17 are relevant. There was a resilver event, which completed at 08:49:48 The vdev state changed event occurred at 08:49:34 That all
mps0: IOC Fault 0x40007e23, Resetting Read More »
I forgot to enable compression on this FreshPorts node. I have no idea why/how. After doing this, I went to a directory with a lot of space and copied stuff around. [aws-1 dan /jails/ingress01/var/db/freshports/message-queues/archive] % sudo mv 2021_03.tar.gz 2021_03.tar.gz.tmp 21:01:28 [aws-1 dan /jails/ingress01/var/db/freshports/message-queues/archive] % sudo mv 2021_03.tar.gz.tmp 2021_03.tar.gz 21:01:38 That will free up *some* space.
I forgot to enable compression on ZFS Read More »
Get FreeBSD 12.2 mfsBSD (because that matches the OS on slocum), burn it to a thumb drive using OSX. NOTE: the following didn’t work. I wound up not having the right values in /lib and /var/db/ I don’t know what I did wrong, but I did manage to fix them up afterwards. I suspect that
Adjusting my ZFS filesystems to conform with standard FreeBSD boot environments Read More »
Today I was updating a FreeBSD server from 12.2 to 13.0 – I was using a new approach for my upgrades. This was my second host to upgrade like this. The first went smoothly. This one, not so much. NOTE: this turned out to be insufficient because /usr was mounted: [dan@slocum:~] $ zfs get canmount
Moving zroot/usr/local to zroot/usr Read More »
I have found this bizarre situation where a FreeBSD 12.2 snapshot cannot be destroyed. This is the simple example, which works: [dan@pkg01:~] $ sudo zfs snapshot tank_fast/poudriere@snapshot-for-backup [dan@pkg01:~] $ sudo zfs destroy tank_fast/poudriere@snapshot-for-backup [dan@pkg01:~] $ This is the simple example, which fails: [dan@pkg01:~] $ sudo zfs snapshot tank_fast/poudriere@snapshot-for-backup [dan@pkg01:~] $ ls /usr/local/poudriere/.zfs/snapshot/snapshot-for-backup cache data jails
snapshot ; ls ; destroy; dataset is busy – WTF? Read More »
I recently discovered that you can delete all snapshot from a ZFS filesystem with a single command. It came to me via fortune: You can delete a range of ZFS snapshots (a-z) in multiple ways. The following will delete d and all earlier snapshots: zfs destroy mypool/data@%d To delete d and all later snapshots: zfs
Moving some ZFS filesystems to the ‘trash’ and removing all their snapshots – sanoid Read More »
Yesterday, I started a zpool replace. It finished overnight, and dropped the suspect drive out of the vdev. The resilver finished in the middle of the night: As you can see, da22p1 has taken the place of da17p1. The resilver took As the new drive is being resilvered, I started looking at metrics. Here is
The replacement – a followup Read More »
In this post I will replace a working, but suspect, drive with another drive. No down time. The server is knew. In this post: FreeBSD 12.2 ZFS TOSHIBA MD04ACA500 5TB drive – the suspect drive: da17 TOSHIBA HDWE150 5TB drive – the replacement: da22 None of these drives are under warranty What drives are in
Replacing a failing drive in a ZFS zpool Read More »
It is time to replace /dev/da17 in knew. The replacement drive is also documented. The replacement procedure has also been documented. An upcoming blog post will document the replacement. I’m seeing these messages: Dec 12 09:23:03 knew smartd[2124]: Device: /dev/da17 [SAT], 40 Currently unreadable (pending) sectors Dec 12 09:53:04 knew syslogd: last message repeated 1
smartctl output TOSHIBA MD04ACA500 653AK2MXFS9A Read More »