I was upgrading a host the other day. The mysql02 (that’s the hostname) jail broke. I suspect I did something like upgrading the package while the server was in use.
I noticed LibreNMS wasn’t running:
*** /var/log/librenms/librenms-service.log *** Feb 10 18:37:30 besser librenms[23957]: MainThread(CRITICAL):ERROR: Could not connect to MySQL database! (2003, "Can't connect to MySQL server on 'mysql02.int.unixathome.org' ([Errno 61] Connection refused)") Feb 10 18:37:30 besser librenms[23957]: (2003, "Can't connect to MySQL server on 'mysql02.int.unixathome.org' ([Errno 61] Connection refused)")
I checked on mysql02 and found:
[18:37 mysql02 dvl ~] % pkg info -x mysql mysql80-client-8.0.45 php83-mysqli-8.3.30
No MySQL 8.4 (as expected, because I had recently updated it from MySQL 8.0)
It turns out, that upgrade procedure is now my recovery procedure.
I suspect this happed because I used the wrong repo during the upgrade. My fault.
Attempt #1
I installed and started mysql84-server:
[18:37 mysql02 dvl ~] % sudo pkg install -f mysql84-server Updating local repository catalogue... local repository is up to date. All repositories are up to date. Checking integrity... done (2 conflicting) - mysql84-client-8.4.8 conflicts with mysql80-client-8.0.45 on /usr/local/bin/comp_err - mysql84-client-8.4.8 conflicts with mysql80-client-8.0.45 on /usr/local/bin/comp_err Checking integrity... done (0 conflicting) The following 6 package(s) will be affected (of 0 checked): New packages to be INSTALLED: mysql84-client: 8.4.8 mysql84-server: 8.4.8 Installed packages to be REINSTALLED: pkg-2.5.1 Installed packages to be REMOVED: mysql80-client: 8.0.45 nagios-plugins: 2.4.4_1,1 nrpe: 4.1.3 Number of packages to be removed: 3 Number of packages to be installed: 2 Number of packages to be reinstalled: 1 The process will require 197 MiB more space. Proceed with this action? [y/N]: y [mysql02.int.unixathome.org] [1/6] Deinstalling nrpe-4.1.3... [mysql02.int.unixathome.org] [1/6] Deleting files for nrpe-4.1.3: 100% ==> You should manually remove the "nagios" user ==> You should manually remove the "nagios" group [mysql02.int.unixathome.org] [2/6] Deinstalling nagios-plugins-2.4.4_1,1... [mysql02.int.unixathome.org] [2/6] Deleting files for nagios-plugins-2.4.4_1,1: 100% [mysql02.int.unixathome.org] [3/6] Deinstalling mysql80-client-8.0.45... [mysql02.int.unixathome.org] [3/6] Deleting files for mysql80-client-8.0.45: 100% [mysql02.int.unixathome.org] [4/6] Installing mysql84-client-8.4.8... [mysql02.int.unixathome.org] [4/6] Extracting mysql84-client-8.4.8: 100% [mysql02.int.unixathome.org] [5/6] Installing mysql84-server-8.4.8... ===> Creating groups Using existing group 'mysql' ===> Creating users Using existing user 'mysql' ===> Creating homedir(s) [mysql02.int.unixathome.org] [5/6] Extracting mysql84-server-8.4.8: 100% [mysql02.int.unixathome.org] [6/6] Reinstalling pkg-2.5.1... [mysql02.int.unixathome.org] [6/6] Extracting pkg-2.5.1: 100% You may need to manually remove /usr/local/etc/nrpe.cfg if it is no longer needed. ===== Message from mysql84-client-8.4.8: -- This is the mysql CLIENT without the server. for complete server and client, please install databases/mysql84-server ===== Message from mysql84-server-8.4.8: -- There is no initial password for first time use of MySQL. Keep in mind to reset it to a secure password. MySQL 8.4 has a default /usr/local/etc/mysql/my.cnf, remember to replace it with your own or set `mysql_optfile="$YOUR_CNF_FILE` in rc.conf. You may need to manually remove /usr/local/etc/pkg.conf if it is no longer needed. [18:38 mysql02 dvl ~] % pkg info -x mysql mysql84-client-8.4.8 mysql84-server-8.4.8 php83-mysqli-8.3.30 [18:38 mysql02 dvl ~] % sudo service mysql-server start Starting mysql. [18:38 mysql02 dvl ~] % ps auwwx USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 9304 0.0 0.0 14432 3204 - SCsJ 16:30 0:00.04 /usr/sbin/syslogd -s root 9307 0.0 0.0 14432 3064 - IJ 16:30 0:00.00 syslogd: syslogd.casper (syslogd) root 9308 0.0 0.0 14432 3032 - IsJ 16:30 0:00.00 syslogd: system.net (syslogd) snmpd 9364 0.0 0.0 38324 15744 - SJ 16:30 0:03.52 /usr/local/sbin/snmpd -p /var/run/net_snmpd.pid -u snmpd -g snmpd -c /usr/local/etc/snmpd.conf -a -r root 9399 0.0 0.0 25200 10768 - IsJ 16:30 0:00.07 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd) root 9437 0.0 0.0 14280 2808 - IsJ 16:30 0:00.05 /usr/sbin/cron -s nagios 9460 0.0 0.0 21796 9044 - IsJ 16:30 0:00.09 /usr/local/sbin/nrpe -c /usr/local/etc/nrpe.cfg -d root 23989 0.0 0.0 25736 12192 - IsJ 18:37 0:00.02 sshd-session: dvl [priv] (sshd-session) dvl 23996 0.0 0.0 25736 12536 - SJ 18:37 0:00.09 sshd-session: dvl@pts/6 (sshd-session) dvl 23998 0.0 0.0 18896 5852 6 SsJ 18:37 0:00.07 -zsh (zsh) dvl 25002 0.0 0.0 14824 3328 6 R+J 18:38 0:00.00 ps auwwx
Except, it did not stay started.
[18:38 mysql02 dvl ~] % cd /var/db/mysql [18:38 mysql02 dvl /var/db/mysql] % ls -lt | head total 5320325 -rw-r----- 1 mysql mysql 31639 2026.02.10 18:38 mysql02.int.unixathome.org.err drwxr-x--- 2 mysql mysql 6 2026.02.10 13:00 #innodb_redo/ -rw-r----- 1 mysql mysql 439968650 2026.02.10 13:00 binlog.000759 -rw-r----- 1 mysql mysql 33554432 2026.02.10 13:00 undo_002 -rw-r----- 1 mysql mysql 33554432 2026.02.10 13:00 undo_001 -rw-r----- 1 mysql mysql 4194304 2026.02.10 13:00 #ib_16384_0.dblwr -rw-rw---- 1 mysql mysql 213909504 2026.02.10 13:00 ibdata1 -rw-r----- 1 mysql mysql 41943040 2026.02.10 13:00 mysql.ibd -rw-r----- 1 mysql mysql 560 2026.02.10 02:35 binlog.index [18:39 mysql02 dvl /var/db/mysql] % sudo tail mysql02.int.unixathome.org.err 2026-02-10T18:38:26.312248Z 0 [System] [MY-010116] [Server] /usr/local/libexec/mysqld (mysqld 8.4.8) starting as process 24705 2026-02-10T18:38:26.325932Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2026-02-10T18:38:26.546350Z 1 [ERROR] [MY-013882] [InnoDB] Missing redo log file ./#innodb_redo/#ib_redo211852 (with start_lsn = 1140755079680). 2026-02-10T18:38:26.546429Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error. 2026-02-10T18:38:26.547856Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine 2026-02-10T18:38:26.548084Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed. 2026-02-10T18:38:26.548169Z 0 [ERROR] [MY-010119] [Server] Aborting 2026-02-10T18:38:26.552286Z 0 [System] [MY-010910] [Server] /usr/local/libexec/mysqld: Shutdown complete (mysqld 8.4.8) Source distribution. 2026-02-10T18:38:26.552306Z 0 [System] [MY-015016] [Server] MySQL Server - end. 2026-02-10T18:38:26.6NZ mysqld_safe mysqld from pid file /var/db/mysql/mysql02.int.unixathome.org.pid ended
That’s not good.
I searched, didn’t want to try recovery, so decided to go to a snapshot.
The snapshots
Here are the snapshots available to me:
[18:45 r730-01 dvl ~] % zfs list -r -t snapshot data02/jails/mysql02 NAME USED AVAIL REFER MOUNTPOINT data02/jails/mysql02@autosnap_2026-02-04_00:00:15_daily 47.0M - 7.64G - data02/jails/mysql02@autosnap_2026-02-05_00:00:01_daily 39.5M - 7.81G - data02/jails/mysql02@autosnap_2026-02-06_00:00:08_daily 39.1M - 7.79G - data02/jails/mysql02@autosnap_2026-02-07_00:00:12_daily 39.4M - 7.78G - data02/jails/mysql02@autosnap_2026-02-08_00:00:16_daily 41.2M - 7.76G - data02/jails/mysql02@autosnap_2026-02-09_00:00:08_daily 27.9M - 7.73G - data02/jails/mysql02@autosnap_2026-02-10_00:00:06_daily 39.1M - 7.89G - data02/jails/mysql02@autosnap_2026-02-10_12:00:07_hourly 18.1M - 7.61G - data02/jails/mysql02@autosnap_2026-02-10_13:00:15_hourly 12.3M - 7.62G - data02/jails/mysql02@mkjail-202602101312 1.09M - 7.59G - data02/jails/mysql02@autosnap_2026-02-10_13:45:10_frequently 212K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_14:00:02_hourly 0B - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_14:00:02_frequently 0B - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_14:15:07_frequently 212K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_14:30:03_frequently 212K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_14:45:07_frequently 212K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_15:00:02_hourly 0B - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_15:00:02_frequently 0B - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_15:15:09_frequently 212K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_15:30:00_frequently 212K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_15:45:08_frequently 212K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_16:00:00_hourly 0B - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_16:00:00_frequently 0B - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_16:15:09_frequently 484K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_16:45:11_frequently 220K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_17:00:05_hourly 0B - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_17:00:05_frequently 0B - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_17:15:07_frequently 220K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_17:30:01_frequently 180K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_17:45:09_frequently 180K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_18:00:00_hourly 0B - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_18:00:00_frequently 0B - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_18:15:09_frequently 232K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_18:30:02_frequently 224K - 7.64G - data02/jails/mysql02@autosnap_2026-02-10_18:45:08_frequently 0B - 7.71G -
Let’s try mkjail-202602101312. Following the instructions from the creation of mysql02, I did this:
[18:45 r730-01 dvl ~] % sudo service jail stop mysql02 Stopping jails: mysql02. [18:47 r730-01 dvl ~] % sudo zfs rename data02/jails/mysql02 data02/jails/mysql02.bad cannot unmount '/jails/mysql02': pool or dataset is busy [18:47 r730-01 dvl ~] % zfs mount | grep mysql02 data02/jails/mysql02 /jails/mysql02 [18:47 r730-01 dvl ~] % sudo zfs rename -f data02/jails/mysql02 data02/jails/mysql02.bad [18:47 r730-01 dvl ~] % sudo zfs clone data02/jails/mysql02.bad@mkjail-202602101312 data02/jails/mysql02 [18:48 r730-01 dvl ~] % zfs get -r -t filesystem mkjail:version data02/jails | grep -v 15.0 [18:48 r730-01 dvl ~] % zfs get mkjail:version data02/jails/mysql02 NAME PROPERTY VALUE SOURCE data02/jails/mysql02 mkjail:version - - [18:48 r730-01 dvl ~] % zfs get mkjail:version data02/jails/mysql02.bad NAME PROPERTY VALUE SOURCE data02/jails/mysql02.bad mkjail:version 15.0-RELEASE-p2 local [18:48 r730-01 dvl ~] % sudo zfs set mkjail:version=15.0-RELEASE-p2 data02/jails/mysql02 [18:49 r730-01 dvl ~] % zfs get mkjail:version data02/jails/mysql02 NAME PROPERTY VALUE SOURCE data02/jails/mysql02 mkjail:version 15.0-RELEASE-p2 local [18:49 r730-01 dvl ~] % zfs get mkjail:version data02/jails/mysql02 data02/jails/mysql02.bad NAME PROPERTY VALUE SOURCE data02/jails/mysql02 mkjail:version 15.0-RELEASE-p2 local data02/jails/mysql02.bad mkjail:version 15.0-RELEASE-p2 local [18:49 r730-01 dvl ~] % sudo service jail start mysql02 Starting jails: mysql02.
Checking the jail, I found MySQL was not running. The errors I found were:
2026-02-10T18:51:40.814206Z 0 [ERROR] [MY-011972] [InnoDB] Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB redo log files. Please refer to http://dev.mysql.com/doc/refman/8.4/en/forcing-innodb-recovery.html for information about forcing recovery. 2026-02-10T18:51:40.971100Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2026-02-10T18:51:41.093682Z 1 [ERROR] [MY-011971] [InnoDB] Tablespace 'mysql' Page [page id: space=4294967294, page number=5] log sequence number 1140756679947 is in the future! Current system log sequence number 1140752966367. 2026-02-10T18:51:41.093713Z 1 [ERROR] [MY-011972] [InnoDB] Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB redo log files. Please refer to http://dev.mysql.com/doc/refman/8.4/en/forcing-innodb-recovery.html for information about forcing recovery. 2026-02-10T18:51:41.212788Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: fut0lst.ic:82:addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA thread 0x13a8e18aa008
That’s not good.
Let’s try an earlier snapshot.
Second time
Next, I tried this:
[18:50 r730-01 dvl ~] % sudo service jail stop mysql02 Stopping jails: mysql02. [18:50 r730-01 dvl ~] % sudo zfs rename -f data02/jails/mysql02 data02/jails/mysql02.bad.part2 [18:50 r730-01 dvl ~] % sudo zfs clone data02/jails/mysql02.bad@autosnap_2026-02-10_13:00:15_hourly data02/jails/mysql02 [18:51 r730-01 dvl ~] % sudo zfs set mkjail:version=15.0-RELEASE-p2 data02/jails/mysql02 [18:51 r730-01 dvl ~] % sudo service jail start mysql02 Starting jails: mysql02.
Nope, no good. Still would not start.
Third try
I decided to review the entire log. I found this lead:
2026-02-08T22:29:56.490860Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. 2026-02-08T22:29:56.490904Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. 2026-02-08T22:29:56.521920Z 0 [System] [MY-010931] [Server] /usr/local/libexec/mysqld: ready for connections. Version: '8.4.7' socket: '/tmp/mysql.sock' port: 3306 Source distribution. 2026-02-08T22:29:56.521913Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '0.0.0.0' port: 33060, socket: /tmp/mysqlx.sock 2026-02-10T12:57:08.371825Z 0 [ERROR] [MY-012894] [InnoDB] Unable to open './#innodb_redo/#ib_redo211852' (error: 1504). 2026-02-10T12:57:09.378123Z 0 [ERROR] [MY-012894] [InnoDB] Unable to open './#innodb_redo/#ib_redo211852' (error: 1504).
It went bad, or was detected bad, at 2026-02-10T12:57.
next, I tried:
[18:52 r730-01 dvl ~] % sudo service jail stop mysql02 Stopping jails: mysql02. [18:54 r730-01 dvl ~] % sudo zfs rename -f data02/jails/mysql02 data02/jails/mysql02.bad.part3 [18:55 r730-01 dvl ~] % sudo zfs rename -f data02/jails/mysql02 data02/jails/mysql02.bad.part3 [18:56 r730-01 dvl ~] % sudo zfs clone data02/jails/mysql02.bad@autosnap_2026-02-10_12:00:07_hourly data02/jails/mysql02 [18:56 r730-01 dvl ~] % sudo zfs set mkjail:version=15.0-RELEASE-p2 data02/jails/mysql02 [18:56 r730-01 dvl ~] % sudo service jail start mysql02 Starting jails: mysql02.
Success. Yes, I have lost some data. I’m OK with that. This database is used mainly for LibreNMS – it’s just metrics.
Next, I upgraded.
The upgrade
I realize now that I was wrong to do the set mkjail:version=15.0-RELEASE-p2 – that will mislead mkjail.
First, I needed to build my MySQL 8.4 specific packages:
[19:56 pkg01 dvl ~] % time sudo poudriere bulk -j 150amd64 -p default -z mysql84 -f /usr/local/etc/poudriere.d/buildlists/mysql84
Now, I wait, about two hours. For those packages to build. Then upgrade the jail like this:
[23:32 r730-01 dvl ~] % time sudo mkjail upgrade -v "15.0-RELEASE" -j mysql02 Upgrading mysql02 jail from 15.0-RELEASE to 15.0-RELEASE... D /.cshrc D /.profile D /etc/ftpusers ... /usr/share/man/man8/devinfo.8.gz Installing updates...certctl: legacy directory /jails/mysql02/etc/ssl/blacklisted can safely be deleted done. sudo mkjail upgrade -v "15.0-RELEASE" -j mysql02 0.03s user 0.13s system 0% cpu 3:53.36 total
Interesting, that bit about “legacy directory /jails/mysql02/etc/ssl/blacklisted can safely be deleted”. I’ll have to look into that.
