hacking on iocage – Dan Langille's Other Diary

Today is the day after BSDCan 2019. The power cables and extension cords from the hacking lounge have been laid to rest in an Ottawa basement until next year.

Sitting in my parents garden, I noticed some Nagios cert warnings:

WARNING - Certificate x8dtu.unixathome.org expires in 21 day(s) (2019-06-10 17:18 +0000/UTC).

I logged into my certificates server (the website from which all my hosts download their certs). The cert looked OK:

[dan@webs01:/usr/local/www/certs.unixathome.org/www/certs/x8dtu.unixathome.org] $ ls -l
total 14
-rw-r--r--  1 rsyncer  rsyncer  1647 May 11 18:17 ca.cer
-rw-r--r--  1 rsyncer  rsyncer  1923 May 11 18:17 x8dtu.unixathome.org.cer
-rw-r--r--  1 rsyncer  rsyncer  3570 May 11 18:17 x8dtu.unixathome.org.fullchain.cer
[dan@webs01:/usr/local/www/certs.unixathome.org/www/certs/x8dtu.unixathome.org] $

Let’s try running cert-puller on the host. Checking /var/log/messages, I found:

[dan@x8dtu:~] $ tail -F /var/log/messages
May 19 15:02:00 x8dtu /usr/sbin/cron[48554]: in openpam_load_module(): no pam_nologin.so found
May 19 15:20:41 x8dtu kernel: Limiting open port RST response from 204 to 200 packets/sec
May 19 16:02:00 x8dtu /usr/sbin/cron[80502]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
May 19 16:02:00 x8dtu /usr/sbin/cron[80502]: in openpam_load_module(): no pam_nologin.so found
May 19 16:09:42 x8dtu kernel: Limiting open port RST response from 220 to 200 packets/sec
May 19 17:02:00 x8dtu /usr/sbin/cron[13093]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
May 19 17:02:00 x8dtu /usr/sbin/cron[13093]: in openpam_load_module(): no pam_nologin.so found
May 19 18:02:00 x8dtu /usr/sbin/cron[44882]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
May 19 18:02:00 x8dtu /usr/sbin/cron[44882]: in openpam_load_module(): no pam_nologin.so found
May 19 18:24:55 x8dtu kernel: Limiting open port RST response from 331 to 200 packets/sec
May 19 18:50:00 x8dtu /usr/sbin/cron[75241]: in try_dlopen(): /usr/lib/pam_nologin.so.6: (null): Undefined symbol "fstat@FBSD_1.5"
May 19 18:50:00 x8dtu /usr/sbin/cron[75241]: in openpam_load_module(): no pam_nologin.so found

Oh. This indicated I needed to run freebsd-update, based on search results for the above error message. The host was on FreeBSD 12.0-RELEASE-p3 and I knew -p5 was out.

The upgrade and reboot went find. However, now I have iocage problems:

[dan@x8dtu:~] $ sudo service iocage start x8dtu-pg02
* [I|O|C] starting jails... 
Usage: iocage [OPTIONS] COMMAND [ARGS]...
Try "iocage --help" for help.

Error: No such command "start".

I know there is an issue raised in March, not yet fixed.

But for starters, I’m confusing the service with the command.

I should not be specifying a jail name when invoking the service command. It’s either start or stop.

Or I could be doing just a iocage start x8dtu-pg02.

Getting it working

Here is what I did to get things working.

As suggested by drjohnnyfever1, I added this line to /usr/local/etc/rc.d/iocage:

export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin

Yes, this allowed me to start a jail.

[dan@x8dtu:~] $ sudo service iocage start x8dtu-pg01
* [I|O|C] starting jails... 
Please set kern.racct.enable -> 1 to set rctl rules
[dan@x8dtu:~] $

As shown, I was still abusing the service command.

And only one jail started. To get all jails running, I did this added these lines to /boot/loader.conf

# iocage is messed up. see https://github.com/iocage/iocage/issues/894#issuecomment-493786157
kern.racct.enable="1"

A reboot then allowed all jails to start. This is a workaround. The code will be fixed soon I hope.

Things for me to do later

Replicate the following on all iocage hosts:

patch my local copy of sysutils/iocage-devel to add that export line
pkg upgrade py36-iocage-devel
~~add kern.racct.enable=”1″ to /boot/loader.conf~~ – no, this is not required. See below.

EDIT 2019-07-04

NOTE: that kern.racct.enable=”1″ is misleading. See this iocage issue for details.

In short, you’ll probably find: iocage get memoryuse YOURJAIL returns 8G:log

Your jail is probably predates the change of iocage to Python. If you keep reading in that issue, you’ll see that part of the discussion.

This fixed the issue for me: