pkg: http://vuxml.freebsd.org/freebsd/vuln.xml.bz2: No address record

I’ve been making use of some FreeBSD-provided scripts within my Nagios monitoring. Recently, I started seeing a problem after some jail maintenance. This post is about that problem and the fix.

Full disclosure: the issue was not what I thought it was and I did not solve it.

I’m using:

• FreeBSD 11.2-RELEASE-p9

The scripts are:

• 405.pkg-base-audit
• 410.pkg-audit

Where are they from?

$ pkg which /usr/local/etc/periodic/security/405.pkg-base-audit
/usr/local/etc/periodic/security/405.pkg-base-audit was installed by package base-audit-0.3

You don’t say? Here’s more about that:

$ pkg info base-audit
base-audit-0.3
Name           : base-audit
Version        : 0.3
Installed on   : Wed Jan 16 17:34:00 2019 UTC
Origin         : security/base-audit
Architecture   : FreeBSD:11:*
Prefix         : /usr/local
Categories     : security
Licenses       : BSD3CLAUSE
Maintainer     : 000.fbsd@quip.cz
WWW            : UNKNOWN
Comment        : Daily periodic check of vulnerabilities in base system
Annotations    :
	repo_type      : binary
	repository     : local
Flat size      : 6.33KiB
Description    :
Audit base system against known vulnerabilities and generate reports 
including references to security advisories.
It uses pkg audit and Vuxml database as is used for packages but this script
checks base system.

Much of the groundwork for security/base-audit was done by Mark Felder.

$ pkg which /usr/local/etc/periodic/security/410.pkg-audit
/usr/local/etc/periodic/security/410.pkg-audit was installed by package pkg-1.10.5_5

And pkg is just ports-mgmt/pkg.

The problem

The issue is with 410.pkg-audit and the output in Nagios is not pretty (and that is my fault) so instead of pasting that, I will run the command from a shell:

[dan@zuul:~] $ sudo /usr/local/etc/periodic/security/410.pkg-audit     

Checking for packages with security vulnerabilities:
Host system:
Database fetched: Sat May 11 09:40:25 UTC 2019

jail: zuul_pg01
Database fetched: Sat May 11 14:40:51 UTC 2019

[redacted]

jail: ioc-zuul-pg01
pkg: http://vuxml.freebsd.org/freebsd/vuln.xml.bz2: No address record
pkg: cannot fetch vulnxml file
pkg: sqlite error while executing ALTER TABLE packages ADD licenselogic INTEGER NOT NULL DEFAULT(1); in file pkgdb.c:2406: no such table: packages
pkg: sqlite error while executing ALTER TABLE packages ADD licenselogic INTEGER NOT NULL DEFAULT(1); in file pkgdb.c:2406: no such table: packages

[redacted]

jail: ioc-webs02_vpn_unixathome_org
[dan@zuul:~] $ 

From within the jail, this worked fine:

[dan@zuul-pg01:~] $ sudo pkg audit -F   
vulnxml file up-to-date
0 problem(s) in the installed packages found.
[dan@zuul-pg01:~] $ 

I copied the original script and started adding some debugging statements. I could see the output of the commands it was going to run:

jail: ioc-zuul-pg01

/usr/local/sbin/pkg -j ioc-zuul-pg01 audit -F -q
pkg: http://vuxml.freebsd.org/freebsd/vuln.xml.bz2: No address record
pkg: cannot fetch vulnxml file
pkg: sqlite error while executing ALTER TABLE packages ADD licenselogic INTEGER NOT NULL DEFAULT(1); in file pkgdb.c:2406: no such table: packages
pkg: sqlite error while executing ALTER TABLE packages ADD licenselogic INTEGER NOT NULL DEFAULT(1); in file pkgdb.c:2406: no such table: packages

Running the same command via the command line gave the same error. Running it within the jail was fine.

What was wrong?

iocage list

For some reason, I ran ifconfig within the jail.

[dan@zuul-pg01:~] $ ifconfig
em0: flags=8843 metric 0 mtu 1500
	options=209b
	ether 74:d0:2b:98:19:22
	hwaddr 74:d0:2b:98:19:22
	inet 10.80.0.128 netmask 0xffffffff broadcast 10.80.0.128 
	media: Ethernet autoselect (1000baseT )
	status: active
em1: flags=8c02 metric 0 mtu 1500
	options=209b
	ether 74:d0:2b:98:19:23
	hwaddr 74:d0:2b:98:19:23
	media: Ethernet autoselect
	status: no carrier
lo0: flags=8049 metric 0 mtu 16384
	options=600003
	groups: lo 
tun0: flags=8051 metric 0 mtu 1500
	options=80000
	groups: tun 
	Opened by PID 1063
lo1: flags=8049 metric 0 mtu 16384
	options=600003
	inet 127.1.54.32 netmask 0xffffffff 
	inet 127.1.0.128 netmask 0xffffffff 
	groups: lo 
pflog0: flags=141 metric 0 mtu 33160
	groups: pflog 

Looking at em0, I see 10.80.0.128. Let’s look on the host:

[dan@zuul:~] $ ifconfig | grep 10.80.0.128
	inet 10.80.0.128 netmask 0xffffffff broadcast 10.80.0.128 
[dan@zuul:~] $ 

Yes, it’s there. It is on the host. Let’s check iocage:

[dan@zuul:~] $ iocage get all zuul-pg01 | grep addr
ip4_addr:none
ip4_saddrsel:1
ip6_addr:none
ip6_saddrsel:1
[dan@zuul:~] $ 

There! ip4_addr should not be empty!

Why does the jail have an IP address yet iocage does not show it? I am not about to delve into that. I want to fix the issue.

Let’s set the IP4 address.

[dan@zuul:~] $ sudo iocage set ip4_addr=10.80.0.128 zuul-pg01
Property: ip4_addr has been updated to 10.80.0.128
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 1745, in json_set_value
    stderr=su.STDOUT)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 609, in checkoutput
    out = su.check_output(*args, **kwargs)
  File "/usr/local/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/local/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['jail', '-m', 'jid=55', 'ip4.addr=10.80.0.128']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in 
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/set.py", line 59, in cli
    jail=jail, skip_jails=True).set(prop, plugin)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1578, in set
    iocjson.json_set_value(prop)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 1748, in json_set_value
    f"{err.output.decode('utf-8').rstrip()}")
RuntimeError: jail: IPv4 addresses clash
[dan@zuul:~] $ 

That didn’t go well.

$ iocage --version
Version	1.1 RELEASE 2019/01

However, it is set:

[dan@zuul:~] $ iocage get ip4_addr zuul-pg01
10.80.0.128

I restarted the jail:

[dan@zuul:~] $ sudo iocage restart zuul-pg01
* Stopping zuul-pg01
  + Executing prestop OK
  + Stopping services OK
  + Removing devfs_ruleset: 20 OK
  + Removing jail process OK
  + Executing poststop OK
* Starting zuul-pg01
  + Start FAILED
jail: ioc-zuul-pg01: IPv4 addresses clash

[dan@zuul:~] $ 

Attempted diagnostics

What I found interesting: the jail did not get stopped. At all. My ssh-session was still active.

Let’s look at the config:

[dan@zuul:/iocage/jails/zuul-pg01] $ cat config.json 
{
    "host_hostname": "zuul-pg01",
    "host_hostuuid": "zuul-pg01",
    "ip4_addr": "10.80.0.128",
    "jail_zfs_dataset": "iocage/jails/zuul-pg01/data",
    "last_started": "2019-05-11 13:39:20",
    "release": "11.2-RELEASE-p9"
}[dan@zuul:/iocage/jails/zuul-pg01] $ 

That looks OK. What’s up with this?

Where is this IPv4 address clash?

I can’t have this jail offline: it’s the database server for BSDCan & PGCon. I don’t want to force it offline either: it might not come back.

When I started writing this blog post, I thought I had the solution, but it seems blocked by something else.

This is not an iocage failure, it is jail failing.

I do not see the IP address used in other jails:

[dan@zuul:/iocage/jails] $ grep 10.80.0.128 */config.json
zuul-pg01/config.json:    "ip4_addr": "10.80.0.128",
[dan@zuul:/iocage/jails] $ 

I also tried:

$ jls -v | grep 10.80.0.128
        10.80.0.128

Nothing. OK, what’s up with that?

Then I got the idea to add in the other two IP addresses:

$ sudo iocage set ip4_addr="10.80.0.128,127.1.54.32,127.1.0.128" zuul-pg01 zuul-pg01
Property: ip4_addr has been updated to 10.80.0.128,127.1.54.32,127.1.0.128

I tried to ssh to the jail. I failed. No connection.

Now I’m worried. This jail is vital. BSDCan start in 2 days.

I console into the jail:

[dan@zuul:/iocage/jails/zuul-pg01] $ sudo iocage console zuul-pg01

root@zuul-pg01:~ # bash
bash: Command not found.

Umm. That should be there.

What has happened to this jail?

root@zuul-pg01:~ # ps auwwx
USER   PID %CPU %MEM  VSZ  RSS TT  STAT STARTED    TIME COMMAND
root 35801  2.0  0.0 7488 3752  4  SJ   18:16   0:00.01 -csh (csh)
root 35207  0.0  0.0 6412 2364  -  SsJ  18:15   0:00.00 /usr/sbin/syslogd -c -ss
root 35261  0.0  0.0 6464 2380  -  IsJ  18:15   0:00.00 /usr/sbin/cron -J 15 -s
root 35800  0.0  0.0 6952 2888  4  SJ   18:16   0:00.01 login [pam] (login)
root 35807  0.0  0.0 6992 2844  4  R+J  18:16   0:00.00 ps auwwx

There is nothing running, just the basics.

I start looking at jail_zfs_dataset, and became overly concerned about that.

I tried restarting:

n@zuul:/iocage/jails/zuul-pg01] $ sudo iocage do iocage restart zuul-pg01
zuul-pg01 is not running!
* Starting zuul-pg01
  + Start FAILED
jail: ioc-zuul-pg01: IPv4 addresses clash

[dan@zuul:/iocage/jails/zuul-pg01] $ 

OK, but it is running:

[dan@zuul:/iocage/jails/zuul-pg01] $ jls | grep pg01
     3  10.80.0.128     zuul-pg01.vpn.unixathome.org  /usr/jails/zuul-pg01

Wait, that’s the wrong path.

That’s ezjail, not iocage.

Doh

This host is running both iocage and ezjail jails.

I started converting the jails and then stopping, leaving bsdcan.org, pgcon.org, and zuul-pg01 still running on iocage.

That IPv4 addresses clash message was jail telling me that something else was already using that.

What was the problem I was trying to solve?

In my mind, with coffee but without much food yet, I thought I had discovered that the jail did not have the required IP address. Easily fixed, I can add it in. What I did not realize was:

1. The jail would not work without that IP address.
2. The jail was working

The problem was not with a running jail, it was with a non-functional jail, which was not running.

Hindsight

That the two localhost IP addresses were not present should have made me pause. They are essential to the operation of the jail.

A the start of this post, I showed the output of the 405.pkg-base-audit script. Upon close examination, I noticed that both zuul-pg01 and ioc-zuul-pg01 were being queried. The former was successful; the latter was failing.

I think I’d like to patch 410.pkg-audit so it does not query non-running jails.

I could set daily_status_security_pkgaudit_jails, but then I’d have to maintain it and add new jails in as they were created. I’d forget something. Jails would become vuln. The system would be pawned. No, I’ll live with this for now.