I started playing with /usr/local/etc/periodic/security/405.pkg-base-audit as part of a monitoring system. It works fine from the command line, but when I use Nagios plugins, I am getting unexpected results.
By unexpected, I mean messages about FreeBSD 10.2. The host in question runs FreeBSD 12.0.
The problem cannot be reproduced on the host, only from the Nagios monitoring host.
Oh wait, the Nagios monitoring host is a jail on the host in question. That *should* not matter.
That thought forced me to verify this, so I ran a quick test. I get the same results when running the query from another host. Conclusion: it is not caused by a host/jail relationship.
What is 405.pkg-base-audit
I first spoke about this idea of using existing OS-provided resources to do more auditing of systems in a tweet from this past weekend. I wrote up some simple scripts and got going.
Where is this script from? That is an interesting question. On both FreeBSD 11.2 and FreeBSD 12.0, I get this:
$ pkg which /usr/local/etc/periodic/security/405.pkg-base-audit /usr/local/etc/periodic/security/405.pkg-base-audit was installed by package base-audit-0.3
Hmmm, I have base packages. How do I list them? That’s for a different blog post I think.
I place this additional configuration setting in /etc/periodic.conf, so it will also audit my jails.
$ grep jail /etc/periodic.conf pkg_jails="*"
However, for this testing, I will comment that out. This will reduce the output of the commands without masking the issue.
Here is the output for the problem host in question, when run from the command line:
[dan@slocum:~] $ sudo /usr/local/etc/periodic/security/405.pkg-base-audit Checking for security vulnerabilities in base (userland & kernel): Database fetched: Fri Mar 22 19:42:56 UTC 2019 0 problem(s) in the installed packages found. 0 problem(s) in the installed packages found. [dan@slocum:~] $ echo $? 0 [dan@slocum:~] $
No problems, as indicated by the exit code of 0.
Good.
Invoking nrpe
I added this line to /usr/local/etc/nrpe3.cfg
command[check_base_audit] =/usr/local/bin/sudo /usr/local/etc/periodic/security/405.pkg-base-audit
Under normal circumstances, I would enclose this call in a shell script, interrogate the exit code, and return an appropriate value. I did that, for my initial testing, but the results were odd, so I resorted to the above to keep things easy to follow.
nrpe runs as the nagios user:
[dan@slocum:~] $ ps auwwx | grep nrpe nagios 67202 0.0 0.0 15640 6736 - Is 21:04 0:00.02 /usr/local/sbin/nrpe3 -c /usr/local/etc/nrpe.cfg -d dan 76885 0.0 0.0 11292 2736 0 S+ 21:07 0:00.00 grep nrpe
There are other nrpe processes running in other jails, but I have removed them from the above output, to keep it simple.
To allow nagios to sudo, I added this line via visudo:
nagios ALL=(ALL) NOPASSWD:/usr/local/etc/periodic/security/405.pkg-base-audit
When I run this command from the Nagios server, I get this:
[dan@knew:~] $ /usr/local/libexec/nagios/check_nrpe3 -H slocum.int.unixathome.org -c check_base_audit Checking for security vulnerabilities in base (userland & kernel): Database fetched: Fri Mar 22 19:42:56 UTC 2019 0 problem(s) in the installed packages found. FreeBSD-10.2 is vulnerable: ntp -- 13 low- and medium-severity vulnerabilities CVE: CVE-2015-7871 CVE: CVE-2015-7855 CVE: CVE-2015-7854 CVE: CVE-2015-7853 CVE: CVE-2015-7852 CVE: CVE-2015-7851 CVE: CVE-2015-7850 CVE: CVE-2015-7849 CVE: CVE-2015-7848 CVE: CVE-2015-7705 CVE: CVE-2015-7704 CVE: CVE-2015-7703 CVE: CVE-2015-7702 CVE: CVE-2015-7701 CVE: CVE-2015-7692 CVE: CVE-2015-7691 WWW: https://vuxml.FreeBSD.org/freebsd/c4a18a12-77fc-11e5-a687-206a8a720317.html FreeBSD-10.2 is vulnerable: FreeBSD -- Heap vulnerability in bspatch CVE: CVE-2014-9862 WWW: https://vuxml.FreeBSD.org/freebsd/7d4f4955-600a-11e6-a6c3-14dae9d210b8.html FreeBSD-10.2 is vulnerable: openssl -- multiple vulnerabilities CVE: CVE-2015-3197 CVE: CVE-2016-0701 WWW: https://vuxml.FreeBSD.org/freebsd/3679fd10-c5d1-11e5-b85f-0018fe623f2b.html FreeBSD-10.2 is vulnerable: FreeBSD -- Multiple libarchive vulnerabilities WWW: https://vuxml.FreeBSD.org/freebsd/1a71a972-8ee7-11e6-a590-14dae9d210b8.html FreeBSD-10.2 is vulnerable: OpenSSL -- multiple vulnerabilities CVE: CVE-2016-2176 CVE: CVE-2016-2109 CVE: CVE-2016-2108 CVE: CVE-2016-2107 CVE: CVE-2016-2106 CVE: CVE-2016-2105 WWW: https://vuxml.FreeBSD.org/freebsd/01d729ca-1143-11e6-b55e-b499baebfeaf.html FreeBSD-10.2 is vulnerable: FreeBSD -- Multiple portsnap vulnerabilities WWW: https://vuxml.FreeBSD.org/freebsd/e7dcd69d-8ee6-11e6-a590-14dae9d210b8.html FreeBSD-10.2 is vulnerable: FreeBSD -- Heap overflow vulnerability in bspatch WWW: https://vuxml.FreeBSD.org/freebsd/ce808022-8ee6-11e6-a590-14dae9d210b8.html FreeBSD-10.2 is vulnerable: libarchive -- multiple vulnerabilities CVE: CVE-2015-2304 CVE: CVE-2013-0211 WWW: https://vuxml.FreeBSD.org/freebsd/7c63775e-be31-11e5-b5fe-002590263bf5.html FreeBSD-10.2 is vulnerable: ntp -- multiple vulnerabilities CVE: CVE-2015-8158 CVE: CVE-2015-8140 CVE: CVE-2015-8139 CVE: CVE-2015-8138 CVE: CVE-2015-7979 CVE: CVE-2015-7978 CVE: CVE-2015-7977 CVE: CVE-2015-7976 CVE: CVE-2015-7975 CVE: CVE-2015-7974 CVE: CVE-2015-7973 WWW: https://vuxml.FreeBSD.org/freebsd/5237f5d7-c020-11e5-b397-d050996490d0.html FreeBSD-10.2 is vulnerable: ntp -- multiple vulnerabilities CVE: CVE-2016-2519 CVE: CVE-2016-2518 CVE: CVE-2016-2517 CVE: CVE-2016-2516 CVE: CVE-2016-1551 CVE: CVE-2016-1550 CVE: CVE-2016-1549 CVE: CVE-2016-1548 CVE: CVE-2016-1547 CVE: CVE-2015-8138 CVE: CVE-2015-7704 WWW: https://vuxml.FreeBSD.org/freebsd/b2487d9a-0c30-11e6-acd0-d050996490d0.html FreeBSD-10.2 is vulnerable: OpenSSL -- multiple vulnerabilities CVE: CVE-2016-6308 CVE: CVE-2016-6307 CVE: CVE-2016-6306 CVE: CVE-2016-2181 CVE: CVE-2016-2179 CVE: CVE-2016-2178 CVE: CVE-2016-2177 CVE: CVE-2016-2180 CVE: CVE-2016-2182 CVE: CVE-2016-6302 CVE: CVE-2016-6303 CVE: CVE-2016-2183 CVE: CVE-2016-6305 CVE: CVE-2016-6304 WWW: https://vuxml.FreeBSD.org/freebsd/43eaa656-80bc-11e6-bf52-b499baebfeaf.html FreeBSD-10.2 is vulnerable: FreeBSD -- Multiple vulnerabilities of ntp CVE: CVE-2016-9311 CVE: CVE-2016-9310 CVE: CVE-2016-7434 CVE: CVE-2016-7433 CVE: CVE-2016-7431 CVE: CVE-2016-7428 CVE: CVE-2016-7427 CVE: CVE-2016-7426 WWW: https://vuxml.FreeBSD.org/freebsd/fcedcdbb-c86e-11e6-b1cf-14dae9d210b8.html FreeBSD-10.2 is vulnerable: FreeBSD -- bhyve(8) virtual machine escape CVE: CVE-2016-1889 WWW: https://vuxml.FreeBSD.org/freebsd/e722e3c6-bbee-11e6-b1cf-14dae9d210b8.html FreeBSD-10.2 is vulnerable: FreeBSD -- link_ntoa(3) buffer overflow CVE: CVE-2016-6559 WWW: https://vuxml.FreeBSD.org/freebsd/0282269d-bbee-11e6-b1cf-14dae9d210b8.html FreeBSD-10.2 is vulnerable: FreeBSD -- Possible login(1) argument injection in telnetd(8) CVE: CVE-2016-1888 WWW: https://vuxml.FreeBSD.org/freebsd/e00304d2-bbed-11e6-b1cf-14dae9d210b8.html FreeBSD-10.2 is vulnerable: openssh -- command injection when X11Forwarding is enabled CVE: CVE-2016-3115 WWW: https://vuxml.FreeBSD.org/freebsd/e4644df8-e7da-11e5-829d-c80aa9043978.html FreeBSD-10.2 is vulnerable: openssl -- multiple vulnerabilities CVE: CVE-2015-3196 CVE: CVE-2015-3195 CVE: CVE-2015-3194 CVE: CVE-2015-3193 CVE: CVE-2015-1794 WWW: https://vuxml.FreeBSD.org/freebsd/4c8d1d72-9b38-11e5-aece-d050996490d0.html FreeBSD-10.2 is vulnerable: FreeBSD -- Multiple integer overflows in expat (libbsdxml) XML parser CVE: CVE-2015-1283 WWW: https://vuxml.FreeBSD.org/freebsd/0da8a68e-600a-11e6-a6c3-14dae9d210b8.html FreeBSD-10.2 is vulnerable: FreeBSD -- rpcbind(8) remote denial of service [REVISED] CVE: CVE-2015-7236 WWW: https://vuxml.FreeBSD.org/freebsd/0e5d6969-600a-11e6-a6c3-14dae9d210b8.html FreeBSD-10.2 is vulnerable: FreeBSD -- OpenSSL Remote DoS vulnerability CVE: CVE-2016-8610 WWW: https://vuxml.FreeBSD.org/freebsd/0fcd3af0-a0fe-11e6-b1cf-14dae9d210b8.html FreeBSD-10.2 is vulnerable: FreeBSD -- Insecure default snmpd.config permissions CVE: CVE-2015-5677 WWW: https://vuxml.FreeBSD.org/freebsd/7a31dfba-600a-11e6-a6c3-14dae9d210b8.html FreeBSD-10.2 is vulnerable: FreeBSD -- Multiple OpenSSL vulnerabilities CVE: CVE-2016-0800 CVE: CVE-2016-0799 CVE: CVE-2016-0798 CVE: CVE-2016-0797 CVE: CVE-2016-0705 CVE: CVE-2016-0704 CVE: CVE-2016-0703 CVE: CVE-2016-0702 WWW: https://vuxml.FreeBSD.org/freebsd/7b1a4a27-600a-11e6-a6c3-14dae9d210b8.html FreeBSD-10.2 is vulnerable: OpenSSH -- PAM vulnerabilities CVE: CVE-2015-6565 CVE: CVE-2015-6564 CVE: CVE-2015-6563 WWW: https://vuxml.FreeBSD.org/freebsd/2920c449-4850-11e5-825f-c80aa9043978.html FreeBSD-10.2 is vulnerable: ntp -- denial of service vulnerability CVE: CVE-2015-5300 WWW: https://vuxml.FreeBSD.org/freebsd/4eae4f46-b5ce-11e5-8a2b-d050996490d0.html FreeBSD-10.2 is vulnerable: openssh -- information disclosure CVE: CVE-2016-0778 CVE: CVE-2016-0777 WWW: https://vuxml.FreeBSD.org/freebsd/dfe0cdc1-baf2-11e5-863a-b499baebfeaf.html FreeBSD-10.2 is vulnerable: FreeBSD -- Multiple ntp vulnerabilities CVE: CVE-2016-4957 CVE: CVE-2016-4956 CVE: CVE-2016-4955 CVE: CVE-2016-4954 CVE: CVE-2016-4953 WWW: https://vuxml.FreeBSD.org/freebsd/7cfcea05-600a-11e6-a6c3-14dae9d210b8.html 1 problem(s) in the installed packages found. [dan@knew:~] $
Where is this coming from?
At one time FreeBSD 10.2 was installed on this server, but not now.
[dan@slocum:~] $ uname -a FreeBSD slocum.int.unixathome.org 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC amd64 [dan@slocum:~] $ freebsd-version -u -k 12.0-RELEASE-p3 12.0-RELEASE-p3 [dan@slocum:~] $
Why is there a difference between nrpe and command line? I have no idea.
How I solved it
I took a copy of /usr/local/etc/periodic/security/405.pkg-base-audit and started adding more and more debugging message to it. I quickly discovered that this line:
usrlv=$(freebsd-version -u | sed 's,^,FreeBSD-,;s,-RELEASE-p,_,;s,-RELEASE$,,')
was getting 10.2-RELEASE but I did know how/why.
Next step, where is this freebsd-version?
[dan@slocum:] $ which freebsd-version /bin/freebsd-version
I took a copy of that and start adding debugging output. I altered my hacked version of 405.pkg-base-audit to use my copy of freebsd-version.
I started off with: what option it is getting?
[dan@slocum:~/bin] $ diff -ruN /bin/freebsd-version ~/bin/freebsd-version --- /bin/freebsd-version 2019-03-22 21:58:21.808382000 +0000 +++ /usr/home/dan/bin/freebsd-version 2019-03-22 22:10:31.096101000 +0000 @@ -119,25 +119,21 @@ # default is -u if [ $((opt_k + opt_r + opt_u)) -eq 0 ] ; then - logger -t dvl u opt_u=1 fi # print installed kernel version if [ $opt_k ] ; then - logger -t dvl k kernel_version fi # print running kernel version if [ $opt_r ] ; then - logger -t dvl r running_version fi # print userland version if [ $opt_u ] ; then - logger -t dvl u userland_version fi }
Running the command, I saw:
[dan@slocum:~] $ /usr/local/libexec/nagios/check_nrpe3 -H slocum.int.unixathome.org -c check_base_audit /var/spool/nagios/bin/405.pkg-base-audit started /usr/home/dan/bin/freebsd-version 10.2-RELEASE freebsd-version 10.2-RELEASE mark 1 mark 2 ...
Looking in /var/log/messages, I saw:
Mar 22 21:59:08 slocum dvl[9315]: u
Confirmed, the code is invoking userland_version(). This short function merely outputs $USERLAND_VERSION which is a hardcoded version.
Immediately I figured: I have a rogue copy of freebsd-version sitting around from FreeBSD 10.2
Using the locate command, I found it: /usr/bin/freebsd-version
Look what I found inside:
[dan@slocum:~] $ grep USERLAND /usr/bin/freebsd-version USERLAND_VERSION="10.2-RELEASE" echo $USERLAND_VERSION [dan@slocum:~] $
I moved that file to ~/tmp/ and reran my script:
[dan@knew:~] $ /usr/local/libexec/nagios/check_nrpe3 -H slocum.int.unixathome.org -c check_base_audit Checking for security vulnerabilities in base (userland & kernel): Database fetched: Fri Mar 22 19:42:56 UTC 2019 0 problem(s) in the installed packages found. 0 problem(s) in the installed packages found. [dan@knew:~] $
Wow. This problem stumped me for several hours.
Full paths or not full paths?
Should shell scripts use full patches to the commands they are invoking?
I see both sides of the argument.
- No, so that what happened to be can be done by design.
- Yes, so that what happened to me never happens to anyone.
To answer the real question: where did that file come from?
I have no idea, but this sounds vaguely familiar, but I have no memory of doing anything like this. Searching my blog found no references. I searched my other hosts and found nothing similar.
Looking at the date on the file, it is from 2015:
[dan@slocum:~] $ ls -l ~/tmp/freebsd-version -r-xr-xr-x 1 root wheel 3196 Aug 12 2015 /usr/home/dan/tmp/freebsd-version
I have nobody to blame but myself.
Oh the irony of this morning:
This morning I thought about two debugging steps which might have helped: