Aug 172017

This morning, I was reviewing the logs and noticed these notices in /var/log/messages:

Aug 16 21:51:38 knew devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=15378250086669402288 vdev_guid=15077920823230281604''
Aug 16 21:51:38 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=15077920823230281604

I guessed that this was directly related to the zfs replace command I had issued the day before, but I wanted to confirm that.

I like to use sudo instead of becoming root. One of the many benefits to this approach is logging. Everything is logged to /var/log/auth.log for later review.

However, by the time I went to look, /var/log/auth.log had rotated off this server:

[dan@knew:~] $ ls -l /var/log/auth.log*
-rw-r-----  1 root  logcheck  49352 Aug 17 12:00 /var/log/auth.log
-rw-r-----  1 root  logcheck   2943 Aug 17 11:00 /var/log/auth.log.0.bz2
-rw-r-----  1 root  logcheck   2849 Aug 17 08:00 /var/log/auth.log.1.bz2
-rw-r-----  1 root  logcheck   2872 Aug 17 05:00 /var/log/auth.log.2.bz2
-rw-r-----  1 root  logcheck   2774 Aug 17 02:00 /var/log/auth.log.3.bz2
-rw-r-----  1 root  logcheck   3553 Aug 16 23:00 /var/log/auth.log.4.bz2
-rw-r-----  1 root  logcheck   3814 Aug 16 21:00 /var/log/auth.log.5.bz2
-rw-r-----  1 root  logcheck   2783 Aug 16 17:00 /var/log/auth.log.6.bz2
[dan@knew:~] $ 

I’ll come back to that rotation issue later in this post.

I restored those files from last night’s backups. Ironically enough, those backups are stored on this very same server. Yes, this is the backup server. Backups are also copied to tape from this server.

That restore didn’t take very long, and here is what I found:

[root@knew:/tmp/bacula-restores/var/log] # ls -l
total 76
-rw-r-----  1 root  logcheck  56995 Aug 17 03:13 auth.log
-rw-r-----  1 root  logcheck   2774 Aug 17 02:00 auth.log.0.bz2
-rw-r-----  1 root  logcheck   3553 Aug 16 23:00 auth.log.1.bz2
-rw-r-----  1 root  logcheck   3814 Aug 16 21:00 auth.log.2.bz2
-rw-r-----  1 root  logcheck   2783 Aug 16 17:00 auth.log.3.bz2
-rw-r-----  1 root  logcheck   2851 Aug 16 14:00 auth.log.4.bz2
-rw-r-----  1 root  logcheck   2832 Aug 16 11:00 auth.log.5.bz2
-rw-r-----  1 root  logcheck   2802 Aug 16 08:00 auth.log.6.bz2
[root@knew:/tmp/bacula-restores/var/log] # 

A quick grep, and I had my confirmation:

# bzgrep 21:51:38 auth.log.1.bz2
Aug 16 21:51:38 knew sudo:      dan : TTY=pts/0 ; PWD=/usr/home/dan ; USER=root ; COMMAND=/sbin/zpool replace system da2p3 da20p3

Yes, confirmed. Those messages were related to the replace.

about newsyslog.conf

One of my pet peeves about /etc/newsyslog.conf is the frequency with which logs can rotate off the server. I like to keep at least a few days of logs on the server for easy review. Yes, they are available from backups, but that’s more work. Disk space is cheap, and time is not.

Fortunately, the fix is simple: adjust /etc/newsyslog.conf to keep more days, not based on size.

This is my new entry for /var/log/auth.log:

# logfilename          [owner:group]    mode count size when  flags [/pid_file] [sig_num]
/var/log/auth.log                       640  15     *   @T00  JC

This will keep 15 days of logs, regardless of size, and will rotate nightly at midnight.

NOTE: when you change the count and size fields, do not overlook the when field. I did that, and was confused.

Website Pin Facebook Twitter Myspace Friendfeed Technorati Digg Google StumbleUpon Premium Responsive