This morning, I was reviewing the logs and noticed these notices in /var/log/messages:
Aug 16 21:51:38 knew devd: Executing 'logger -p kern.notice -t ZFS 'vdev state changed, pool_guid=15378250086669402288 vdev_guid=15077920823230281604'' Aug 16 21:51:38 knew ZFS: vdev state changed, pool_guid=15378250086669402288 vdev_guid=15077920823230281604
I guessed that this was directly related to the zfs replace command I had issued the day before, but I wanted to confirm that.
I like to use sudo instead of becoming root. One of the many benefits to this approach is logging. Everything is logged to /var/log/auth.log for later review.
However, by the time I went to look, /var/log/auth.log had rotated off this server:
[dan@knew:~] $ ls -l /var/log/auth.log* -rw-r----- 1 root logcheck 49352 Aug 17 12:00 /var/log/auth.log -rw-r----- 1 root logcheck 2943 Aug 17 11:00 /var/log/auth.log.0.bz2 -rw-r----- 1 root logcheck 2849 Aug 17 08:00 /var/log/auth.log.1.bz2 -rw-r----- 1 root logcheck 2872 Aug 17 05:00 /var/log/auth.log.2.bz2 -rw-r----- 1 root logcheck 2774 Aug 17 02:00 /var/log/auth.log.3.bz2 -rw-r----- 1 root logcheck 3553 Aug 16 23:00 /var/log/auth.log.4.bz2 -rw-r----- 1 root logcheck 3814 Aug 16 21:00 /var/log/auth.log.5.bz2 -rw-r----- 1 root logcheck 2783 Aug 16 17:00 /var/log/auth.log.6.bz2 [dan@knew:~] $
I’ll come back to that rotation issue later in this post.
I restored those files from last night’s backups. Ironically enough, those backups are stored on this very same server. Yes, this is the backup server. Backups are also copied to tape from this server.
That restore didn’t take very long, and here is what I found:
[root@knew:/tmp/bacula-restores/var/log] # ls -l total 76 -rw-r----- 1 root logcheck 56995 Aug 17 03:13 auth.log -rw-r----- 1 root logcheck 2774 Aug 17 02:00 auth.log.0.bz2 -rw-r----- 1 root logcheck 3553 Aug 16 23:00 auth.log.1.bz2 -rw-r----- 1 root logcheck 3814 Aug 16 21:00 auth.log.2.bz2 -rw-r----- 1 root logcheck 2783 Aug 16 17:00 auth.log.3.bz2 -rw-r----- 1 root logcheck 2851 Aug 16 14:00 auth.log.4.bz2 -rw-r----- 1 root logcheck 2832 Aug 16 11:00 auth.log.5.bz2 -rw-r----- 1 root logcheck 2802 Aug 16 08:00 auth.log.6.bz2 [root@knew:/tmp/bacula-restores/var/log] #
A quick grep, and I had my confirmation:
# bzgrep 21:51:38 auth.log.1.bz2 Aug 16 21:51:38 knew sudo: dan : TTY=pts/0 ; PWD=/usr/home/dan ; USER=root ; COMMAND=/sbin/zpool replace system da2p3 da20p3
Yes, confirmed. Those messages were related to the replace.
about newsyslog.conf
One of my pet peeves about /etc/newsyslog.conf is the frequency with which logs can rotate off the server. I like to keep at least a few days of logs on the server for easy review. Yes, they are available from backups, but that’s more work. Disk space is cheap, and time is not.
Fortunately, the fix is simple: adjust /etc/newsyslog.conf to keep more days, not based on size.
This is my new entry for /var/log/auth.log:
# logfilename [owner:group] mode count size when flags [/pid_file] [sig_num] /var/log/auth.log 640 15 * @T00 JC
This will keep 15 days of logs, regardless of size, and will rotate nightly at midnight.
NOTE: when you change the count and size fields, do not overlook the when field. I did that, and was confused.
