Sometimes stuff gets swapped out. When it does, it’s good to know what is swapped. I was getting this Nagios alert this morning.
SWAP CRITICAL - 32% free (1302 MB out of 4095 MB) [1302 (32%)]
I started searching. I found this post about showing what is using swap via:
[dan@knew:~] $ ps ax | awk 'NR==1{print};$3 ~ /W/'
PID TT STAT TIME COMMAND
12 - WL 54:27.37 [intr]
10147 - IWsJ 0:00.00 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/usr/local/etc/mysql/my.cnf --based
12791 - IWsJ 0:00.00 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql -
15305 - IWsJ 0:00.00 nginx: master process /usr/local/sbin/nginx
15306 - IWJ 0:00.00 nginx: worker process (nginx)
19848 - IWsJ 0:00.00 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/usr/local/etc/mysql/my.cnf --based
12717 3 IWs 0:00.00 -sh (sh)
12718 3 IW 0:00.00 bash
12722 3 IW 0:00.00 su
12723 3 IW 0:00.00 _su (csh)
96806 0 IWs 0:00.00 -sh (sh)
96915 1 IWsJ 0:00.00 -sh (sh)
96916 1 IWJ 0:00.00 bash
4558 2 IWsJ 0:00.00 -sh (sh)
4560 2 IWJ 0:00.00 bash
[dan@knew:~] $
It does not tell me what jails.
I started searching for displaying jail information via ps. I found this post:
ps -o pid,jid -awux
Let’s combine the two.
In the first command, NR==1{print}; says to always print the first row (so we see the column headers. $3 ~ /W/ says that if the third column contains a W, print it.
The 3rd column for the first command is STAT. In our second command, that column is number column 9. Hence, our new combined command is:
[dan@knew:~] $ ps -o pid,jid -awux | awk 'NR==1{print};$9 ~ /W/'
PID JID USER %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
12 0 root 0.8 0.0 0 704 - WL Thu08PM 54:29.29 [intr]
10147 8 mysql 0.0 0.0 17096 0 - IWsJ - 0:00.00 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/usr/loc
12791 9 mysql 0.0 0.0 17096 0 - IWsJ - 0:00.00 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/
15305 11 root 0.0 0.0 29348 0 - IWsJ - 0:00.00 nginx: master process /usr/local/sbin/nginx
15306 11 www 0.0 0.0 29348 0 - IWJ - 0:00.00 nginx: worker process (nginx)
19848 15 mysql 0.0 0.0 17096 0 - IWsJ - 0:00.00 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/usr/loc
12717 0 dan 0.0 0.0 17100 0 3 IWs - 0:00.00 -sh (sh)
12718 0 dan 0.0 0.0 17892 0 3 IW - 0:00.00 bash
12722 0 root 0.0 0.0 47748 0 3 IW - 0:00.00 su
12723 0 root 0.0 0.0 23604 0 3 IW - 0:00.00 _su (csh)
96806 0 dan 0.0 0.0 17100 0 0 IWs - 0:00.00 -sh (sh)
96915 15 dan 0.0 0.0 17096 0 1 IWsJ - 0:00.00 -sh (sh)
96916 15 dan 0.0 0.0 17876 0 1 IWJ - 0:00.00 bash
4558 15 dan 0.0 0.0 17096 0 2 IWsJ - 0:00.00 -sh (sh)
4560 15 dan 0.0 0.0 17876 0 2 IWJ - 0:00.00 bash
[dan@knew:~] $
Success.
So, what now?
Once stuff has been swapped out, it’s hard to get it swapped back in. For me, these are not critical task. Just annoying. I take the easy & lazy way out. I restart the jails in question.
You can also run swapoff to remove the devices from swap. That forces the stuff back into RAM.
After restarting my jails, I was at:
$ swapinfo Device 1K-blocks Used Avail Capacity /dev/mirror/swap 4194300 360300 3834000 9%
Yep, this is a dirty approach.
