Yesterday, I mentioned (in more than one place) that I planned to move a 2017 Digital Ocean droplet over to Azure. As I sit here, with coffee, on the balcony, overlooking lot of green trees, at 7:45 AM, I’m trying to put into words the plan I came up with about 30 minutes ago.
In this post:
- Digital Ocean
- Microsoft Azure
- FreeBSD 14.1
Why move?
There is no technical issue or dissatisfaction involved in my move. It is purely economics and because I can. There are two tidbits though:
- Digital Ocean stopped supplying FreeBSD droplets
- Microsoft Azure is providing free credits to run Azure to FreeBSD committers
From a money point of view, this should save me about $76.32 per year. In terms of time, it’s not really worth it to swap, but if I do swap, I can buy a decent meal with the annual savings.
Before you start
Be aware of a problem I hit. In /boot/loader.conf.local I found:
vfs.root.mountfrom="zfs:zroot/ROOT/default"
This is incompatible with using boot environments (BE). The host will always boot from zroot/ROOT/default, which will override any BE you have selected.
I commented out that line, and BE started to work as expected. I think this file originated with the VM I was copying from. That loader.conf.local issue – it is NOT a problem with the Azure templates – the file originated in the VM I was migrating from DigitalOcean to Azure. Looking at the original droplet, the file is dated 2017
The droplet
Digital Ocean hosts are often referred to as droplets. The droplet I have was created 7 years ago on Mar 23 2017 10:19 pm (I don’t know if that’s UTC, but it could be). I created that droplet to manage my vast and humongous empire from outside my infrastructure. I wanted to monitor it as my glorious public would view it. If it’s down for them, I want to know.
It’s not a huge system, disk-wise:
[dvl@nagios02:~] $ zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 17.9G 8.24G 9.64G - - 71% 46% 1.00x ONLINE -
That’s about 8.5GB – pretty tidy. It has 1GB of RAM and isn’t using all of it (the following is from top):
last pid: 13041; load averages: 0.96, 1.01, 0.81 up 2+21:43:39 11:54:36
35 processes: 2 running, 33 sleeping i
CPU: 0.0% user, 0.0% nice, 0.8% system, 0.0% interrupt, 99.2% idle
Mem: 19M Active, 100M Inact, 441M Wired, 9216B Buf, 394M Free
ARC: 128M Total, 19M MFU, 69M MRU, 1568K Anon, 3001K Header, 31M Other
61M Compressed, 277M Uncompressed, 4.52:1 Ratio
Swap: 2048M Total, 2048M Free
The host responds, but certainly is never as snappy as my Nagios instance in the basement.
It has both IP4 and IPv6 IP addresses, CPU load is at about 4%, disk I/O is about 100KB/s, and it transfers about 4GB/month outwards.
The Azure VM
I will not go into detail regarding the creation of the Azure VM. I will supply a few highlights.
The FreeBSD images aren’t listed on the main page – click onto See all images.
Search for FreeBSD on the next page
On the FreeBSD image you want, click on Select at the bottom of that column.
For what it’s worth, I took the mag image.
During the creation process, I supplied a user name and a public ssh key for access. The deployment took about 10 minutes. I was able to ssh in:
The list of installed packages, fairly reasonable.
zfs – the key to this transfer
I plan to use zfs send/receive to copy the Digital Ocean droplet to this new Azure instance.
Here is the source:
[dvl@nagios02:~] $ zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 7.30G 10.0G 19K none zroot/ROOT 6.14G 10.0G 19K none zroot/ROOT/default 6.14G 10.0G 4.90G legacy zroot/tmp 1.46M 10.0G 39K /tmp zroot/usr 4.80M 10.0G 19K /usr zroot/usr/home 4.72M 10.0G 4.51M /usr/home zroot/usr/ports 28K 10.0G 19K /usr/ports zroot/usr/src 28K 10.0G 19K /usr/src zroot/var 1.06G 10.0G 19K /var zroot/var/audit 19K 10.0G 19K /var/audit zroot/var/crash 19K 10.0G 19K /var/crash zroot/var/log 1.06G 10.0G 1.04G /var/log zroot/var/mail 46.5K 10.0G 35.5K /var/mail zroot/var/tmp 276K 10.0G 256K /var/tmp
This is the destination:
dvl@nagios03:~ $ zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 7.11G 21.5G 424K none zroot/ROOT 7.10G 21.5G 424K none zroot/ROOT/14.1-RELEASE_2024-07-04_123032 8K 21.5G 5.22G / zroot/ROOT/default 7.10G 21.5G 5.21G / zroot/home 472K 21.5G 472K /home zroot/tmp 428K 21.5G 428K /tmp zroot/usr 1.64M 21.5G 424K /usr zroot/usr/obj 420K 21.5G 420K /usr/obj zroot/usr/ports 420K 21.5G 420K /usr/ports zroot/usr/src 420K 21.5G 420K /usr/src zroot/var 2.61M 21.5G 424K /var zroot/var/audit 428K 21.5G 428K /var/audit zroot/var/crash 424K 21.5G 424K /var/crash zroot/var/log 552K 21.5G 552K /var/log zroot/var/mail 416K 21.5G 416K /var/mail zroot/var/tmp 424K 21.5G 424K /var/tmp dvl@nagios03:~ $
Some sanity checks
The two hosts have the same uname -a output. I’m not sure that’ critical to success.
[dvl@nagios02:~] $ uname -a FreeBSD nagios02 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64 dvl@nagios03:~ $ uname -a FreeBSD nagios03 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64
This, binary compatibility, I think is much more important. If I was copying from ARM to x86, I might have a bad time (for example).
[dvl@nagios02:~] $ file /bin/sh /bin/sh: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.1, FreeBSD-style, stripped dvl@nagios03:~ $ file /bin/sh /bin/sh: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.1, FreeBSD-style, stripped
Silence motd
While a fine service, I do not use motd. This is now I silence that:
dvl@nagios03:~ $ sudo service motd disable motd disabled in /etc/rc.conf dvl@nagios03:~ $ sudo rm /var/run/motd dvl@nagios03:~ $
What’s next?
I’m going to take the dog for a walk and think about this a bit more. I know I can see into the new host from the old host. I will use that for zfs send | recv after enabling root ssh login. That enabling is not something I recommend, except in specific circumstances:
- ssh key login only
- ssh root login is disabled after use
First, I need a root password, for use from the console.
dvl@nagios03:~ $ sudo passwd root Changing local password for root New Password: Retype New Password: dvl@nagios03:~ $
I’ll also have to setup ssh-keys for root. I’m not showing that process here, but now I can ssh in as root:
[9:45 air01 dan ~] % ssh root@[redacted] root@nagios03:~ #
The man thing I must do though is ensure I have console access. I know this won’t work the first time and I will lock myself out.
However, I can’t see how to get console access yet. Let’s ask on Twitter and Mastodon.
A short while later
Winner! Winner! Chicken dinner!
The answer, in the above thread, is enabling Boot diagnostics for the VM and this entry in /boot/loader.conf:
console="vidconsole comconsole"
Create a new BE
Let’s create a new Boot Environment (BE).
dvl@nagios03:~ $ sudo bectl create before-send-recv dvl@nagios03:~ $ bectl list BE Active Mountpoint Space Created before-send-recv - - 208K 2024-07-04 22:23 default NR / 5.22G 1970-01-01 00:00 dvl@nagios03:~ $ zfs list -r -t snapshot NAME USED AVAIL REFER MOUNTPOINT zroot/ROOT/default@2024-07-04-22:23:19-0 200K - 5.22G - dvl@nagios03:~ $ zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 5.23G 23.3G 424K none zroot/ROOT 5.22G 23.3G 424K none zroot/ROOT/before-send-recv 8K 23.3G 5.22G / zroot/ROOT/default 5.22G 23.3G 5.22G / zroot/home 472K 23.3G 472K /home zroot/tmp 420K 23.3G 420K /tmp zroot/usr 1.64M 23.3G 424K /usr zroot/usr/obj 420K 23.3G 420K /usr/obj zroot/usr/ports 420K 23.3G 420K /usr/ports zroot/usr/src 420K 23.3G 420K /usr/src zroot/var 2.75M 23.3G 424K /var zroot/var/audit 428K 23.3G 428K /var/audit zroot/var/crash 424K 23.3G 424K /var/crash zroot/var/log 700K 23.3G 700K /var/log zroot/var/mail 416K 23.3G 416K /var/mail zroot/var/tmp 420K 23.3G 420K /var/tmp
What am I sending?
Let’s look at sending this BE:
[dvl@nagios02:~] $ bectl list BE Active Mountpoint Space Created default NR / 6.16G 2017-04-10 23:39 [dvl@nagios02:~] $ sudo bectl create for-sending [dvl@nagios02:~] $ bectl list BE Active Mountpoint Space Created default NR / 6.16G 2017-04-10 23:39 for-sending - - 87K 2024-07-04 22:26 [dvl@nagios02:~] $
My first attempt
I’m bad at remembering some commands. I searched for a solution and found Fun with ZFS send and receive
[dvl@nagios02:~] $ sudo zfs send zroot/ROOT/for-sending@autosnap_2024-07-04_22:30:00_daily \ | ssh root@20.94.49.41 zfs receive zroot/ROOT/from-nagios02@autosnap_2024-07-04_22:30:00_daily
That took about two minutes.
Let’s see what we have:
dvl@nagios03:~ $ zfs list -r zroot/ROOT NAME USED AVAIL REFER MOUNTPOINT zroot/ROOT 11.6G 16.9G 424K none zroot/ROOT/before-send-recv 8K 16.9G 5.22G / zroot/ROOT/default 5.22G 16.9G 5.22G / zroot/ROOT/from-nagios02 6.42G 16.9G 6.42G none dvl@nagios03:~ $ bectl list BE Active Mountpoint Space Created before-send-recv - - 344K 2024-07-04 22:23 default NR / 5.22G 1970-01-01 00:00 from-nagios02 - - 6.42G 2024-07-04 22:34 dvl@nagios03:~ $
I am positive that if I switched to that BE and rebooted, it would not come up. I suspect I have to modify at least these files:
- /etc/rc.conf
- /boot/loader.conf
Updating the received files
Let’s mount the stuff over in /mnt.
NOTE: Later, I realized I could have used bectl mount instead.
dvl@nagios03:~ $ sudo zfs mount -o mountpoint=/mnt zroot/ROOT/from-nagios02 cannot mount 'zroot/ROOT/from-nagios02': no mountpoint set dvl@nagios03:~ $
Fail.
Let’s try this instead:
dvl@nagios03:~ $ sudo zfs set canmount=noauto zroot/ROOT/from-nagios02 dvl@nagios03:~ $ sudo zfs set mountpoint=/ zroot/ROOT/from-nagios02 dvl@nagios03:~ $ sudo zfs mount -o mountpoint=/mnt zroot/ROOT/from-nagios02 dvl@nagios03:~ $
OK, let’s get in there and edit.
But first, what about backups? We have that, in the form of snapshots – we can get back to the original version from the snapshots.
dvl@nagios03:~ $ sudoedit /etc/rc.conf /mnt/etc/rc.conf sudoedit: no writable temporary directory found
OH. What’s going on?
dvl@nagios03:~ $ mount zroot/ROOT/default on / (zfs, local, nfsv4acls) devfs on /dev (devfs) /dev/gpt/efiesp on /boot/efi (msdosfs, local) zroot/home on /home (zfs, local, nfsv4acls) zroot/tmp on /tmp (zfs, local, nosuid, nfsv4acls) zroot/var/log on /var/log (zfs, local, noexec, nosuid, nfsv4acls) zroot/var/crash on /var/crash (zfs, local, noexec, nosuid, nfsv4acls) zroot/usr/obj on /usr/obj (zfs, local, nfsv4acls) zroot/usr/src on /usr/src (zfs, local, nfsv4acls) zroot/usr/ports on /usr/ports (zfs, local, nosuid, nfsv4acls) zroot/var/audit on /var/audit (zfs, local, noexec, nosuid, nfsv4acls) zroot/var/mail on /var/mail (zfs, local, nfsv4acls) zroot/var/tmp on /var/tmp (zfs, local, nosuid, nfsv4acls) /dev/da1p1 on /mnt/resource (ufs, local, soft-updates) zroot/ROOT/from-nagios02 on / (zfs, local, nfsv4acls)
Hmm, I think I’m all messed up now and have to reboot.
dvl@nagios03:~ $ sudo zfs umount zroot/ROOT/from-nagios02 No such file or directory
Yes, I’ done for I think. I messed up on that mount, despite specifying a mount point via mountpoint=/mnt
After a couple of reboots, I came up with this:
dvl@nagios03:~ $ sudo zfs set mountpoint=/mnt/from-nagios zroot/ROOT/from-nagios02 dvl@nagios03:~ $ sudo zfs mount zroot/ROOT/from-nagios02
Now I can edit: dvl@nagios03:~ $ sudoedit /etc/rc.conf /mnt/from-nagios/etc/rc.conf
After making my changes, I tried this:
dvl@nagios03:~ $ sudo bectl activate -t from-nagios02 Successfully activated boot environment from-nagios02 for next boot dvl@nagios03:~ $ sudo zfs umount zroot/ROOT/from-nagios02 dvl@nagios03:~ $ sudo zfs set -u mountpoint=/ zroot/ROOT/from-nagios02 dvl@nagios03:~ $
After rebooting, I didn’t have what I expected:
dvl@nagios03:~ $ bectl list BE Active Mountpoint Space Created before-send-recv - - 1.82M 2024-07-04 22:23 default NR / 5.22G 1970-01-01 00:00 from-nagios02 - - 6.43G 2024-07-04 22:34 dvl@nagios03:~ $ uptime 11:05PM up 50 secs, 1 user, load averages: 0.33, 0.09, 0.03 dvl@nagios03:~ $
That’s not the BE I expected. My theory: it booted, panicked, rebooted, and here I am.
I forgot about /boot/loader.conf and I updated that next. You’ll have to use your own judgement about want needs to be copied over.
I repeated the process from earlier and modified that /boot/loader.conf. I did the bectl activate and did a shutdown -r now – still came up in the old partition:
dvl@nagios03:~ $ bectl list BE Active Mountpoint Space Created before-send-recv - - 1.82M 2024-07-04 22:23 default NR / 5.22G 1970-01-01 00:00 from-nagios02 - - 6.43G 2024-07-04 22:34 dvl@nagios03:~ $
NOTE: this was a problem with /boot/loader.conf.local, It contained:
vfs.root.mountfrom="zfs:zroot/ROOT/default"
I commented out that line, and BE started to work as expected. That loader.conf.local issue – it is NOT a problem with the Azure templates – the file originated in the VM I was migrating from DigitalOcean to Azure. Looking at the original droplet, the file is dated 2017
Sorry about the false alarm.
Current status
It’s booting using the new /etc/rc.conf…
What I’m missing:
- /usr/local
- /var/db
I know this because I don’t have /usr/local/www and pkg info doesn’t list the expected packages.
Copying those files
On the old host, I did this:
[dvl@nagios02:/usr/local] $ sudo tar -czf ~dvl/usr.local.tar.gz /usr/local tar: Removing leading '/' from member names [dvl@nagios02:/usr/local] $ sudo tar -czf ~dvl/var.tar.gz /var/net-snmp /var/db/mysql /var/db/pkg tar: Removing leading '/' from member names
Those archives were copied to the new host and unzipped after taking snapshots:
dvl@nagios03:/ $ sudo zfs snapshot zroot/usr@before.nagios02 dvl@nagios03:/ $ sudo zfs snapshot zroot/var@before.nagios02 dvl@nagios03:/ $ sudo tar -xzf ~/usr.local.tar.gz dvl@nagios03:/ $ sudo tar -xzf ~/var.tar.gz
I reset the BE and restarted the host:
dvl@nagios03:/ $ sudo bectl activate from-nagios02 Successfully activated boot environment from-nagios02 dvl@nagios03:/ $ bectl list BE Active Mountpoint Space Created before-send-recv - - 268M 2024-07-04 22:23 default N / 6.57G 1970-01-01 00:00 from-nagios02 R - 6.43G 2024-07-04 22:34 dvl@nagios03:/ $ shutdown -r now -sh: shutdown: Permission denied dvl@nagios03:/ $ sudo shutdown -r now Shutdown NOW! shutdown: [pid 1434] dvl@nagios03:/ $ *** FINAL System shutdown message from dvl@nagios03 *** System going down IMMEDIATELY
Looking good
After rebooting, I have the expected packages installed:
dvl@nagios03:~ $ pkg info SamDruckerClientShell-0.2.6 Client for sending package information to SamDrucker abseil-20230125.3 Abseil Common Libraries (C++) anvil-0.0.20 Tools for distributing ssl certificates apache24-2.4.60 Version 2.4.x of Apache web server apr-1.7.3.1.6.3_1 Apache Portability Library bash-5.2.26_1 GNU Project's Bourne Again SHell bind-tools-9.18.27_1 Command line tools from BIND: delv, dig, host, nslookup... brotli-1.1.0,1 Generic-purpose lossless compression algorithm curl-8.8.0 Command line tool and library for transferring data with URLs cyrus-sasl-2.1.28_4 RFC 2222 SASL (Simple Authentication and Security Layer) easy-rsa-3.1.7,1 Small RSA key management package based on openssl expat-2.6.2 XML 1.0 parser written in C fontconfig-2.15.0_2,1 XML-based font configuration API for X Windows freetype2-2.13.2 Free and portable TrueType font rendering engine fstrm-0.6.1_1 Implementation of the Frame Streams data transport protocol in C gdbm-1.23 GNU database manager gettext-runtime-0.22.5 GNU gettext runtime libraries and programs giflib-5.2.2 Tools and library routines for working with GIF images gmp-6.3.0 Free library for arbitrary precision arithmetic gnupg-2.4.5 Complete and free PGP implementation gnutls-3.8.5_3 GNU Transport Layer Security library groff-1.23.0_3 Software typesetting package hidapi-0.14.0 Library to access USB HID-class devices icu-74.2_1,1 International Components for Unicode (from IBM) indexinfo-0.3.1 Utility to regenerate the GNU info page index jansson-2.14 C library for encoding, decoding, and manipulating JSON data jbigkit-2.1_3 Lossless compression for bi-level images such as scanned pages, faxes jo-1.6_1 Small utility to create JSON objects joe-4.6_1,1 Joe's Own Editor jpeg-turbo-3.0.3 SIMD-accelerated JPEG codec which replaces libjpeg json-c-0.17 JSON (JavaScript Object Notation) implementation in C jsoncpp-1.9.5 JSON reader and writer library for C++ lerc-4.0.0 C++ library for Limited Error Raster Compression libargon2-20190702_1 Memory hard password hashing program and library libassuan-2.5.7 IPC library used by GnuPG and gpgme libcbor-0.11.0 CBOR protocol implementation for C and others libcjson-1.7.18_2 Ultralightweight JSON parser in ANSI C libdeflate-1.20 Fast, whole-buffer DEFLATE-based compression library libedit-3.1.20240517,1 Command line editor library libevent-2.1.12 API for executing callback functions on events or timeouts libffi-3.4.6 Foreign Function Interface libfido2-1.15.0 Provides library functionality for FIDO 2.0 libgcrypt-1.11.0 General purpose cryptographic library based on the code from GnuPG libgd-2.3.3_13,1 Graphics library for fast creation of images libgpg-error-1.50 Common error values for all GnuPG components libiconv-1.17_1 Character set conversion library libidn2-2.3.7 Implementation of IDNA2008 internationalized domain names libksba-1.6.6 Library to make X.509 certificates liblockfile-1.17_1 Standard lockfile library libltdl-2.4.7 System independent dlopen wrapper liblz4-1.9.4_1,1 LZ4 compression library, lossless and very fast libnghttp2-1.62.1 HTTP/2.0 C Library libpaper-1.1.28_1 Library providing routines for paper size management libpsl-0.21.5_1 C library to handle the Public Suffix List libssh2-1.11.0_1,3 Library implementing the SSH2 protocol libtasn1-4.19.0_1 ASN.1 structure parser library libunistring-1.2 Unicode string library libunwind-20240221 Generic stack unwinding library libuv-1.48.0 Multi-platform support library with a focus on asynchronous I/O libxml2-2.11.8 XML parser library for GNOME lockfile-progs-0.1.19_1 Programs for locking and unlocking files and mailboxes logcheck-1.4.3_2 Auditing tool for system logs on Unix boxes lzo2-2.10_1 Portable speedy, lossless data compression library lzop-1.04_1 Fast file compressor similar to gzip, using the LZO library mbuffer-20240107 Tool for buffering data streams mhash-0.9.9.9_6 Easy-to-use library for strong hashes such as MD5 and SHA1 mime-construct-1.11_2 Construct and optionally send MIME messages from command line mod_php83-8.3.8 PHP Scripting Language (8.3.X branch) mpdecimal-4.0.0 C/C++ arbitrary precision decimal floating point libraries mtr-nox11-0.95_2 Traceroute and ping in a single network diagnostic tool mysql80-client-8.0.35 Multithreaded SQL database (client) nagios-3.5.1_12 Powerful network monitoring system nagios-plugins-2.4.4,1 Plugins for Nagios ncdu-1.20 NCurses Disk Usage (LTS version written in C) ncurses-6.5 Library for terminal-independent, full-screen output net-snmp-5.9.4_5,1 Extendable SNMP implementation nettle-3.10_1 Low-level cryptographic library nmap-7.94_3 Port scanning utility for large networks npth-1.7 New GNU Portable Threads nrpe-4.1.0 Nagios Remote Plugin Executor openldap26-client-2.6.8 Open source LDAP client implementation openvpn-2.6.11 Secure IP/Ethernet tunnel daemon p11-kit-0.25.3_2 Library for loading and enumerating of PKCS#11 modules p5-Capture-Tiny-0.48 Capture STDOUT and STDERR from Perl, XS, or external programs p5-Config-IniFiles-3.000003_1 Read .ini-style configuration files p5-File-ReadBackwards-1.06 Perl module to read file backwards by lines p5-File-Slurp-9999.27 Perl module for single call read and write file routines p5-IO-Stringy-2.113 Use IO handles with non-file objects p5-IPC-Signal-1.00_1 Utility functions to deal with signals in Perl p5-JSON-4.10 Perl extension to convert to JSON (JavaScript Object Notation) p5-MIME-Base64-3.16 Perl5 module for Base64 and Quoted-Printable encodings p5-MIME-Types-2.26 Perl extension for determining MIME types p5-Proc-WaitStat-1.00_1 Interpret and act on wait() status values pam_ssh_agent_auth-0.10.4_5 PAM module which permits authentication via ssh-agent pcre2-10.43 Perl Compatible Regular Expressions library, version 2 perl5-5.38.2_1 Practical Extraction and Report Language php83-8.3.8 PHP Scripting Language (8.3.X branch) pinentry-1.3.0 Collection of simple PIN or passphrase entry dialogs pinentry-curses-1.3.0 Curses version of the GnuPG password dialog pkcs11-helper-1.29.0_3 Helper library for multiple PKCS#11 providers pkg-1.21.3 Package manager png-1.6.43 Library for manipulating PNG images postgresql16-client-16.3 PostgreSQL database (client) protobuf-24.4,1 Data interchange format library protobuf-c-1.4.1_2 Code generator and libraries to use Protocol Buffers from pure C psutils-1.17_6 Utilities for manipulating PostScript documents pv-1.8.10 Pipe throughput monitor python-3.11_3,2 "meta-port" for the default version of Python interpreter python3-3_4 Meta-port for the Python interpreter 3.x python311-3.11.9 Interpreted object-oriented programming language python39-3.9.19 Interpreted object-oriented programming language readline-8.2.10 Library for editing command lines as they are typed rsync-3.3.0 Network file distribution/synchronization utility sanoid-2.2.0 Policy-driven snapshot management and replication tools serf-1.3.10_1 Serf HTTP client library sqlite3-3.46.0,1 SQL database engine in a C library subversion-1.14.3_7 Version control system sudo-1.9.15p5_4 Allow others to run commands as root tiff-4.6.0 Tools and library routines for working with TIFF images tmux-3.3a_3 Terminal Multiplexer uchardet-0.0.8_1 Universal charset detection library utf8proc-2.9.0 UTF-8 processing library webp-1.4.0_1 Google WebP image format conversion tool wget-1.24.5 Retrieve files from the Net via HTTP(S) and FTP xtail-2.1_1 Watches the growth of files or directories xxhash-0.8.2_1 Extremely fast non-cryptographic hash algorithm zstd-1.5.6 Fast real-time compression algorithm
And the expected processes, including apache. Nagios is not yet running. That’ll be next:
dvl@nagios03:~ $ ps auwwx USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 200.0 0.0 0 32 - RNL 01:09 8:40.24 [idle] root 0 0.0 0.0 0 1264 - DLs 01:09 0:00.41 [kernel] root 1 0.0 0.0 11704 1328 - ILs 01:09 0:00.03 /sbin/init root 2 0.0 0.0 0 32 - WL 01:09 0:00.08 [clock] root 3 0.0 0.0 0 48 - DL 01:09 0:00.00 [crypto] root 4 0.0 0.0 0 48 - DL 01:09 0:00.00 [cam] root 5 0.0 0.0 0 688 - DL 01:09 0:00.11 [zfskern] root 6 0.0 0.0 0 16 - DL 01:09 0:00.02 [rand_harvestq] root 7 0.0 0.0 0 48 - DL 01:09 0:00.03 [pagedaemon] root 8 0.0 0.0 0 16 - DL 01:09 0:00.00 [vmdaemon] root 9 0.0 0.0 0 96 - DL 01:09 0:00.01 [bufdaemon] root 10 0.0 0.0 0 16 - DL 01:09 0:00.00 [audit] root 12 0.0 0.0 0 80 - WL 01:09 0:00.83 [intr] root 13 0.0 0.0 0 48 - DL 01:09 0:00.00 [geom] root 14 0.0 0.0 0 16 - DL 01:09 0:00.00 [sequencer 00] root 15 0.0 0.0 0 16 - DL 01:09 0:00.00 [vnlru] root 16 0.0 0.0 0 16 - DL 01:09 0:00.00 [syncer] root 325 0.0 0.0 13160 2544 - Is 01:09 0:00.00 dhclient: system.syslog (dhclient) root 328 0.0 0.0 13160 2688 - Is 01:09 0:00.00 dhclient: hn0 [priv] (dhclient) _dhcp 394 0.0 0.0 13164 2740 - ICs 01:09 0:00.00 dhclient: hn0 (dhclient) root 654 0.0 0.0 12732 2376 - Ss 01:09 0:00.01 /usr/sbin/hv_kvp_daemon root 811 0.0 0.0 14400 3796 - Is 01:09 0:00.00 /sbin/devd root 1013 0.0 0.0 12880 2704 - Is 01:09 0:00.01 /usr/sbin/syslogd -s ntpd 1089 0.0 0.1 23428 8124 - Ss 01:09 0:00.03 /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g root 1146 0.0 0.1 22836 10244 - Is 01:09 0:00.00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd) root 1162 0.0 0.0 12920 2536 - Is 01:09 0:00.00 /usr/sbin/cron -s root 1185 0.0 0.1 23024 10748 - Is 01:10 0:00.02 sshd: dvl [priv] (sshd) root 1186 0.0 0.2 35248 18924 - Ss 01:10 0:00.03 /usr/local/sbin/httpd -DNOHTTPACCEPT www 1204 0.0 0.2 35248 18980 - I 01:10 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT www 1205 0.0 0.2 35248 18980 - S 01:10 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT www 1206 0.0 0.2 35248 18980 - I 01:10 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT www 1207 0.0 0.2 35248 18980 - I 01:10 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT www 1208 0.0 0.2 35248 18980 - I 01:10 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT dvl 1251 0.0 0.1 23024 11064 - S 01:10 0:00.02 sshd: dvl@pts/0 (sshd) root 1304 0.0 0.5 69660 40816 - S 01:10 0:00.83 /usr/local/bin/python3.9 -u /usr/local/sbin/waagent -run-exthandlers root 1172 0.0 0.5 51584 37904 u0- S 01:10 0:00.60 /usr/local/bin/python3.9 /usr/local/sbin/waagent -daemon root 1197 0.0 0.0 12848 2320 u0 Is+ 01:10 0:00.00 /usr/libexec/getty 3wire ttyu0 root 1189 0.0 0.0 12848 2320 v0 Is+ 01:10 0:00.00 /usr/libexec/getty Pc ttyv0 root 1190 0.0 0.0 12848 2320 v1 Is+ 01:10 0:00.00 /usr/libexec/getty Pc ttyv1 root 1191 0.0 0.0 12848 2320 v2 Is+ 01:10 0:00.00 /usr/libexec/getty Pc ttyv2 root 1192 0.0 0.0 12848 2324 v3 Is+ 01:10 0:00.00 /usr/libexec/getty Pc ttyv3 root 1193 0.0 0.0 12848 2324 v4 Is+ 01:10 0:00.00 /usr/libexec/getty Pc ttyv4 root 1194 0.0 0.0 12848 2332 v5 Is+ 01:10 0:00.00 /usr/libexec/getty Pc ttyv5 root 1195 0.0 0.0 12848 2324 v6 Is+ 01:10 0:00.00 /usr/libexec/getty Pc ttyv6 root 1196 0.0 0.0 12848 2320 v7 Is+ 01:10 0:00.00 /usr/libexec/getty Pc ttyv7 dvl 1252 0.0 0.0 13380 3264 0 Ss 01:10 0:00.01 -sh (sh) dvl 1344 0.0 0.0 13456 3116 0 R+ 01:14 0:00.00 ps auwwx dvl@nagios03:~ $
Fixing Nagios
When starting Nagios, I noticed this error:
Checking misc settings...
Error: Unable to write to check_result_path ('/var/spool/nagios/checkresults') - No such file or directory
Then I noticed missing users. These were not full users, more like system users. I ran vipw on both hosts, and copy paste them over. There were some home directory changes: /usr/home/minion became /home/minion
minion:*:1002:1002::0:0:User &:/home/minion:/bin/sh nagios:*:181:181::0:0:Nagios pseudo-user:/var/spool/nagios:/usr/sbin/nologin mysql:*:88:88::0:0:MySQL Daemon:/var/db/mysql:/usr/sbin/nologin anvil:*:217:217::0:0:anvil certificate dropper:/var/db/anvil:/bin/sh logcheck:*:915:915::0:0:Logcheck system account:/var/lib/logcheck:/usr/local/bin/bash openvpn:*:301:301::0:0:OpenVPN pseudo-user:/nonexistent:/usr/sbin/nologin cyrus:*:60:60::0:0:the cyrus mail server:/nonexistent:/usr/sbin/nologin bacula:*:910:910::0:0:Bacula Daemon:/var/db/bacula:/usr/sbin/nologin snmpd:*:344:344::0:0:Net-SNMP Daemon:/nonexistent:/usr/sbin/nologin
Similarly, there were groups to copy over:
minion:*:1002: nagios:*:181:www mysql:*:88: anvil:*:217: logcheck:*:915: openvpn:*:301: cyrus:*:60: bacula:*:910: snmpd:*:344:
After that Nagios started:
dvl@nagios03:/ $ sudo service nagios start Performing sanity check of nagios configuration: OK Starting nagios. dvl@nagios03:/ $
DigitalOcean Dynamic Configuration
After each boot, these lines appear in /etc/rc.conf – I don’t know what places them there.
hostname="" # DigitalOcean Dynamic Configuration lines and the immediate line below it, # are removed each boot. Hostname is set each boot. # DigitalOcean Dynamic Configuration defaultrouter="" # DigitalOcean Dynamic Configuration ifconfig_vtnet0="inet netmask " # DigitalOcean Dynamic Configuration ifconfig_vtnet0_alias0="inet netmask " # DigitalOcean Dynamic Configuration ifconfig_vtnet1="inet netmask " # DigitalOcean Dynamic Configuration ifconfig_vtnet0_ipv6="inet6 prefixlen " # DigitalOcean Dynamic Configuration ipv6_defaultrouter="" # DigitalOcean Dynamic Configuration ipv6_activate_all_interfaces="yes" [dvl@nagios03:~] $
Found it, via grep:
[dvl@nagios03:~] $ sudo grep -r 'DigitalOcean Dynamic Configuration' /usr/local/etc/ /usr/local/etc/rc.d/digitalocean:# DigitalOcean Dynamic Configuration lines and the immediate line below it, /usr/local/etc/rc.d/digitalocean:# DigitalOcean Dynamic Configuration /usr/local/etc/rc.d/digitalocean:# DigitalOcean Dynamic Configuration /usr/local/etc/rc.d/digitalocean:# DigitalOcean Dynamic Configuration /usr/local/etc/rc.d/digitalocean:# DigitalOcean Dynamic Configuration /usr/local/etc/rc.d/digitalocean:# DigitalOcean Dynamic Configuration /usr/local/etc/rc.d/digitalocean:# DigitalOcean Dynamic Configuration /usr/local/etc/rc.d/digitalocean:# DigitalOcean Dynamic Configuration [dvl@nagios03:~] $ sudo rm /usr/local/etc/rc.d/digitalocean [dvl@nagios03:~] $
Wrapping it up
I did remember to go back and disable root logins. :)
I had to do some other things, like create a certificate specific to the new hostname, adjust firewall rules, but most of that was minor.
I will keep the old VM around for a month or so, just in case.
