I started trying a Root-on-ZFS upgrade using beadm and quickly encountered an anomaly: no /usr/bin
Then, it was there, then it wasn’t. I wasn’t sure of what steps I took to reproduce it. So I went through logs and eventually understood what happened. Let me show you.
Here is what I did:
# beadm create 10.1-RELEASE # beadm mount 10.1-RELEASE /mnt # mount -t devfs devfs /mnt/dev # chroot /mnt csh: Cannot open /etc/termcap. csh: using dumb terminal settings. # bash Shared object "libintl.so.8" not found, required by "bash" #
That got me interested, so I tried:
# which bash /usr/local/bin/bash # file /usr/local/bin/bash file: Command not found. #
No file?
# ls /usr/bin ls: /usr/bin: No such file or directory # ls /usr local obj src #
Now I’m just confused. This was based on a running system. Where is the rest of /usr?
Eventually, I found the contents of /usr, but it took a while to reproduce what I stumbled upon.
# exit exit # ls /mnt/usr local obj src # beadm umount 10.1-RELEASE ERROR: Cannot umount 'system/bootenv/10.1-RELEASE' dataset
That failed because of the mount at /mnt/dev but look what I found:
# ls /mnt/usr bin home jails lib32 libexec obj sbin src games include lib libdata local ports share #
Now, there’s what you’d expect to see in /usr/ but why didn’t I see it before?
What’s mounted?
# mount | grep /mnt system/bootenv/10.1-RELEASE on /mnt (zfs, local, noatime, nfsv4acls)
Oh. Now, that’s interesting.
Let’s clear things up and look at what we get:
# umount /mnt/dev # beadm umount 10.1-RELEASE Unmounted successfully # beadm mount 10.1-RELEASE /mnt Mounted successfully on '/mnt' # zfs mount | grep /mnt system/bootenv/10.1-RELEASE /mnt system/bootenv/10.1-RELEASE/tmp /mnt/tmp system/bootenv/10.1-RELEASE/usr /mnt/usr system/bootenv/10.1-RELEASE/usr/local /mnt/usr/local system/bootenv/10.1-RELEASE/usr/obj /mnt/usr/obj system/bootenv/10.1-RELEASE/usr/src /mnt/usr/src system/bootenv/10.1-RELEASE/var /mnt/var system/bootenv/10.1-RELEASE/var/audit /mnt/var/audit system/bootenv/10.1-RELEASE/var/empty /mnt/var/empty system/bootenv/10.1-RELEASE/var/log /mnt/var/log system/bootenv/10.1-RELEASE/var/tmp /mnt/var/tmp
My theory: the contents of system/bootenv/10.1-RELEASE/usr is being over written by some other mount.
Let’s see what is mounted from the default BE:
# zfs mount | grep system/bootenv/default system/bootenv/default / system/bootenv/default/tmp /tmp system/bootenv/default/usr/local /usr/local system/bootenv/default/usr/obj /usr/obj system/bootenv/default/usr/src /usr/src system/bootenv/default/var/audit /var/audit system/bootenv/default/var/empty /var/empty system/bootenv/default/var/log /var/log system/bootenv/default/var/tmp /var/tmp
Ahh, there is nothing mounted at /usr from system/bootenv/default like there is at /mnt for system/bootenv/10.1-RELEASE.
If I remove that mount, everything looks normal:
# zfs umount system/bootenv/10.1-RELEASE/usr # ls -l /mnt/usr total 296 drwxr-xr-x 2 root wheel 477 Feb 25 21:52 bin drwxr-xr-x 2 root wheel 2 Dec 4 2012 games drwxr-xr-x 4 root wheel 5 Oct 3 12:28 home drwxr-xr-x 53 root wheel 283 Aug 11 2014 include drwxr-xr-x 39 root wheel 39 Mar 9 00:38 jails drwxr-xr-x 8 root wheel 558 Feb 25 21:52 lib drwxr-xr-x 5 root wheel 5 Aug 11 2014 lib32 drwxr-xr-x 5 root wheel 5 Dec 4 2012 libdata drwxr-xr-x 7 root wheel 62 Jan 16 19:34 libexec drwxr-xr-x 3 root wheel 3 Sep 10 2013 local drwxr-xr-x 2 root wheel 2 Mar 8 23:56 obj drwxr-xr-x 2 root wheel 2 May 3 2013 ports drwxr-xr-x 2 root wheel 295 Feb 25 21:52 sbin drwxr-xr-x 30 root wheel 30 May 27 2014 share drwxr-xr-x 2 root wheel 2 Mar 8 23:56 src
My questions are:
- Why does beadm mount system/bootenv/10.1-RELEASE/usr?
- Why isn’t system/bootenv/default/usr mounted?
Can they be mounted?
# zfs get canmount system/bootenv/default/usr \
system/bootenv/10.1-RELEASE/usr
NAME PROPERTY VALUE SOURCE
system/bootenv/10.1-RELEASE/usr canmount off local
system/bootenv/default/usr canmount off local
No, but.. they are.
I raised an issue.
Then I started wondering if there was anything useful in that location, so I checked:
$ sudo zfs mount -o altmnt=/mnt system/bootenv/default/usr cannot mount 'system/bootenv/default/usr': 'canmount' property is set to 'off'
OK, I can’t mount it, so let’s change it so I can:
$ sudo zfs set canmount=auto system/bootenv/default/usr cannot set property for 'system/bootenv/default/usr': 'canmount' must be one of 'on | off | noauto'
Oops, try again:
$ sudo zfs set canmount=noauto system/bootenv/default/usr $ sudo zfs mount -o altmnt=/mnt system/bootenv/default/usr $ sudo ls -l /mnt total 0 $
OK, nothing there. Let’s umount that.
$ sudo zfs umount system/bootenv/default/usr bash: /usr/local/bin/sudo: No such file or directory
Oh oh… Yeah, this wasn’t going anywhere. I couldn’t even login at the console; it failed.
Fortunately, CONTROL-ALT-DELETE allowed me to reboot that server.
I see now… I tried to specify an alternate mountpoint, but that option does not exist, so it was mounted over the existing /usr. Oops.
Trying, safely this time
Let’s try this instead:
$ sudo mount -t zfs system/bootenv/default/usr /mnt $ ls -l /mnt total 33 drwxr-xr-x 2 root wheel 2 Mar 11 15:12 obj drwxr-xr-x 2 root wheel 2 Mar 11 15:12 src $ ls -l /mnt/* /mnt/obj: total 0 /mnt/src: total 0 $
OK, good, there isn’t much in there.
Is our original stuff still OK?
$ ls -l /usr total 296 drwxr-xr-x 2 root wheel 477 Feb 25 21:52 bin drwxr-xr-x 2 root wheel 2 Dec 4 2012 games drwxr-xr-x 4 root wheel 5 Oct 3 12:28 home drwxr-xr-x 53 root wheel 283 Aug 11 2014 include drwxr-xr-x 39 root wheel 39 Mar 9 00:38 jails drwxr-xr-x 8 root wheel 558 Feb 25 21:52 lib drwxr-xr-x 5 root wheel 5 Aug 11 2014 lib32 drwxr-xr-x 5 root wheel 5 Dec 4 2012 libdata drwxr-xr-x 7 root wheel 62 Jan 16 19:34 libexec drwxr-xr-x 19 root wheel 20 Dec 27 20:35 local drwxr-xr-x 3 root wheel 3 Jan 16 21:47 obj drwxr-xr-x 70 root wheel 88 Mar 8 23:57 ports drwxr-xr-x 2 root wheel 295 Feb 25 21:52 sbin drwxr-xr-x 30 root wheel 30 May 27 2014 share drwxr-xr-x 22 root wheel 31 Jan 16 21:46 src
Phew!
I am quite sure I can destroy this dataset without repercussions but I think I’ll just rename it out from under the bootenv just to be sure.
The renaming
My first attempt:
$ sudo zfs rename system/bootenv/default/usr system/zzz-DELETEME.bootenv.default.usr cannot unmount '/usr/local': Device busy $
I think I’ll have to do this in single user mode, or perhaps jump back to my mfsBSD live BSD drive. But that is going to be a different post. Just because.
