Mar 112015
 

I started trying a Root-on-ZFS upgrade using beadm and quickly encountered an anomaly: no /usr/bin

Then, it was there, then it wasn’t. I wasn’t sure of what steps I took to reproduce it. So I went through logs and eventually understood what happened. Let me show you.

Here is what I did:

# beadm create 10.1-RELEASE
# beadm mount 10.1-RELEASE /mnt
# mount -t devfs devfs /mnt/dev
# chroot /mnt
csh: Cannot open /etc/termcap.
csh: using dumb terminal settings.
# bash 
Shared object "libintl.so.8" not found, required by "bash"
# 

That got me interested, so I tried:

# which bash
/usr/local/bin/bash
# file /usr/local/bin/bash
file: Command not found.
# 

No file?

# ls /usr/bin
ls: /usr/bin: No such file or directory
# ls /usr
local	obj	src
# 

Now I’m just confused. This was based on a running system. Where is the rest of /usr?

Eventually, I found the contents of /usr, but it took a while to reproduce what I stumbled upon.

# exit
exit
# ls /mnt/usr
local obj   src
# beadm umount 10.1-RELEASE
ERROR: Cannot umount 'system/bootenv/10.1-RELEASE' dataset

That failed because of the mount at /mnt/dev but look what I found:

# ls /mnt/usr
bin     home    jails   lib32   libexec obj     sbin    src
games   include lib     libdata local   ports   share
# 

Now, there’s what you’d expect to see in /usr/ but why didn’t I see it before?

What’s mounted?

# mount | grep /mnt 
system/bootenv/10.1-RELEASE on /mnt (zfs, local, noatime, nfsv4acls)

Oh. Now, that’s interesting.

Let’s clear things up and look at what we get:

# umount /mnt/dev
# beadm umount 10.1-RELEASE
Unmounted successfully
# beadm mount 10.1-RELEASE /mnt
Mounted successfully on '/mnt'
# zfs mount | grep /mnt
system/bootenv/10.1-RELEASE            /mnt
system/bootenv/10.1-RELEASE/tmp        /mnt/tmp
system/bootenv/10.1-RELEASE/usr        /mnt/usr
system/bootenv/10.1-RELEASE/usr/local  /mnt/usr/local
system/bootenv/10.1-RELEASE/usr/obj    /mnt/usr/obj
system/bootenv/10.1-RELEASE/usr/src    /mnt/usr/src
system/bootenv/10.1-RELEASE/var        /mnt/var
system/bootenv/10.1-RELEASE/var/audit  /mnt/var/audit
system/bootenv/10.1-RELEASE/var/empty  /mnt/var/empty
system/bootenv/10.1-RELEASE/var/log    /mnt/var/log
system/bootenv/10.1-RELEASE/var/tmp    /mnt/var/tmp

My theory: the contents of system/bootenv/10.1-RELEASE/usr is being over written by some other mount.

Let’s see what is mounted from the default BE:

# zfs mount | grep system/bootenv/default
system/bootenv/default             /
system/bootenv/default/tmp         /tmp
system/bootenv/default/usr/local   /usr/local
system/bootenv/default/usr/obj     /usr/obj
system/bootenv/default/usr/src     /usr/src
system/bootenv/default/var/audit   /var/audit
system/bootenv/default/var/empty   /var/empty
system/bootenv/default/var/log     /var/log
system/bootenv/default/var/tmp     /var/tmp

Ahh, there is nothing mounted at /usr from system/bootenv/default like there is at /mnt for system/bootenv/10.1-RELEASE.

If I remove that mount, everything looks normal:

# zfs umount system/bootenv/10.1-RELEASE/usr
# ls -l /mnt/usr
total 296
drwxr-xr-x   2 root  wheel  477 Feb 25 21:52 bin
drwxr-xr-x   2 root  wheel    2 Dec  4  2012 games
drwxr-xr-x   4 root  wheel    5 Oct  3 12:28 home
drwxr-xr-x  53 root  wheel  283 Aug 11  2014 include
drwxr-xr-x  39 root  wheel   39 Mar  9 00:38 jails
drwxr-xr-x   8 root  wheel  558 Feb 25 21:52 lib
drwxr-xr-x   5 root  wheel    5 Aug 11  2014 lib32
drwxr-xr-x   5 root  wheel    5 Dec  4  2012 libdata
drwxr-xr-x   7 root  wheel   62 Jan 16 19:34 libexec
drwxr-xr-x   3 root  wheel    3 Sep 10  2013 local
drwxr-xr-x   2 root  wheel    2 Mar  8 23:56 obj
drwxr-xr-x   2 root  wheel    2 May  3  2013 ports
drwxr-xr-x   2 root  wheel  295 Feb 25 21:52 sbin
drwxr-xr-x  30 root  wheel   30 May 27  2014 share
drwxr-xr-x   2 root  wheel    2 Mar  8 23:56 src

My questions are:

  1. Why does beadm mount system/bootenv/10.1-RELEASE/usr?
  2. Why isn’t system/bootenv/default/usr mounted?

Can they be mounted?

# zfs get canmount system/bootenv/default/usr \
                   system/bootenv/10.1-RELEASE/usr
NAME                             PROPERTY  VALUE     SOURCE
system/bootenv/10.1-RELEASE/usr  canmount  off       local
system/bootenv/default/usr       canmount  off       local

No, but.. they are.

I raised an issue.

Then I started wondering if there was anything useful in that location, so I checked:

$ sudo zfs mount -o altmnt=/mnt system/bootenv/default/usr
cannot mount 'system/bootenv/default/usr': 'canmount' property is set to 'off'

OK, I can’t mount it, so let’s change it so I can:

$ sudo zfs set canmount=auto system/bootenv/default/usr
cannot set property for 'system/bootenv/default/usr': 'canmount' must be one of 'on | off | noauto'

Oops, try again:

$ sudo zfs set canmount=noauto system/bootenv/default/usr
$ sudo zfs mount -o altmnt=/mnt system/bootenv/default/usr
$ sudo ls -l /mnt
total 0
$ 

OK, nothing there. Let’s umount that.

$ sudo zfs umount system/bootenv/default/usr
bash: /usr/local/bin/sudo: No such file or directory

Oh oh… Yeah, this wasn’t going anywhere. I couldn’t even login at the console; it failed.

Fortunately, CONTROL-ALT-DELETE allowed me to reboot that server.

I see now… I tried to specify an alternate mountpoint, but that option does not exist, so it was mounted over the existing /usr. Oops.

Trying, safely this time

Let’s try this instead:

$ sudo mount -t zfs system/bootenv/default/usr /mnt

$ ls -l /mnt
total 33
drwxr-xr-x  2 root  wheel  2 Mar 11 15:12 obj
drwxr-xr-x  2 root  wheel  2 Mar 11 15:12 src

$ ls -l /mnt/*
/mnt/obj:
total 0

/mnt/src:
total 0
$ 

OK, good, there isn’t much in there.

Is our original stuff still OK?

$ ls -l /usr
total 296
drwxr-xr-x   2 root  wheel  477 Feb 25 21:52 bin
drwxr-xr-x   2 root  wheel    2 Dec  4  2012 games
drwxr-xr-x   4 root  wheel    5 Oct  3 12:28 home
drwxr-xr-x  53 root  wheel  283 Aug 11  2014 include
drwxr-xr-x  39 root  wheel   39 Mar  9 00:38 jails
drwxr-xr-x   8 root  wheel  558 Feb 25 21:52 lib
drwxr-xr-x   5 root  wheel    5 Aug 11  2014 lib32
drwxr-xr-x   5 root  wheel    5 Dec  4  2012 libdata
drwxr-xr-x   7 root  wheel   62 Jan 16 19:34 libexec
drwxr-xr-x  19 root  wheel   20 Dec 27 20:35 local
drwxr-xr-x   3 root  wheel    3 Jan 16 21:47 obj
drwxr-xr-x  70 root  wheel   88 Mar  8 23:57 ports
drwxr-xr-x   2 root  wheel  295 Feb 25 21:52 sbin
drwxr-xr-x  30 root  wheel   30 May 27  2014 share
drwxr-xr-x  22 root  wheel   31 Jan 16 21:46 src

Phew!

I am quite sure I can destroy this dataset without repercussions but I think I’ll just rename it out from under the bootenv just to be sure.

The renaming

My first attempt:

$ sudo zfs rename system/bootenv/default/usr system/zzz-DELETEME.bootenv.default.usr 
cannot unmount '/usr/local': Device busy
$

I think I’ll have to do this in single user mode, or perhaps jump back to my mfsBSD live BSD drive. But that is going to be a different post. Just because.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive