openSUSE:usr merge

Modern Linux operating systems no longer stick to the historic

and in may ways arbitrary separation of eg /bin vs /usr/bin. Instead all vendor supplied operating system components are nowadays unified below the /usr tree. Arguments and references to background can be found at freedesktop.org.

In order to stay compatible and make all files available both via eg /bin and /usr/bin, the goal is to replace the top level directories /bin, /sbin, /lib and depending on architecure /lib64 with symlinks to the directories of the same name in /usr.

State in openSUSE

UsrMerge is active in openSUSE Tumbleweed since 20210527. Pre-usrmerge systems get converted to the new layout on upgrade of the filesystem package.

Installation of the filesystem may fail if the conversion didn't work for some reason. Do not continue if the filesystem package fails to install. See the FAQ below. If in doubt contact the openSUSE Factory mailinglist or file a bug directly.

Packaging

The UsrMerge is active in Tumbleweed as such any existing methods to detect building for Tumbleweed can be used in spec files. A common one for example would be

   %if 0%{?suse_version} < 1550
       [handling of legacy locations in older distros]
   %endif

The use of the %usrmerged macro aided the transition but is considered deprecated meanwhile. Also, when backporting packages to older distributions, please do not introduce new compat symlinks. Move binaries to the legacy location instead. New compat symlinks may break upgrades later.

FAQ

In the middle of the upgrade, no binary launches anymore

In the middle of the upgrade, usually after some error message from the filesystem package was ignored (don't do that), no program can be run anymore.

   # ls
   -bash: /usr/bin/ls: No such file or directory

That usually means that the dynamic linker was already moved to /usr/lib64 but /lib64 is still a directory rather than a symlink to /usr/lib64 (/lib resp /usr/lib on 32bit systems). A temporary workaround is to symlink the linker and glibc to /lib64. The name and location of the dynamic linker is architecture dependent, on x86_64 it's ld-linux-x86-64.so.2. So the following commands may restore the ability to run binaries:

   /usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ln -s /usr/lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
   /usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ln -s /usr/lib64/libc.so.6 /lib64/libc.so.6

How to use rollback in case of troube?

If the system is on btrfs with snapshots enabled, the easiest way to recover from failure is to boot from a previous, known good snapshot. The process is describe in the Leap reference manual

Was the UsrMerge really necessary?

Adjusting a hundred packages is a lot of effort. Getting the conversion right is tricky with potential breakage all over the place. So maybe just keep the status quo? The current state is an inconsistent mess though. Some stuff in /bin and some in /usr/bin, links from /bin to /usr/bin and also vice versa. Libraries randomly in either /usr/lib or /lib etc. So that deserves to be cleaned up to reach some consistency again. Undoing the previous efforts however wouldn't work as existing script may already use eg /usr/bin/cp. So the way out is to finally do it.

Do we need to fix scripts to use eg /usr/bin/sh?

Fortunately not. The point of the merge is that both locations are valid. So scripts may continue to use /bin/sh. Also, ELF binaries will continue to use eg /lib/ld-linux.so.2

Can't we create the compat symlinks via brp scripts?

That would be nice as packages would automatically adopt to usrmerged or not situations at build time without the %usrmerged macro. Unfortunately brp scripts can't modify the file list so that won't help.

How does the filesystem package end up in rpm transactions?

Glibc pre-requires the filesystem package. So as long as core Linux utilities (including /bin/sh) are linked against glibc the filesystem package will be pulled into transactions.

Will kernel modules also move to /usr?

Yes! Kernel packages install files into /usr/lib/modules/$kernelversion. The compat symlink /lib is in place so everything can still be accessed through /lib/modules of course.

Incompatibilities

Packages that have relative symlinks in eg /sbin that point to a binary in /usr/bin will end up with a stale symlink in usrmerge case.

How the conversion works technically

RPM itself cannot replace a directory such as /bin with a symlink. Anyway there is no guarantee that the directory would be empty at the time of replacing it with a symlink. So in case of UsrMerge the content of eg /bin would have to be merged with /usr/bin and only then /bin replaced with a symlink. The replacement has to be atomic in order to do it at run time. Normal rename(2) ie mv cannot do that. Fortunately the Linux kernel has a syscall "renameat2" that can exchange two arbitrary file system objects.

So with that at hand the filesystem package can actually contain the /bin -> /usr/bin symlink and have a %pre script to perform the conversion. A shell script plus small helper binary for the renameat2 syscall can be used. The cp program from coreutils provides all required options to recursively copy all file types, permissions attributes etc. The script would only be called on updates where an already working system can be assumed. New installations would not require the conversion obviously.

Conversion script:

for DIR in bin, sbin, lib, lib64
copy /$DIR to /usr/$DIR.usrmerge, using hard links if possible (legacy installation may have separate /usr still).
hardlink copy $DIR into /usr/$DIR.usrmerge, creating backups of existing files
iterate over backed up files and delete symlinks, ie the legacy /bin/foo -> /usr/bin/foo links. If the backup is not a link, move it back to the original name (was reverse link then).
exchange /usr/$DIR.usrmerge with /usr/$DIR
create /$DIR.usrmerge -> /usr/$DIR symlink
exchange /$DIR.usrmerge with /$DIR
remove /$DIR.usrmerge and /usr/$DIR.usrmerge

Caveats

the filesystem package has to require the package containing the conversion script as long as there are distros around where we support upgrading from. Fedora had a patch for rpm that made it provide 'rpmlib(X-CheckUnifiedSystemdir)'. The patch was rejected as the filesystem's %pretrans script actually modifies the filesystem so that this provides would have to change during the transaction. Also, libsolv doesn't actually look at rpmlib() provides in the first place, so would be useless anyway.
The system to upgrade has to have a working /usr/bin/cp for the process to work. This cannot be added as (pre)requires though as the filesystem package is supposed to be installed first and not depend on anything.

Known Problems

Upgrading layers of containers does not work (Bug 1187027). This is due to overlayfs. There is no workaround. Please re-deploy the base image instead.
Locked packages can make the UsrMerge fail (Bug 1029961). Make sure to check your locked package list before the upgrade
Conversion fails if there's a mount point below (/usr)/{bin,sbin,lib,lib64}. Workaround: unmount before upgrading.
There is one case of a not yet identified symbol lookup error (Bug 1186730). Cause and therefore workaround unknown. Make sure to have a rescue system around!
~~3rd party kernel modules do not compile due to stale symlink (Bug 1186710).~~ Workaround:
```
cd /usr; ln -s . /usr/usr
```
~~File systems that do not support RENAME_EXCHANGE such as ZFS or NFS cannot perform live conversion (Bug 1186637). Workaround: Wait for https://build.opensuse.org/request/show/900396~~
~~Upgrading with /usr on separate partition does not work (Bug 1186781). Workaround: remove the -l parameter in one of the cp call in /usr/libexec/convertfs~~

~~Conversion fails in WSL due to missing renameat2 syscall. Workaround: use WSL2, wait for https://build.opensuse.org/request/show/900396~~

Communication

openSUSE Factory mailinglist