The wikis are now using the new authentication system.
If you did not migrate your account yet, visit

openSUSE:usr merge

Jump to: navigation, search

Modern Linux operating systems no longer stick to the historic

and in may ways arbitrary separation of eg /bin vs /usr/bin. Instead all vendor supplied operating system components are nowadays unified below the /usr tree. Arguments and refences to background can be found at

In order to stay compatible and make all files available both via eg /bin and /usr/bin, the goal is to replace the top level directories /bin, /sbin, /lib and depending on architecure /lib64 with symlinks to the directories of the same name in /usr.

State in openSUSE

UsrMerge is active in openSUSE Tumbleweed since 20210527. Pre-usrmerge systems get converted to the new layout on upgrade of the filesystem package.

Installation of the filesystem may fail if the conversion didn't work for some reason. Do not continue if the filesystem package fails to install. Contact the openSUSE Factory mailinglist or file a bug directly.

Known Problems:

  • 3rd party kernel modules do not compile due to stale symlink (Bug 1186710). Workaround:
    cd /usr; ln -s . /usr/usr
  • File systems that do not support RENAME_EXCHANGE such as ZFS or NFS cannot perform live conversion (Bug 1186637). Workaround: Wait for
  • There is one case of a not yet identified symbol lookup error (Bug 1186730). Cause and therefore workaround unknown. Make sure to have a rescue system around!
  • Conversion fails if there's a mount point below (/usr)/{bin,sbin,lib,lib64}. Workaround: unmount before upgrading.
  • Upgrading with /usr on separate partition does not work (Bug 1186781). Workaround: remove the -l parameter in one of the cp call in /usr/libexec/convertfs
  • Upgrading layers of containers does not work (Bug 1187027). Please re-deploy the base image.
  • Locked packages can make the UsrMerge fail (Bug 1029961). Make sure to check your locked package list before the upgrade
  • Conversion fails in WSL due to missing renameat2 syscall. Workaround: use WSL2, wait for


In the middle of the upgrade, no binary launches anymore

In the middle of the upgrade, usually after some error message from the filesystem package was ignored (don't do that), no program can be run anymore.

   # ls
   -bash: /usr/bin/ls: No such file or directory

That usually means that the dynamic linker was already moved to /usr/lib64 but /lib64 is still a directory rather than a symlink to /usr/lib64 (/lib resp /usr/lib on 32bit systems). A temporary workaround is to symlink the linker and glibc to /lib64. The name and location of the dynamic linker is architecture dependent, on x86_64 it's So the following commands may restore the ability to run binaries:

   /usr/lib64/ /usr/bin/ln -s /usr/lib64/ /lib64/
   /usr/lib64/ /usr/bin/ln -s /usr/lib64/ /lib64/

How to use rollback in case of troube?

If the system is on btrfs with snapshots enabled, the easiest way to recover from failure is to boot from a previous, known good snapshot. The process is describe in the Leap reference manual

Was the UsrMerge really necessary?

Adjusting a hundred packages is a lot of effort. Getting the conversion right is tricky with potential breakage all over the place. So maybe just keep the status quo? The current state is an inconsistent mess though. Some stuff in /bin and some in /usr/bin, links from /bin to /usr/bin and also vice versa. Libraries randomly in either /usr/lib or /lib etc. So that deserves to be cleaned up to reach some consistency again. Undoing the previous efforts however wouldn't work as existing script may already use eg /usr/bin/cp. So the way out is to finally do it.

Do we need to fix scripts to use eg /usr/bin/sh?

Fortunately not. The point of the merge is that both locations are valid. So scripts may continue to use /bin/sh. Also, ELF binaries will continue to use eg /lib/

Can't we create the compat symlinks via brp scripts?

That would be nice as packages would automatically adopt to usrmerged or not situations at build time without the %usrmerged macro. Unfortunately brp scripts can't modify the file list so that won't help.

How does the filesystem package end up in rpm transactions?

Glibc pre-requires the filesystem package. So as long as core Linux utilities (including /bin/sh) are linked against glibc the filesystem package will be pulled into transactions.

Will kernel modules also move to /usr?

So far kernel modules are packaged in /lib/modules. Due to the /lib -> /usr/lib link they end up getting installed in /usr/lib/modules anyway. So there is not strict requirement to change the packaging to /usr. Nevertheless in order to have consistent rpm packages it would be advisable to change all packages to install to /usr. The location /lib/modules would still work for applications that access kernel modules. Similar arguments apply to /lib/firmware.


Packages that have relative symlinks in eg /sbin that point to a binary in /usr/bin will end up with a stale symlink in usrmerge case.

How the conversion works technically

RPM itself cannot replace a directory such as /bin with a symlink. Anyway there is no guarantee that the directory would be empty at the time of replacing it with a symlink. So in case of UsrMerge the content of eg /bin would have to be merged with /usr/bin and only then /bin replaced with a symlink. The replacement has to be atomic in order to do it at run time. Normal rename(2) ie mv cannot do that. Fortunately the Linux kernel has a syscall "renameat2" that can exchange two arbitrary file system objects.

So with that at hand the filesystem package can actually contain the /bin -> /usr/bin symlink and have a %pretrans script to perform the conversion. A shell script plus small helper binary for the renameat2 syscall can be used. The cp program from coreutils provides all required options to recursively copy all file types, permissions attributes etc. The script would only be called on updates where an already working system can be assumed. New installations would not require the conversion obviously.

Conversion script:

  • for DIR in bin, sbin, lib, lib64
  • copy /$DIR to /usr/$DIR.usrmerge, using hard links if possible (legacy installation may have separate /usr still).
  • hardlink copy $DIR into /usr/$DIR.usrmerge, creating backups of existing files
  • iterate over backed up files and delete symlinks, ie the legacy /bin/foo -> /usr/bin/foo links. If the backup is not a link, move it back to the original name (was reverse link then).
  • exchange /usr/$DIR.usrmerge with /usr/$DIR
  • create /$DIR.usrmerge -> /usr/$DIR symlink
  • exchange /$DIR.usrmerge with /$DIR
  • remove /$DIR.usrmerge and /usr/$DIR.usrmerge


  • the filesystem package has to require the package containing the conversion script as long as there are distros around where we support upgrading from. Fedora had a patch for rpm that made it provide 'rpmlib(X-CheckUnifiedSystemdir)'. The patch was rejected as the filesystem's %pretrans script actually modifies the filesystem so that this provides would have to change during the transaction. Also, libsolv doesn't actually look at rpmlib() provides in the first place, so would be useless anyway.
  • The system to upgrade has to have a working /usr/bin/cp for the process to work. This cannot be added as (pre)requires though as the filesystem package is supposed to be installed first and not depend on anything.


openSUSE Factory mailinglist

See also

External links