The wikis are now using the new authentication system.
If you did not migrate your account yet, visit https://idp-portal-info.suse.com/

openSUSE:usr merge

Jump to: navigation, search

Modern Linux operating systems no longer stick to the historic

and in may ways arbitrary separation of eg /bin vs /usr/bin. Instead all vendor supplied operating system components are nowadays unified below the /usr tree. Arguments and refences to background can be found at freedesktop.org

In order to stay compatible and make all files available both via eg /bin and /usr/bin, the goal is to replace the top level directories /bin, /sbin, /lib and depending on architecure /lib64 with symlinks to the directories of the same name in /usr.



How Fedora did it

Fedora introduced a filesystem package that contained /bin etc as symlinks to /usr. Since rpm can't replace a directory with a symlink itself, they introduced a magic rpmlib(X-CheckUnifiedSystemdir) provides tag in rpm itself. Rpm would only offer that that if /bin was a symlink. So the new filesystem package could not be installed on legacy installations.

Since replacing directories and especially four of them couldn't be done atomically the conversion itself was delegated to dracut. When manually adding a "rd.convertfs" parameter on the boot command line dracut would replace the top level directories with symlinks after mounting the root file system but before starting init.

The actual code still exists and does the following:

  • copy eg /usr/bin to /usr/bin.usrmove-new by using hardlinks.
  • copy /bin into /usr/bin.usrmove-new overwriting but backing up duplicates
  • delete all backed up symlinks. Probably assuming that's links pointing to the top level dirs eg /usr/bin/foo -> /bin/foo.
  • restore backed up binaries if the new file is a symlink. Ie assuming /bin/foo -> /usr/bin/foo where the latter is the binary.

Current state in openSUSE

A previous attempt of the UsrMerge from 2012 was never finished. That approach moved all binaries into /usr and packaged legacy symlinks in /bin (see previous revision of this page). Therefore openSUSE has almost one hundred packages with symlinks in /bin pointing to /usr.

The previous approach did not explain how to transition from that situation to the actual directories as symlinks though.

Taking the existing dracut code to run it on openSUSE leads to the following findings:

  • Users have to manually trigger the conversion. That means some probably never will and then keep running into issues with the new filesystem package.

    The Fedora idea was to Require: rpmlib(X-CheckUnifiedSystemdir) in the filesystem package that ships the top level dirs as symlink. To actually get that into the repodata, createrepo_c has to be patched as rpmlib() strings are normally ignored.

    On a system that does not have the rpmlib(X-CheckUnifiedSystemdir) property (ie with legacy /bin), zypper behaves the following way:
       # zypper dup
       Loading repository data...
       Reading installed packages...
       Computing distribution upgrade...
 
       Problem: nothing provides rpmlib(X-CheckUnifiedSystemdir) needed by filesystem-15.5-41.1.x86_64
        Solution 1: deinstallation of filesystem-15.5-33.1.x86_64
        Solution 2: keep obsolete filesystem-15.5-33.1.x86_64
        Solution 3: break filesystem-15.5-41.1.x86_64 by ignoring some of its dependencies
 
       Choose from above solutions by number or cancel [1/2/3/c/d/?] (c): 

The result can be tweaked a bit by preventing deinstallation as option:

       # echo requires:filesystem >> /etc/zypp/systemCheck
       # zypper dup
       Loading repository data...
       Reading installed packages...
       Computing distribution upgrade...
 
       Problem: This request will break your system!
         nothing provides rpmlib(X-CheckUnifiedSystemdir) needed by filesystem-15.5-41.1.x86_64
 
        Solution 1: Following actions will be done:
         ignore the warning of a broken system (requires:filesystem)
         deinstallation of filesystem-15.5-33.1.x86_64
        Solution 2: keep obsolete filesystem-15.5-33.1.x86_64
        Solution 3: break filesystem-15.5-41.1.x86_64 by ignoring some of its dependencies
 
       Choose from above solutions by number or cancel [1/2/3/c/d/?] (c):

So choosing that approach means existing systems would not be able to upgrade to a new TW snapshot that has the UsrMerge implemented.

  • After the conversion installed packages that had files in both top level and usr directories are broken according to rpm --verify as only one of the files can exist.
  • After the conversion, packages that contain files with the same name in top level directories as well as /usr would no longer be installable due to file conflicts with themselves.
   # rm -rf /tmp/r
   # mkdir -p /tmp/r/usr/bin
   # ln -s usr/bin /tmp/r/bin
   # rpm --root /tmp/r --initdb
   # rpm -Uvh --noscripts --nodeps --root /tmp/r bash-4.4-lp152.11.80.x86_64.rpm
      Preparing...                          ################################# [100%]
           file /usr/bin/sh conflicts between attempted installs of bash-4.4-lp152.11.80.x86_64 and bash-4.4-lp152.11.80.x86_64
   # echo $?
   1
  • The conversion algorithm has gaps. The kbd package in openSUSE for example contains this:
 /bin/psfgettable -> /usr/bin/psfgettable
 /usr/bin/psfgettable -> psfxtable

The algorithm would leave a broken /usr/bin/psfgettable symlink pointing to itself.

Preparation

Packages

The first step going forward would be to create an environment that allows to test ways to do the actual conversion. For that the compat symlinks have to get out of the way. So packages with compat symlinks in /(s)bin have to add an %if condition around the compat sections checking for %usrmerged. That allows to rebuild all affected packages in a consistent via prjconf swtich without actually changing package content in Factory right away.

Example:

       %install
       ...
       %if !0%{?usrmerged}
       mkdir -p %{buildroot}/sbin
       ln -s %{_sbindir}/%{name} %{buildroot}/sbin
       %endif
       ...
       %files
       ...
       %if !0%{?usrmerged}
       /sbin/%{name}
       %endif


Build System

openSUSE Factory is fully self hosting. Means built packages are used automatically to build other packages. Means that for example changing the filesystem package to contain the `/bin -> /usr/bin` symlink would immediately leads to breaking builds of all other packages. That's because eg bash has both /bin/bash and /usr/bin/bash, ie as outlined earlier would lead to a file conflict with itself, so bash couldn't be installed anymore. Doing it the other way around ie building bash without including /bin/bash, the build system would also break as /bin/bash would be missing then while required by other packages and the build script itself.

There are two ways to overcome this chicken and egg problem:

  1. use file triggers to automatically create /bin symlinks
  2. turn off the `useforbuild` setting of OBS until all packages are migrated.

Both methods have pros and cons. The symlink method takes quite some time (packages building on after another) and attention to do the right thing in the right moment. Also, the file trigger code is non-trivial (contained in compat-usrmerge package). The method works on standalone, self hosting projects.

The useforbuild method on the other hand requires less steps and is rather quick as all affected packages can be built in parallel. The method does not work for self hosted projects though, it relies on a parent project to contain all required packages for building.

The steps for methods are the following:

Symlink Method

  • All packages need to be prepared with %usrmerged macro
  • disable building of project and set rebuild=local
  • adjust prjconf: %usrmerged 1
  • enable compat-usrmerge
  • add compat-usermerge and compat-usrmerge-build as Support in prjconf
  • enable bash
  • add bash-legacybin to prjconf as Preinstall
  • explicitly disable building of filesystem, glibc and gcc10 (gcc for speed reasons, not critical)
  • enable building of project again, remove explicit enable from bash
  • add Support: !post-build-checks for filesystem as installing the package over the old one won't work here
  • remove disabling of filesystem
  • remove post-build-check disabling from prjconf
  • remove bash-legacybin from prjconf
  • enable glibc
  • enable gcc10 and switch to rebuild=transitive again

Useforbuild method

  • All packages need to be prepared with %usrmerged macro
  • disable building of project and set rebuild=local, disable useforbuild
  • adjust prjconf:
 %usrmerged 1
 %_pamdir /usr/%{_lib}/security
 %_firmwaredir /usr/lib/firmware
  • enable compat-usrmerge [only as long as it's not in Factory already]
  • enable useforbuild for compat-usrmerge
  • add compat-usrmerge-build as Support in prjconf
  • enable building
  • trigger all affected packages
  • remove disabling of useforbuild
  • rebuild failing packages
  • prjconf for gawk
 Support: !post-build-checks
  • enable useforbuild and switch to transitive for full rebuild


Overall the useforbuild method seems easier. For Factory that means

  • Ring0 should be build disabled to make sure there's a working bootstrap set
  • Factory needs to build against it's last snapshot during the conversion
  • Stagings build against aggregated binaries from Ring0 so should work.
  • Finally Ring0 could build against an aggregated version of itself, ie copy aggregates from a staging.

Conversion

RPM itself cannot replace a directory such as /bin with a symlink. Anyway there is no guarantee that the directory would be empty at the time of replacing it with a symlink. So in case of UsrMerge the content of eg /bin would have to be merged with /usr/bin and only then /bin replaced with a symlink. The replacement has to be atomic in order to do it at run time. Normal rename(2) ie mv cannot do that. Fortunately the Linux kernel has a syscall "renameat2" that can exchange two arbitrary file system objects.

So with that at hand the filesystem package can actually contain the /bin -> /usr/bin symlink and have a %pretrans script to perform the conversion. A shell script plus small helper binary for the renameat2 syscall can be used. The cp program from coreutils provides all required options to recursively copy all file types, permissions attributes etc. The script would only be called on updates where an already working system can be assumed. New installations would not require the conversion obviously.

Conversion script:

  • for DIR in bin, sbin, lib, lib64
  • copy /$DIR to /usr/$DIR.usrmerge, using hard links if possible (legacy installation may have separate /usr still).
  • hardlink copy $DIR into /usr/$DIR.usrmerge, creating backups of existing files
  • iterate over backed up files and delete symlinks, ie the legacy /bin/foo -> /usr/bin/foo links. If the backup is not a link, move it back to the original name (was reverse link then).
  • exchange /usr/$DIR.usrmerge with /usr/$DIR
  • create /$DIR.usrmerge -> /usr/$DIR symlink
  • exchange /$DIR.usrmerge with /$DIR
  • remove /$DIR.usrmerge and /usr/$DIR.usrmerge


Caveats

  • the filesystem package has to require the package containing the conversion script as long as there are distros around where we support upgrading from. Fedora had a patch for rpm that made it provide 'rpmlib(X-CheckUnifiedSystemdir)'. The patch was rejected as the filesystem's %pretrans script actually modifies the filesystem so that this provides would have to change during the transaction. Also, libsolv doesn't actually look at rpmlib() provides in the first place, so would be useless anyway.
  • The system to upgrade has to have a working /usr/bin/cp for the process to work. This cannot be added as (pre)requires though as the filesystem package is supposed to be installed first and not depend on anything.

Issues to solve

Provides of /(s)bin files

File provides of UsrMerged packages for stuff in /bin resp /sbin do no longer work. There are two solutions:

  • packages that uses to have files in /bin or /sbin have to manually add provides
  • a provides generator automatically adds legacy provides

selinux

Do we need to relabel?

update-alternatives

✅ fixed in update-alternatives: https://build.opensuse.org/request/show/863143

update-alternatives cannot handle switching to the /usr version as master

   # /usr/sbin/update-alternatives --install /bin/foo foo /usr/bin/true 15 --slave /usr/bin/foo usr-bin-foo /usr/bin/true
   update-alternatives: using /usr/bin/true to provide /bin/foo (foo) in auto mode

Declaring the usr version as master does not work:

   # /usr/sbin/update-alternatives --install /usr/bin/foo usr-bin-foo /usr/bin/true 15
   update-alternatives: error: alternative usr-bin-foo can't be master: it is a slave of foo

Moving the /bin variant to /usr leads to missing symlinks:

   # /usr/sbin/update-alternatives --install /usr/bin/foo foo /usr/bin/true 15
   update-alternatives: renaming foo link from /bin/foo to /usr/bin/foo
   update-alternatives: warning: forcing reinstallation of alternative /usr/bin/true because link group foo is broken
   # l {/usr,}/bin/foo
   /usr/bin/ls: cannot access '/usr/bin/foo': No such file or directory
   /usr/bin/ls: cannot access '/bin/foo': No such file or directory
   # l /etc/alternatives/foo
   lrwxrwxrwx 1 root root 13 Nov 30 16:29 /etc/alternatives/foo -> /usr/bin/true*


Can be fixed by calling --auto once:

   # update-alternatives --auto foo
   update-alternatives: warning: forcing reinstallation of alternative /usr/bin/true because link group foo is broken
   # l {/usr,}/bin/foo
   /usr/bin/ls: cannot access '/bin/foo': No such file or directory
   lrwxrwxrwx 1 root root 21 Nov 30 16:30 /usr/bin/foo -> /etc/alternatives/foo*
   # update-alternatives --auto foo
   #

FAQ

Do we really need to do this?

Adjusting a hundred packages is a lot of effort. Getting the conversion right is tricky with potential breakage all over the place. So maybe just keep the status quo? The current state is an inconsistent mess though. Some stuff in /bin and some in /usr/bin, links from /bin to /usr/bin and also vice versa. Libraries randomly in either /usr/lib or /lib etc. So that deserves to be cleaned up to reach some consistency again. Undoing the previous efforts however wouldn't work as existing script may already use eg /usr/bin/cp. So the way out is to finally do it.

Do we need to fix scripts to use eg /usr/bin/sh?

Fortunately not. The point of the merge is that both locations are valid. So scripts may continue to use /bin/sh. Also, ELF binaries will continue to use eg /lib/ld-linux.so.2

Can't we create the compat symlinks via brp scripts?

That would be nice as packages would automatically adopt to usrmerged or not situations at build time without the %usrmerged macro. Unfortunately brp scripts can't modify the file list so that won't help.

How does the filesystem package end up in rpm transactions?

Glibc pre-requires the filesystem package. So as long as core Linux utilities (including /bin/sh) are linked against glibc the filesystem package will be pulled into transactions.

Will kernel modules also move to /usr?

So far kernel modules are packaged in /lib/modules. Due to the /lib -> /usr/lib link they end up getting installed in /usr/lib/modules anyway. So there is not strict requirement to change the packaging to /usr. Nevertheless in order to have consistent rpm packages it would be advisable to change all packages to install to /usr. The location /lib/modules would still work for applications that access kernel modules. Similar arguments apply to /lib/firmware.

Incompatibilities

Packages that have relative symlinks in eg /sbin that point to a binary in /usr/bin will end up with a stale symlink in usrmerge case.

Communication

openSUSE Factory mailinglist


See also

External links