Kernel Module Packages
Kernel Module Packages (KMPs) are a way for anyone to package kernel modules in such a way that they are cleanly handled in case of kernel updates.
A loaded kernel module is part of the Linux kernel and has access to all data structures of the kernel. As such it needs to be compiled with exactly the right kernel configuration the right compiler flags and -- most importantly -- the right kernel sources. In real life, only the exported symbols are normally accessed -- but there are more than 6000 exported kernel symbols in recent 2.6 kernels.
Due to the fact that kernel internal interfaces (kABI) are quite volatile and change frequently, kernel updates can quite easily change some of the interfaces. This will break the kernel modules that have been compiled for the older version. Fortunately this is detected by symbol versioning, thus preventing the module from loading into the kernel and thus preventing data corruption.
Novell does actually spend quite some effort not to change the kABI in incompatible ways when doing maintenance updates for the enterprise kernels; this does not change the fact that someone providing kernel modules independently from the kernel provider needs to be prepared for kABI change -- only much less often. Change happens when a service pack is released or possibly when a critical bug or a severe security issue can not be fixed otherwise.
This close connection between the kernel and its modules made distributors (incl. SUSE/Novell) compile them all together and provide them all in one binary kernel RPM. Every kernel module (mostly drivers) that was needed by users needed to be included in the kernel, so it would be recompiled on every kernel security update.
This approach of handling kernel modules has come to its limits:
- Despite the willingness to accept modules in the tree, third party kernel modules existed, and no provisions were made to handle this correctly in case of kernel updates. In the worst case, this could lead to a system that was not bootable any more after a kernel update -- clearly not the best user experience.
- Modules can not always be provided. There are legal restrictions, issues with code quality or portability or simply missing possibilities to test (be it due to missing hardware or due to missing manpower or knowledge).
- Sometimes, kernel modules just are not ready in time a distribution is cut -- a way to provide modules/drivers later (or updated versions of existing modules) is highly useful then.
Enter Kernel Module Packages
The idea is to cleanly integrate kernel modules that are not part of the distributor-provided kernel RPM. Anyone with the right knowledge can create a kernel module that matches a certain kernel version and works with it. The real difficult problem to solve is to maintain the module-provided functionality on a kernel security update. This is achieved by reusing the module if possible and by requiring a recompiled modules if needed.
To address this, the following steps are taken on a kernel update:
- detect third party kernel modules
- for every module, check whether it still works with the new kernel (by looking at all the needed symbols and their versions)
- if it still works, make sure it can and will be reused by the new kernel
- if it is not compatible with the new kernel anymore
- try to download a new kernel module RPM
- if this is not available display an error message (and by default abort the update)
This is all that the user gets to see -- once kernel module packages are installed, updates will automatically be downloaded and installed if needed.
But to make this happen, the distributor and the module provider need to do a bit of work:
To make sure new kernel modules get recompiled and provided on a server, the distributor can offer to have people register their modules; the module creators then can be notified on kABI changes that affect them. This is optional, but saves time as these notifications usually can be done prior to a kernel release. Later, this should also be possible to do within the build service provided by openSUSE.
Alternatively, the distributor can also offer to recompile the modules itself -- Novell does do so for some extra modules it provides this way -- and Novell Partner Engineering (formerly: Developer Services) offers it as a service for partners -- see the Linux drivers page to get more information if this is interesting for your company.
In case of kABI changes, a mere recompilation of the module against the right kernel sources does do the job -- it is Novell's policy not to require code changes (read do kAPI changes) in security updates. This even applies to SLES service packs, though exceptions are not completely impossible there. But as even this simple recompilation can create a bit of work, Novell does watch changes closely nowadays and tries hard to avoid kABI changes for security/maintenance updates for SLES and SUSE Linux. However, SLES service packs will typically change a good part of the kABI -- and openSUSE factory kernels are expected to have kABI changes very often, as the upstream development is tracked there.
Kernel Module Package format
There are two different mechanisms used to check compatibility of the module with the kernel, depending on the version of SUSE Linux. For SLES9SP3/NLD9SP3/OESv1SP2 and SUSE Linux 10.0 (these versions subsequently referred to as Code9), there's a separate tool called kernel-update-tool that does the work. It gets invoked on kernel updates and does all the checks and downloads. For SLES10/SLED10/OESv2 and SUSE Linux 10.1 and later (subsequently referred to as Code10), the solution has been integrated with the normal system installation, configuration and update tools. Consequently, the dependencies on kernel symbols have been mapped to RPM dependencies that the dependency resolver will then try to fulfill using the available installation/update sources and this way automatically download needed modules or detect missing dependencies.
The kernel module packages are normal RPMs that contain the kernel modules (.ko files) and a bit of metadata. For the third party kernel modules to be identified, updates to be found, etc., the kernel module packages need to conform to packaging conventions. Packaging conventions are documented separately for the older Code9 (openSUSE 10.0 and SLES9) solution, for Code10 (openSUSE 10.1 and SLES10) solution, and for newer one Code11 (openSUSE 11.1 and SLES11) solution.
The solution to provide downloadable updates is also slightly different. Whereas the Code9 requires the RPM to arranged in a specific directory structure on a http (or ftp) server, the Code10 and Code11 solution requires the format of a YaST installation source -- we recommend using the yum format -- Code11 will also work with rpm-md repositories.
Like any other RPM, the kernel module packages should have a GnuPG signature. The updater will check it and complain if the package is not signed, signed by an unknown party or is corrupted. On initial installation, the install documentation or install script should thus recommend to import the GnuPG signature key of the module package provider into the RPM keyring to mark that RPM package provider as trusted. Key import should not happen in secret, but with the explicit consent of the admin installing the package.
Initial Module Installation
The above deals with how third party kernel modules can survive kernel updates without creating trouble for the user. But how can additional modules be found initially?
- The module packages can be part of a product. Vendors may include the RPMs on driver CDs or whatever the normal software distribution channel is. For Code10, a tiny script that registers an update source and offers to import the GnuPG package signature key into the RPM keyring are recommended.
- The module packages can be offered for download. RPMs can just be downloaded (Code9) or the download location can just be entered as installation and update source during installation time (or later) in the installer/updater (Code10/11).
- The user can find the download sites by searching the vendor's site, reading documentation from the vendor or from others, following hints from openSUSE, search engines, etc.
- To save search work, a better automated mechanism that saves the searching work for the end user using a software/driver portal where software providers could register additional installation/update sources which are offered by YaST optionally at installation time is in preparation. Offering packages as function of the detected hardware is also part of the design.
The packaging format specification also takes care that modules that have been created for a different kernel version, but are compatible with the one that's installed on the system will get used for the installed kernel.
Support and Certification
Support and Certification are only relevant for the Enterprise products (SLES, NLD/SLED, OES). Support in this context refers to the ability of customers to open support calls and get guaranteed individual attention (including engineering efforts) to the problem they are reporting.
A kernel module has access to the kernel address space and can easily put the stability and reliability of the operating system at risk. Modules unknown to Novell have the status "unsupported" and upon loading them will generate a warning message to the syslog and "taint" the kernel. If a user has such an unsupported kernel module loaded, Novell can not do very much to help with kernel problems. Instead the problem should be reproduced without such a module loaded to be debuggable.
However, it is possible for a module provider to get the module in the "eXternally supported" status; for this it's necessary that he works closely with Novell. The appropriate support channel interfaces need to be in place to allow Novell support and the module provider to work jointly on a customer issue. The module provider ust commit to maintain the module for the lifetime of SLES. The kernel module package needs to be built according to the rules specified above -- best by having it built by Novell's build service (Novell Partner Engineering). Modules built this way can also be tested and certified and thus become part of certified systems.
In the specfile preamble, request that kernel-syms (pulls in kernel-devel) and modutils be present.
Requires: kernel-syms modutils SourceN: %name-kmp-preamble
Right before %description, specify this to have kmp subpackage produced:
%kernel_module_package -p %name-kmp-preamble -x ec2 xen
The -p argument specifies additional rpm tags to be added to each kmp subpackage. The -x flag, if provided, requests the exclusion of these kernel flavors from the build. The reverse is omitting -x and just specifying flavors to which only build for.
A few tags are sourced from the tags of an otherwise empty "KMP" subpackage:
%package KMP Summary: The kernel modules for xyz Group: System/Kernel %description KMP This provides blahblah.
In the kmp preamble file, %1 denotes the flavor, and %2 the package version. In this file, there may be one or more of the following lines:
- Enhances: kernel-%1
“Enhances” is a reverse “Suggests”. The mypkg-kmp-flavor subpackage may be suggested during the installation of kernel-flavor.
- Supplements: packageand(kernel-%1:%name)
“Supplements” is a reverse “Recommends”. The installation of the mypkg-kmp-flavor subpackage will be recommended if kernel-flavor too will be or is present in the system.
Each kmp already provides "mypkg-kmp" and "mypkg-kmp-flavor".