openSUSE:Build Service Concept Backend Worker API
Abstract
Currently, the buildsystem has use cases were calling code in the worker/local build is interrupted to do special processing. The code itself does not belong to the target OS and architecture, so these packages should not be in the target repo/arch and should therefore also not be installed from the target repo/arch.
Use-cases
In the following sections there is a use-case based description of the new worker API first, to get an impression about why we need this new API and what you can do with it.
Worker allocation
There should be a configurable way to assign jobs to workers according to capabilities of the machine, the architecure or other parameters. This should be part of the new API. E.g. not only should the worker setup be interceptable by callbacks, but also the worker allocation.
Imaging
Lets start with the latest changes to the kiwi-inside-obs code. Now the worker uses build and not the hostsystem. This is great, but has some side-effects here: I use here basically Base:build, busybox and some more libs to build a mini-distro. Before the change, the workers used the hosts kiwi, which worked ok. Now i don't have the needed dependencies (kiwi, createrepo, python, sqlite, ...) in my small repository. They have to be inside the "image" repo and "image" needs also the be derived from the topmost repo (if repos were derived/stacked). As kiwi executes the imaging in an own chroot, the kiwi package and its dependencies could be taken e.g. from 11.0. This would make imaging possible also for repos which don't have kiwi itself or make using newer kiwi really easy. The imaging itself isn't affected by this.
Thinking of cross-kiwi, this is mandantory for this case, as the kiwi-chroot has to be in $HOSTARCH.
As conclusion, there's some need to make the kiwi-chroot independent of the repositories used for imaging.
Cross-build and virtualization in a worker
Now think of cross-build using XEN-workers: To use the qemu-user-mode inside a XEN-worker, a minimal System plus qemu in $HOSTARCH (!) is needed to start the XEN-VM. First step is then to register the qemu-binfmt-magic. All other Packages can then be $TARGETARCH, as the user-mode is active. This means: Preinstall (and Vminstall) plus qemu must be taken from $HOSTARCH and not from the repo/arch of the building package. This is the place for Martin and the worker-api ;), but its basically the same procedure as above.
build/baselibs.conf
Currently, the baselibs support is provided by special treatment inside the OBS worker or target OS packages. This special treatment could be removed and be done in the same way as qemu preprocessing or image/KWI processing is done with the new feature. So it is no longer needed to call target repo/arch code from the worker host os/arch directly.
rpmlint
Similiar to build/baselibs.conf case is the use of the rpmlint checker to check build results. I has like build/baselibs.conf an own, optional configuration file. Similiar is also that first an additional package, rpmlint, is installed. Similiar is also that after the build result, some actions are done.
Testframework
Another application similiar to the ones listed is calling a testframework, which is independent from tests performed inside a package. This could be more "static" analsysis like rpmlint (which in fact also checks compile message) or runtime tests performed with the packages (like ABI tests, consistency tests).
build-compare
The build iterator "build-compare" is another example of a post build callback.
post-build cleanup
There are packages existing that modify the build environment and provide setups like mounts that are not all recognized by the automatic worker cleanup, even not with a virtual machine. These additional cleanup handlers could be called by the new method.
pre-build setup
Not all pre-build setups can be automatically done or are even useful to do. The current virtual machine setup can be too restrictive, or too limited for specific cases like for example imaging.
Functions
Backend worker API solution
All the use-cases listed above have a communality. The currently applied special hardcoded handling is the expression of the fact that host OS worker setup is missing in addition to the normal target OS worker chroot setup. The common solution is:
- Host packages code can be installed from packages inside the workers chroot
- Callbacks can intercept the execution, before the chroot or the virtual machine or emulator starts
- Callbacks can intercept the execution, after the worker finished and before results are handed back to backend
This method allows to dynamically change the currently hardcoded bahavior and to change the behavior also on a project specific basis.