openSUSE:Build Service Tutorial
Prerequisites
You should have a general understanding about RPMs and how they are created. See the packaging guidelines for openSUSE or a similar document of another supported packaging system such as dpkg. This document is not meant to be a replacement for packaging documentation, which can be found at the above links.
You should be familiar with the source code environment your project is using for your package. The Build Service can work around some common mistakes and will try to guide you in case of failures. We have a buildservice-mailinglist which can be a source of help and advice. However, decisions on which patches to apply, what compiler flags to use, etc. are ultimately up to you.
Requirements
To make full use of the Build Service, you need to login with your openSUSE/SUSE account (same as wiki, bugzilla...). If you have no account yet, click on the "sign up" link at the top of the page to create one. Keep in mind that if you change your password someday that you also need to change ~/.oscrc or ~/.config/osc/oscrc, and if you fail to do so and run osc commands that involve the server nevertheless, the user account may be blocked after repeated tries with an incorrect password.
Futhermore, if you start the build as normal user (good idea!), you will be asked for the root password of your local machine. You can avoid that if you add your user to /etc/sudoers with the following procedure:
- Run the following command:
sudo /usr/sbin/visudo
- Add the following line and replace the placeholder LOGIN with your login name:
LOGIN ALL = NOPASSWD: /usr/bin/build LOGIN ALL = NOPASSWD: /usr/bin/osc
Terminology
The Build Service contains projects (you can view a list of them). Each project contains the resources needed to build one or more packages (i.e. RPMs/DEBs/etc.). These resources include source archives, patch files, spec files, etc. The output of a project is one or more repositories. A repository is an old familiar concept: simply a bunch of RPMs organized in a directory hierarchy along with some index/meta-data files that make it easy for tools like zypper to search and resolve dependencies. The repositories a project outputs correspond to the different operating system versions such as openSUSE 11.2, etc.
As far as the projects that exist, there are "official" openSUSE projects that build the RPMs provided in the standard openSUSE distributions. The "factory" project is the work-in-progress project that will become the next version of openSUSE. There are also lots of area-specific projects such as Apache and network:telephony. Finally, each user has their own "playground" project named home:username.
RPMs tend to have lots of dependencies on other RPMs, and often these RPMs come from different projects within the Build Service. This has two important implications.
First, if your package depends on some other package at runtime ("Requires"), it will often also depend on it at build time (i.e., "BuildRequires"). The Build Service does not automatically go search and find build dependencies for you (other than what is included in the standard distribution you are building for). So somehow you have to tell the Build Service where to get the required package.
Secondly, a nice goal is for a repository to be transitively closed, i.e. any dependency of any package in the repository is also in the repository (packages in the standard distribution excepted). This makes life easier for people installing the RPMs your project provides. But this is not required: users can always find these dependencies manually using the search interface.
The Build Service provides a few different ways to facilitate handling of these dependencies.
First, you can directly add the required package(s) to your repository. This is certainly the approach you must take if no other project builds the required packages. Typically though, those packages are already being built by some other project. Consider re-using their work.
The second option is to link the other project's repository to your repository. This is called layering. It is done by editing the meta-data of your project. Add the other project/repository as an additional path. This simply adds to the list of repositories in which the Build Service will search for "BuildRequires" dependencies at build time. This will allow your package's build to succeed, addressing the first problem, but it does not address the "transitively closed" goal at all: users will have to go fetch the required package themselves. However, this is a good choice when there are several dependencies from your project into another project and/or users are likely to be pulling from both repositories anyway.
The third option is called linking and is a way to allow your project to re-use a package that already exists in another project. When you link a package into your project, dependent packages will be able to build, and the package will also appear in your project's repository, therefore solving both problems without duplicating any work.
There are two types of linking: link and aggregate. When you link, you can optionally modify the way the package is built. You can add patches, and enable the build for additional repositories. Your build of the package's RPM will have a different build number from the original project's build of it. Note, this could cause confusion for users. Really, you are building a different version of a package with the same name.
Unless you need to modify the required package, you should aggregate instead of link. When you aggregate, you are performing a "read-only" link. The package is not built within your project; instead, the already-built package is copied into your project from the original project. So the same RPM (with the same build number) will appear in both projects. For the user, there is less confusion because the RPMs are identical.
The Build Service automatically detects changes in linked packages and triggers rebuilds of any packages depending on them.
Workflow
The following steps outline a normal workflow to create a project and add packages to it. Of course, in a real world example you might fail at some steps and have to repeat it until it does not fail anymore. This outline is just to give you a feeling about what we are trying to achieve.
We will show you two different ways if possible:
- the Web client way
- the Command line client way
- An alternate method is the MonoOSC client way, explained in the MonoOSC guide, so no need to duplicate.
Step One – Login and one time Local Project setup
If you already have an openSUSE Account, this is the easiest step.
- Web client: Open http://build.opensuse.org/ and use the login link on the upper right to login. After that, your home project is available by clicking on your username.
- Command line: At first, you have to install the command-line client on your machine. You can find osc packages for different distributions in the openSUSE-Tools software download repository (yes: this is also a Build Service Project). Use your favorite package manager to install the osc package.
Assuming that you are using openSUSE Tumbleweed, installation of the osc package looks like:
zypper in osc
Afterwards, cd into the directory you want to use for your project files. Now everybody familiar with SVN will feel "at home": try to checkout your home project using
cd <directory_to_contain_project_root> osc checkout home:<username> cd home:<username>
- (please replace <username> with your login).
- You will be prompted for your username and password — afterwards, osc will try to checkout packages in your home project and create a new directory called home:<username>.
- You can edit your settings in the file ~/.oscrc or ~/.config/osc/oscrc.
- If you enter wrong password you need to delete the whole api section for the service with wrong password for osc to ask for password again.
- If you set plaintext_passwd = 0 and delete the api section osc asks for username and password again and stores the password base64 encoded. It can be decoded easily but is not seen at a glance.
- If you want to use multiple build services (or service other than https://api.opensuse.org) use the -A option. You can shorten the -A argument by setting aliases option in the api section. You can also write a my_buildservice script saying something like exec osc -A https://api.my.build.server "$@"
Step Two – Create & Upload packages
You can use your home project as a "playground" to test packages which will be transferred to other, more visible projects if everything is alright.
- Web client: On the right side click on "Home Project" to open your home project, then click on "Users" and then "Add user" to add yourself as a maintainer. After that, click on "create new package" in the Overview > Packages tab. You should fill out the following three textfields: "Name" (mandatory), "Title" and "Description". Just use the package name as "Name", the package summary as "Title" and the package description as "Description".
- After the package is created, go to the "Sources" tab to add the files for your package. You need to upload the source code of your package and at least a spec file (see also packaging guidelines).
- Command line:
osc meta pkg -e home:<username> <packagename>
osc will open a template xml file in your favorite editor (based on the EDITOR environment variable) and you can just add the same things (Name, Title and Description) as described above.
Now call:
osc co home:<username> <packagename>
To add files via the command line, just cd into the project directory, copy the relevant files (typically a .tar.xz and support files).
openSUSE RPM packages have their build instructions in a specfile. See the packaging guidelines how to create this. An easier approach is to copy and adapt a specfile from a similar package, or from within the tar-ball, if available. When the files are ready call
osc add *
this will mark the files in the directory for the next submit. To submit the files, call
osc commit
A commit automatically triggers the build process. You may want to delay the commit, until after you successfully built the package locally, see below.
Step Three – Choose build targets
Now you have to select for which distributions (e.g. openSUSE 13.1, Ubuntu 14.04 etc.) your packages should get built.
- Web client: Go to the "Repositories" tab on your project, and click on Add repositories and choose one of the available distributions and architectures.
- Command line: There is no difference between distribution and any other project in OBS. You need to find the right project name in the list of projects. For openSUSE it can be openSUSE:Factory, openSUSE:13.1, SUSE:SLE-11:SP3 and so on.
$ osc ls / ... openSUSE:Factory openSUSE:Factory:ARM openSUSE:Factory:ARM:Live ...
OBS server administrator can also configure a public list of distributions to show to users. If there is such list, you can get it with this command.
$ osc dists
After you've found the project to build against, get a list of its repositories:
$ osc repositories openSUSE:Factory standard x86_64 standard i586 snapshot x86_64 snapshot i586 ports ppc64le ports ppc64 ports ppc ports armv6l ports armv7l ports aarch64 images local images i586 images x86_64
then add repositories to your project by editing your project metadata:
osc meta prj -e home:<username>
and add the repository like:
<repository name="openSUSE_Factory"> <path project="openSUSE:Factory" repository="standard" /> <arch>x86_64</arch> <arch>i586</arch> </repository>
The repository="standard" is just for future extensions (forks of a repository).
Step Four – Build your package
Your package is scheduled for build automatically after it is committed or some files have changed. If a required package is rebuilt, your package will automatically be rebuilt, too.
You can also manually trigger a rebuild if you need:
osc rebuildpac <project> <package> [<repo> [<arch>]]
With the optional <repo> and <arch> arguments, the rebuild can be limited to a certain repository or architecture.
If your project is named home:username, you can now find your project at http://download.opensuse.org/repositories/home:/username/
Build your package locally
Sometimes, it can be faster to build your package on your local machine instead of waiting for the results from the Build Service. osc supports local builds of your package if your local hardware supports it (on x86_64 you can build for i586 and x86_64, on i586 only for i586).
Ensure you have the latest sources
Use osc checkout (osc co) or osc up to ensure you have the latest version of the source.
If it is project/package you havn't checkout at all:
cd <your_obs_working_dir> osc co <project> <package> cd <project>/<package>
or a new package of an existing checked out project
cd <your_obs_working_dir>/<project> osc co <package> cd <package>
or enter an existing local copy of the package and update
cd <your_obs_working_dir>/<project>/<package> osc up
Perform the local build
osc build <platform> <arch> <specfile> [--clean|--noinit]
for example
~/obs/home:user/my_project/my-package # osc build openSUSE_Leap_42.1 x86_64 my-package.spec
osc will connect to the OBS repository server and download all needed RPMs to /var/tmp/osbuild-packagecache/platform/repository/arch as cache directory. If you want to avoid network traffic, its possible to fill the cache beforehand with rpms from a DVD or iso. For that, copy the rpms from the DVD to the cache dir.
osc will create a chroot environment in /var/tmp/build-root/ and start the build of your package. If you only have minor changes, you can avoid the re-creation of the build environment with the option --noinit. If you suspect that your chroot environment is broken, you can trigger a complete rebuild with the option --clean. You can configure the chroot directory; see the comments in your ~/.oscrc or ~/.config/osc/oscrc file.
sudo rpm --import - <<_END_KEY $(osc signkey offending-project) _END_KEY
After your packages are built in this chroot environment, you can find the resulting packages in /var/tmp/build-root/home/abuild/rpmbuild/RPMS/ (older versions of rpmbuild use /usr/src/packages/RPMS/.
If your package uses a URL download service, you may have to execute the following command first:
zypper ar -r http://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_11.3/openSUSE:Tools.repo
The complete log file of your local build is stored in /var/tmp/build-root/.build.log.
Correct Errors in the Local Build Process
The main reason why you would need to compile a new package for openSUSE or any other distro is to assert compatibility if your package has not yet been compiled for your operating system version and release. However, that way you may encounter new errors in the build process which need to be fixed. The easiest way to fix errors is to chroot to the build environment and create a fix there. You may want to use openroot instead of chroot in order to get X11 access and all the other necessary directories mounted.
osc chroot openSUSE_12.1 x86_64
or old-fashioned and cumbersome
chroot /var/tmp/build-root/ cd /home/abuild/rpmbuild/BUILD/your-package-dir ls or: openroot /var/tmp/build-root/ 'cd /home/abuild/rpmbuild/ILD/your-package-dir; ls; bash' ... exit
Dependencies
If you get a dependency error during your build, add a line containing the build dependencies, like:
BuildRequires: cmake libkde4-devel
In this case, cmake and libkde4-devel will be installed before your package is built.
Install extra packages to build root
For debugging purposes, you might need to install extra packages to your local build root to debug and fix build related problems. This can be done through the ~/.oscrc file and variable extra-pkgs. For example:
extra-pkgs = vim gdb strace valgrind
Install privileges
If you get an error message like this:
error: Bad exit status from /var/tmp/rpm-tmp.qrRAn2 (%install)
this means that your %install step has failed (and all others before went well). This can be because of missing write privileges if you try to install to the wrong place. In this case add the following make install command to your spec file:
make install DESTDIR=%buildroot
Submit your work back to OBS
Once you have your <package> directory the way you want it, use the below commands to submit your work back to OBS.
add a new file to the package
osc add
remove a file from the package
osc rm
update the change log (ie. *.changes)
osc vc
submit your updated files back to OBS
osc commit
Patches
Whenever possible, try to work upstream to make any changes to the source rather than patching up a service or application. But sometimes it will be necessary to create a patch to include a fix that is unavailable upstream or is specific to a SUSE deployment.
There are several ways how to create a new patch. For instance you can use a simple tool called diff or advanced tool quilt. If the source code is available in git, then the git command line can also produce a patch.
For more information about patches visit patches guidelines.
Using diff
If you plan to patch a file, copy it before editing to .orig, retry the desired step in the build process until it succeeds and then create a patch for it.
To make the build more verbose you may want to insert a set -x in your specfile making bash recite all executed commands (set +x disables reciting afterwards).
diff -Pdpru /var/tmp/build-root/home/abuild/rpmbuild/BUILD/your-package-dir/Makefile.orig \
/var/tmp/build-root/home/abuild/rpmbuild/BUILD/your-package-dir/Makefile \
>/osc/home:user/your-package-dir/my.patch
Now add the patch to the specfile by listing Patch67: my.patch (where 67 is usually replaced by the lowest unoccupied number of the patch) in the header. Then let it be applied at the appropriate position (usually %setup) in the build process by %patch67 -p7 (-p7 strips seven directory levels if you have not manually edited the file directories in the header of the patch file).
Using git --format-patch
If the change is contained in a single commit that is on the head of a git tree, a patch may be generated by running this command:
git format-patch HEAD^
This will produce a file 0001-<first line of commit message>.patch. This file can be included in the .spec file as described in the Using diff section above. Note that the paths in the patch will be one level deep, so use -p1 in the .spec file.
Using quilt
You may find it easier to use a special program for automatic patch generation like quilt:
osc co yourproject/yourpackage
cd yourproject/yourpackage
quilt setup -v *spec
cd yourpackage-*/
quilt push -a # apply old patches
quilt new yourpackage-version_fixbuild.patch
quilt edit src/foo.c
quilt refresh
foo-fixbuild.patch will automatically be created in the parent dir. If you work on a package which doesnt have a patch yet. You have to remember to copy the patch from the patch directory to your package directory. Rerun quilt setup to get an initial patch. You can remove patches from your working copy with quilt pop.
An example .quiltrc file:
# Options passed to GNU diff when generating patches
QUILT_DIFF_OPTS="--show-c-function"
# QUILT_DIFF_OPTS=""
# Options passed to GNU patch when applying patches
#QUILT_PATCH_OPTS="--ignore-whitespace --unified-reject"
# Options to pass to commands (QUILT_${COMMAND}_ARGS)
QUILT_PUSH_ARGS="--color=auto"
QUILT_DIFF_ARGS="--color=auto"
QUILT_REFRESH_ARGS="--backup -p0"
QUILT_PATCH_OPTS="--unified-reject-files --backup"
Step Five: Check the logfiles
The buildservice produces one big logfile for each build of a package.
- Web client: Just click on the status of the Build result tab in the package view.
- Command line: You have a few choices depending on your needs (
projectandpackageare optional if you are in the package directory):
osc prjresults [<project>]
Shows the aggregated build results of an entire project. Or you can do:
osc results [<project> <package>]
Shows the build results of a single package.
osc buildlog <platform> <arch>
Shows the log file from a package (you need to be inside a package directory).
Create Patterns
Since Code12 (SLES-12/openSUSE-13.2) we utilize a new method of defining patterns, as opposed to the older XML-based patterns files. A pattern and it's dependencies are now represented by a rpm package. The few pattern specific properties are provided by this patterns-package. No extra XML-file is needed.
Some more details about the patterns-packages can be found here and in the project obs://build.opensuse.org/system:install:head/patterns-openSUSE maintaining the openSUSE patterns.
OBSOLETE SINCE SLES-12/openSUSE-13.2: The part below refers to the old way od defining patterns with XML-files:
Patterns are files which contain a list of packages together with a description of what they are useful for. Additionally the Build Service creates .ymp files for each generated repository pattern. These .ymp files can be used for a One Click Install by the user.
In short, patterns are useful for installing a set of software for a typical need without creating dependencies between packages.
Submitting patterns is possible using the api directly, or using osc:
- to open a pattern in $EDITOR (creating it if it doesn't exist yet)
osc meta pattern -e <project> <pattern>
- to list existing patterns
osc meta pattern <project>
- get an existing pattern
osc meta pattern <project> <pattern>
- You can also submit an existing file as below:
osc meta pattern --file <local_file> <project> <pattern>
To test: clicking on the .ymp in konqueror should launch the installer, if you do not have konqueror installed, you can try launching from shell as normal user:
/sbin/yast2 MetaPackageHandler http://download.opensuse.org/repositories/<project>/<SUSE_Factory or openSUSE_10.2>/<pattern>.ymp
The following file is an example pattern file from the KDE:KDE4 project. You can see the generated .ymp file from it here.
<pattern
xmlns="http://novell.com/package/metadata/suse/pattern"
xmlns:rpm="http://linux.duke.edu/metadata/rpm"
>
<name>KDE 4 Games</name>
<summary>KDE 4 Games</summary>
<description>A number of games for KDE 4.</description>
<uservisible/>
<category lang="en">Desktop Functions</category>
<rpm:recommends>
<rpm:entry name="kde4-kpat"/>
<rpm:entry name="kde4-kmahjongg"/>
<rpm:entry name="kde4-kmines"/>
<rpm:entry name="kde4-kreversi"/>
<rpm:entry name="kde4-ksudoku"/>
</rpm:recommends>
<rpm:suggests>
<rpm:entry name="kde4-katomic"/>
<rpm:entry name="kde4-kbattleship"/>
<rpm:entry name="kde4-ksquares"/>
<rpm:entry name="kde4-bovo"/>
<rpm:entry name="kde4-kiriki"/>
<rpm:entry name="kde4-kwin4"/>
<rpm:entry name="kde4-kolf"/>
<rpm:entry name="kde4-klines"/>
<rpm:entry name="kde4-ksame"/>
<rpm:entry name="kde4-lskat"/>
<rpm:entry name="kde4-kgoldrunner"/>
<rpm:entry name="kde4-kblackbox"/>
<rpm:entry name="kde4-kbounce"/>
<rpm:entry name="kde4-ktuberling"/>
<rpm:entry name="kde4-knetwalk"/>
<rpm:entry name="kde4-kjumpingcube"/>
<rpm:entry name="kde4-kspaceduel"/>
<rpm:entry name="kde4-konquest"/>
<rpm:entry name="kde4-kshisen"/>
</rpm:suggests>
</pattern>
Some Tag descriptions:
| Tag | Description |
|---|---|
<rpm:requires> <rpm:entry name="example" /> </rpm:requires> |
Requires RPM example: this package must be installed - otherwise the pattern is not fulfilled. |
<rpm:recommends> <rpm:entry name="example" /> </rpm:recommends> |
Recommends RPM example: if available and all dependencies of this package are fulfilled, the package would be installed. If the package is not available, there are not error messages. If the package dependencies are not met, the package would be visible but not installed. |
<rpm:suggests> <rpm:entry name="example" /> </rpm:suggests> |
Suggests RPM example: would be shown in the pattern but not installed per default |
A start to end example of a simple change
The goal here is to have a specific example that can be used as a tutorial.
A common activity is to branch an existing package in an existing project, make a change and submit that change back to the original package.
The basic steps are:
- Branch the original package: osc branch <original_project> <original_package> This creates a new branch project that is specific for you named home:<your_user_name>:branches:<original_project_name> and therein it creates a new package with the same name as original package that is basically a copy of the original package.
- Checkout the branched package: osc checkout home:<your_user_name>:branches:<original_project_name>/<original_package_name> This downloads the source files of the branched package from the server into a local sub-directory named home:<your_user_name>:branches:<original_project_name>/<original_package_name>
- Go to the local sub-directory: cd home:<your_user_name>:branches:<original_project_name>/<original_package_name> and set the usual default umask umask 0022
- Work on your local copy of the package until it works for you:
- Change the local source files (e.g. edit the specfile or create a patch)
- Perform a local build
- Do a local package installation
- Test the local installed package
- Update the changes file: osc vc This opens an editor (usually 'vi') and creates an appropriate header for a RPM changes entry. If there was a bug report it must be referenced as bsc#123456
- If you added new files (e.g. new patches) or removed files (e.g. obsolete patches), update the version control status of the local source files: osc addremove and verify with osc status that there are no files with problematic version control status like '?' or '!'
- Commit the local source files into the branched package: osc commit This uploads the local source files into the branched package on the server and that triggers an automated re-build of the branched package.
- Review the build results of the branched package for all build repositories that are enabled for the original package: osc results --verbose home:<your_user_name>:branches:<original_project_name> <original_package_name> To list the build repositories that are enabled for the original package get its build results: osc results <original_project> <original_package>
- If the re-build of the branched package "succeeded" for all build repositories that are enabled for the original package, create a request to submit the branched package back to the original package: osc submitrequest --message='< a short message that describes what you changed plus bsc#123456 if there exists a matching bug report>' home:<your_user_name>:branches:<original_project_name> <original_package_name> <original_project> <original_package> and remenber the request ID number.
- From time to time check what happened with your request: osc request show <request_ID_number> If you need to get in direct contact with the maintainers of the original package: osc maintainer <original_project> <original_package> shows their user names and osc whois <user_name> shows fullname and email of a buildservice user.
It's a few steps, but once you get a hang on it, it becomes second nature.
This assumes you already have an account on OBS. If not, go ahead and set that up at http://build.opensuse.org/ and use the login link on the upper right to login. OBS uses the same authentication system as the rest of the openSUSE infrastructure like bugzilla, so you likely already have an account, you just need to login for the first time and the OBS account home project will be automatically created.
Here's a real example:
One time setup from the command line
sudo zypper in osc mkdir ~/obs
Then you need to branch a local copy of a package.
cd ~/obs umask 0022 osc branch security:forensics sleuthkit osc co home:<your_user_name>:branches:security:forensics/sleuthkit cd ~/obs/home:<your_user_name>:branches:security:forensics/sleuthkit
Now that you have an existing package already, it's as easy as:
# this untars the tarball and does some magic quilt setup sleuthkit.spec # chdir to sources cd sleuthkit-3.2.3 # apply all patches quilt push -a # add new patch quilt new testing.patch # add a file to the patch quilt add some-file vi <some-file> # or alternatively: quilt edit <some-file> # and finally quilt refresh -p1 # copy the patch up to the project dir cp patches/testing.patch .. # and now handle the spec file cd .. vi sleuthkit.spec # Add patch to it by creating a Patch0 entry in the header area and a %patch0 -p1 line to the %prep section # tell OBS that the package now includes a new file osc add testing.patch # update the changes file osc vc -m "Fix some typos." # and build, install and test your work osc build # perform a local install. At the end of the osc build output the full path of the RPM file should be shown zypper in -f <full_path_to_rpm> # repeat until happy. Go back to quilt edit if not happy. # send your edits back to OBS osc commit # wait a period of time for new packages to build on OBS # check the build status via the WebUI for your branched package # Once published, install the RPM from OBS and test again # submit your changes back to the original package. If there is a bugzilla entry, be sure to reference it
When you're done testing, submit the change back to the original package you branched
osc sr