Build Service/Tutorial

Prerequisites

You should have a general understanding about RPMs and how they are created (see also packaging conventions for SUSE), or else a similar understanding of another supported packaging system such as dpkg. This document is not meant to be a replacement for packaging documentation, which can be found at the above links.

You should be familiar with the source code environment your project is using for your package. The Build Service can work around some common mistakes and will hopefully alert you to some problems in your source code via logfiles and simple build failures. We have a buildservice-mailinglist which can be a source of help and advice. However, decisions on which patches to apply, what compiler flags to use, etc. are ultimately up to you.

Requirements

To make use of the Build Service you first need an account to login. ;-)

Your buildservice account is the same as your wiki/bugzilla account. If you have no account see the link at the top of the page to create one.

After you have created your wiki/bugzilla login, you go to http://build.opensuse.org/ and login there. At the first login you need to fill out a request form and explain why you want an account and what you want to package. Then you have to wait until your account is granted. While waiting you can subscribe to the opensuse-buildservice mailinglist and give a short self-introduction. Just tell us your Name and something about the project you want to create and maintain (some words about your packaging skill would also be welcome).

If you want to test and/or build your packages at home (always a good idea!) you should have a look at the SUSE Build Tutorial to get a first overview about the requirements for a local build environment .

Terminology

The Build Service contains projects (you can view a list of them (login required)). Each project contains the resources needed to build one or more packages (i.e., RPMs). These resources include source archives, patch files, spec files, etc. The output of a project is one or more repositories. A repository is an old familiar concept: simply a bunch of RPMs organized in a directory hierarchy along with some index/meta-data files that make it easy for tools like zypper to search and resolve dependencies. The repositories a project outputs correspond to the different operating system versions such as SuSE 10.3, etc.

As far as the projects that exist, there are "official" SuSE projects that build the RPMs provided in the standard SuSE distributions. The "factory" project is the work-in-progress project that will become the next version of SuSE. There are also lots of area-specific projects such as Apache and network:telephony. Finally, each user gets their own "playground" project named home:username.

RPMs tend to have lots of dependencies on other RPMs, and often these RPMs come from different projects within the Build Service. This has two important implications.

First, if your package depends on some other package at runtime ("Requires"), it will often also depend on it at build time (i.e., "Build-Requires"). The Build Service does not automatically go search and find build dependencies for you (other than what's included in the standard distribution you're building for). So somehow you have to tell the Build Service where to get the required package.

Secondly, a nice goal is for a repository to be transitively closed, i.e., any dependency of any package in the repository is also in the repository (packages in the standard distribution excepted). This makes life easier for people installing the RPMs your project provides. But this is not required: users can always find these dependencies manually using the search interface.

The Build Service provides a few different ways to facilitate handling of these dependencies.

First, you can directly add the required package(s) to your repository. This is certainly the approach you must take if no other project builds the required packages. Typically though, those packages are already being built by some other project. How then can you re-use their work?

The second option is to link the other project's repository to your repository. This is done by editing the meta-data of your project. What this does is simply add the other repository to the list of repositories in which the Build Service will search for "Build-Requires" dependencies at build time. This will allow your package's build to succeed, addressing the first problem, but it doesn't address the "transitively closed" goal at all: users will have to go fetch the required package themselves. However, this is a good choice when there are several dependencies from your project into another project and/or users are likely to be pulling from both repositories anyway.

The third option is called linking and is a way to allow your project to re-use a package that already exists in another project. When you link a package into your project, dependent packages will be able to build, and the package will also appear your project's repository, therefore solving both problems without duplicating any work.

There are two types of linking: link and aggregate. When you link, you can optionally modify the way the package is built. Your build of the package's RPM will have a different build number from the original project's build of it. Note, this could cause confusion for users. Really, you are building a different version of a package with the same name.

Unless you need to modify the required package, you should aggregate instead of link. When you aggregate, you are performing a "read-only" link. The package is not built within your project; instead, the already-built package is copied into your project from the original project. So the same RPM (with the same build number) will appear in both projects. For the user, there is less confusion because the RPMs are identical.

The Build Service automatically detects changes in linked packages and triggers rebuilds of any packages depending on them.

Workflow

The following steps outline a normal workflow to create a project and add packages to it. Of course in a real world example you might fail at some step and have to repeat it until it does not fail anymore. This outline is just to give you a feeling what we are trying to achieve.

We'll show you two different ways if possible:

• the Webclient way
• the Commandline Client way (we'll use osc for our example)

Step One - Login and one time Local Project setup

If you already have your Build Service Account, this is the easiest step.

• Webclient: Each and every time you want to interact with your project via the webclient you will need to login.

To do so, open http://build.opensuse.org/ and click on one of the links "List of All Projects" or "Watched Projects". You'll be prompted for your username and password for the openSUSE Build Service. After that, you'll see the list of Projects and in the upper right corner you can find a link to your "Home Project". Please follow that link.

• Commandline: If you are ever going to interact with your project via CLI, then you must install the software tools and create your local working directory.

At first, you have to install the Commandline Client on your local Client. You can find osc-packages for different distributions in the openSUSE-Tools software download repository (yes: this is also a Build Service Project). Use your favorite package manager to install the osc package. Note: for SUSE Linux 9.3 or older you additionally need python-elementtree and python-urlgrabber from here.

Afterwards "cd" into the directory you want to use for your project files. Now everybody familiar with SVN will feel "at home": try to checkout your home project using

 cd <directory_to_contain_project_root>
 osc checkout home:<username>
 cd home:<username> 

(please replace <username> with your login). You'll be prompted for your username and password - afterwards osc will try to checkout packages in your home project and create a new directory called home:<username>. You can edit your settings in the file ~/.oscrc.

Step Two - Create & Upload packages

It's not really necessary to start in your home project - as you can also upload packages directly to an existing project - but it's a good starting point and no-one should use a home project of another user as build dependency. So you can use your home project as a "playground" to test packages which will be transferred to other projects if everything is alright.

• Webclient: Just click on the [Add Package] Link. You should fill out the following three textfields: "Name", "Title" and "Description". Just use the package name as "Name", the package summary as "Title" and the package description as "Description".

Afterwards click on the [Add File] link to add the files for your package. You need to upload the source code of your package and at least a spec file.

CLARIFICATION what is required for debian packages?

• Commandline:

osc meta pkg -e home:<username> <packagename>

osc will open a template xml file in your favorite editor (based on the EDITOR environment variable) and you can just add the same things (Name, Title and Description) as described above.

Now call

osc up

and you'll get a new directory with the name of your new package. To add files via the command line, just 'cd in the new folder, copy the relevant files (typically a tar.gz and support files) and call

osc add *

this will mark the files in the directory for the next submit. To submit the files, call

osc commit

Step Three - Add Repositories

Return to your OBS home page. Now you should make a decision, for which distribution repositories you will build your package.

• Webclient: Just click on [Add Repository] and choose one of the available Distributions and Architectures.

• Commandline: first get a list of available repositories

osc ls

then edit your project metadata;

osc meta prj -e home:<username>

and add the repository like:

 <repository name="SUSE_Linux_Factory">
   <path project="openSUSE:Factory" repository="standard" />
   <arch>x86_64</arch>
   <arch>i586</arch>
 </repository>

Note: the repository="standard" is just for future extensions (forks of a repository).

Step Four: Build your package

Normally your package is scheduled for build after it is committed or some files have changed. If a required package is rebuilt, your package will automatically be rebuilt, too. You can also manually trigger a rebuild if you need (but please don't do this too often to save build power for the other packages).

Build your package locally

Sometimes it could be faster to build your package on your local machine instead of waiting for the results from the buildservice. osc supports local builds of your package, if your local hardware supports it (on x86_64 you can build for i586 and x86_64, on i586 only for i586).

 osc build <platform> <arch> <specfile> [--clean|--noinit]

If you start the build as normal user (good idea!), you will be asked for the root password of your local machine. You can avoid that if you add your user to /etc/sudoers and edit your ~/.oscrc :

su-wrapper = sudo

and with visudo add the line (as root):

<your login name> ALL = NOPASSWD: /usr/bin/build

to the sudo configuration (of course without '<' and '>'). osc will connect to the repository-server and download all needed RPMs to /var/tmp/osbuild-packagecache/<plattform>/<repository>/<arch> as cache directory. (So if you already have a complete repository, you can link the RPMs in this directory to avoid huge download traffic.)

For example for openSUSE_10.2 repository you can use Retail Box DVD iso as below:

 mount openSUSE-10.2.iso /mnt/openSUSE-10.2 -o loop
 mkdir -p /var/tmp/osbuild-packagecache/openSUSE\:10.2
 ln -s /mnt/openSUSE-10.2/suse/* /var/tmp/osbuild-packagecache/openSUSE:10.2/standard

The above will give you repositories for x86 as well as x86_64

Packages can be now built locally like this:

 osc build openSUSE_10.2 i586 beryl-core-snapshot.spec

osc will create a chroot environment in /var/tmp/osc-build-root/ and start the build of your package. If you only have minor changes, you can avoid the re-creation of the build environment with the option --noinit. If you suspect that your chroot environment is broken, you can trigger a complete rebuild with the option --clean. You can configure the chroot directory; see the comments in your ~/.oscrc file.

After your packages are built in this chroot environment, you can find the resulting packages in /var/tmp/osc-build-root/usr/src/packages/RPMS/.

The complete log file of your local build is stored in /var/tmp/osc-build-root/.build.log.

Build your package in the build service

Building in the build service is easier than building local - but perhaps takes longer.

• Webclient: If you want to trigger a rebuild manually, just click on [Trigger Rebuild] on the bottom of the package site.

• Commandline: With the optional <repo> and <arch> arguments, the rebuild can be limited to a certain repository or architecture.

osc rebuildpac <project> <package> [<repo> [<arch>]]

To write your own .spec file, you should read the corresponding HOWTO. Then you have to upload that.

Step Five: Check the logfiles

The buildservice produces one big logfile for each build of a package.

• Webclient: Just click on the link [Build Log] in the package view.

• Commandline: You have a few choices depending on your needs (packagedir is optional if you are in the package directory):

osc prjresults [packagedir]

Shows the aggregated build results of an entire project. Or you can do:

osc results [packagedir]

Shows the build results of a single package.

osc buildlog <platform> <arch>

Shows the log file from a package (you need to be inside a package directory).

Create Patterns

Patterns are files which contain a list of packages together with a description of what they are useful for. (In the future, the Build Service will export patterns to repositories so they become visible in YaST.) Additionally the Build Service creates .ymp files for each generated repository pattern. These .ymp files can be used for a One Click Install by the user.

In short, patterns are useful for installing a set of software for a typical need without creating dependencies between packages.

Submitting patterns is possible using the api directly, or using osc:

• to open a pattern in $EDITOR (creating it if it doesn't exist yet)

osc meta pattern -e <project> <pattern>

• to list existing patterns

osc meta pattern <project>

• get an existing pattern

osc meta pattern <project> <pattern>

• You can also submit an existing file as below:

osc meta pattern --file <local_file> <project> <pattern>

To test: clicking on the .ymp in konquerer should launch the installer, if you do not have konqueror installed, you can try launching from shell as normal user:

/sbin/yast2 MetaPackageHandler http://download.opensuse.org/repositories/<project>/<SUSE_Factory or openSUSE_10.2>/<pattern>.ymp

The following file is an example pattern file from the KDE:KDE4 project. You can see the generated .ymp file from it here.

<pattern
 xmlns="http://novell.com/package/metadata/suse/pattern"
 xmlns:rpm="http://linux.duke.edu/metadata/rpm"
>
    <name>KDE 4 Games</name>
    <summary>KDE 4 Games</summary>
    <description>A number of games for KDE 4.</description>
    <uservisible/>
    <category lang="en">Desktop Functions</category>
    <rpm:recommends>
      <rpm:entry name="kde4-kpat"/>
      <rpm:entry name="kde4-kmahjongg"/>
      <rpm:entry name="kde4-kmines"/>
      <rpm:entry name="kde4-kreversi"/>
      <rpm:entry name="kde4-ksudoku"/>
    </rpm:recommends>
    <rpm:suggests>
      <rpm:entry name="kde4-katomic"/>
      <rpm:entry name="kde4-kbattleship"/>
      <rpm:entry name="kde4-ksquares"/>
      <rpm:entry name="kde4-bovo"/>
      <rpm:entry name="kde4-kiriki"/>
      <rpm:entry name="kde4-kwin4"/>
      <rpm:entry name="kde4-kolf"/>
      <rpm:entry name="kde4-klines"/>
      <rpm:entry name="kde4-ksame"/>
      <rpm:entry name="kde4-lskat"/>
      <rpm:entry name="kde4-kgoldrunner"/>
      <rpm:entry name="kde4-kblackbox"/>
      <rpm:entry name="kde4-kbounce"/>
      <rpm:entry name="kde4-ktuberling"/>
      <rpm:entry name="kde4-knetwalk"/>
      <rpm:entry name="kde4-kjumpingcube"/>
      <rpm:entry name="kde4-kspaceduel"/>
      <rpm:entry name="kde4-konquest"/>
      <rpm:entry name="kde4-kshisen"/>
    </rpm:suggests>
</pattern>

Some Tag descriptions:

<rpm:requires>
<rpm:entry name="example" />
</rpm:requires>

<rpm:recommends>
<rpm:entry name="example" />
</rpm:recommends>

<rpm:suggests>
<rpm:entry name="example" />
</rpm:suggests>

See also

• General Tips and Tricks for the buildservice

• How to contribute your changes to someone else packages

• Detailed description of different Workflows

• Build Service News

• Cross distribution package how to

• Writing osc plugins

• Doing A Self Check/QA for RPMS