Home Wiki > Difference between revisions of "openSUSE:Build Service Tutorial"
Sign up | Login

Difference between revisions of "openSUSE:Build Service Tutorial"

tagline: From openSUSE

m (Requirements)
m
Line 277: Line 277:
  
 
* [[openSUSE:Build_Service_Collaboration|How to contribute your changes to someone else packages]]
 
* [[openSUSE:Build_Service_Collaboration|How to contribute your changes to someone else packages]]
 +
 +
* [[openSUSE:Build_Service_SubmitPackage|Submit package to upstream]]
  
 
* [[openSUSE:Build_Service_cross_distribution_howto|Cross distribution package how to]]
 
* [[openSUSE:Build_Service_cross_distribution_howto|Cross distribution package how to]]

Revision as of 18:40, 22 October 2010

This document should give an overview of the Build Service and a tutorial on how to build packages for different distributions using this great tool. We'll try to show all actions on an example application so you can follow the steps to produce your own packages.

Prerequisites

You should have a general understanding about RPMs and how they are created. See the packaging guidelines for openSUSE or a similar document of another supported packaging system such as dpkg. This document is not meant to be a replacement for packaging documentation, which can be found at the above links.

You should be familiar with the source code environment your project is using for your package. The Build Service can work around some common mistakes and will try to guide you in case of failures. We have a buildservice-mailinglist which can be a source of help and advice. However, decisions on which patches to apply, what compiler flags to use, etc. are ultimately up to you.

Requirements

To make full use of the Build Service you need to login with your openSUSE/Novell account (same as wiki, bugzilla...). If you have no account yet, click on the 'sign up' link at the top of the page to create one.

Terminology

The Build Service contains projects (you can view a list of them). Each project contains the resources needed to build one or more packages (i.e., RPMs). These resources include source archives, patch files, spec files, etc. The output of a project is one or more repositories. A repository is an old familiar concept: simply a bunch of RPMs organized in a directory hierarchy along with some index/meta-data files that make it easy for tools like zypper to search and resolve dependencies. The repositories a project outputs correspond to the different operating system versions such as openSUSE 11.2, etc.

As far as the projects that exist, there are "official" openSUSE projects that build the RPMs provided in the standard openSUSE distributions. The "factory" project is the work-in-progress project that will become the next version of openSUSE. There are also lots of area-specific projects such as Apache and network:telephony. Finally, each user has his own "playground" project named home:username.

RPMs tend to have lots of dependencies on other RPMs, and often these RPMs come from different projects within the Build Service. This has two important implications.

First, if your package depends on some other package at runtime ("Requires"), it will often also depend on it at build time (i.e., "Build-Requires"). The Build Service does not automatically go search and find build dependencies for you (other than what's included in the standard distribution you're building for). So somehow you have to tell the Build Service where to get the required package.

Secondly, a nice goal is for a repository to be transitively closed, i.e., any dependency of any package in the repository is also in the repository (packages in the standard distribution excepted). This makes life easier for people installing the RPMs your project provides. But this is not required: users can always find these dependencies manually using the search interface.

The Build Service provides a few different ways to facilitate handling of these dependencies.

First, you can directly add the required package(s) to your repository. This is certainly the approach you must take if no other project builds the required packages. Typically though, those packages are already being built by some other project. How then can you re-use their work?

The second option is to link the other project's repository to your repository. This is done by editing the meta-data of your project. What this does is simply add the other repository to the list of repositories in which the Build Service will search for "Build-Requires" dependencies at build time. This will allow your package's build to succeed, addressing the first problem, but it doesn't address the "transitively closed" goal at all: users will have to go fetch the required package themselves. However, this is a good choice when there are several dependencies from your project into another project and/or users are likely to be pulling from both repositories anyway.

The third option is called linking and is a way to allow your project to re-use a package that already exists in another project. When you link a package into your project, dependent packages will be able to build, and the package will also appear your project's repository, therefore solving both problems without duplicating any work.

There are two types of linking: link and aggregate. When you link, you can optionally modify the way the package is built. Your build of the package's RPM will have a different build number from the original project's build of it. Note, this could cause confusion for users. Really, you are building a different version of a package with the same name.

Unless you need to modify the required package, you should aggregate instead of link. When you aggregate, you are performing a "read-only" link. The package is not built within your project; instead, the already-built package is copied into your project from the original project. So the same RPM (with the same build number) will appear in both projects. For the user, there is less confusion because the RPMs are identical.

The Build Service automatically detects changes in linked packages and triggers rebuilds of any packages depending on them.

Workflow

The following steps outline a normal workflow to create a project and add packages to it. Of course in a real world example you might fail at some step and have to repeat it until it does not fail anymore. This outline is just to give you a feeling what we are trying to achieve.

We'll show you two differents ways if possible:

  • the Webclient way
  • the Commandline Client way
  • An alternate method is the MonoOSC client way, an tutorial already exist here, so no need to duplicate.

Step One - Login and one time Local Project setup

If you already have an openSUSE Account, this is the easiest step.

  • Webclient: Open http://build.opensuse.org/ and use the login link on he upper right to login. After that your home project is available by clicking on your username.
  • Commandline:

At first, you have to install the Commandline Client on your machine. You can find osc packages for different distributions in the openSUSE-Tools software download repository (yes: this is also a Build Service Project). Use your favorite package manager to install the osc package.

Afterwards "cd" into the directory you want to use for your project files. Now everybody familiar with SVN will feel "at home": try to checkout your home project using

 cd <directory_to_contain_project_root>
 osc checkout home:<username>
 cd home:<username> 
(please replace <username> with your login). You'll be prompted for your username and password - afterwards osc will try to checkout packages in your home project and create a new directory called home:<username>. You can edit your settings in the file ~/.oscrc.

Step Two - Create & Upload packages

Your can use your home project as a "playground" to test packages which will be transferred to other, more visible projects if everything is alright.

  • Webclient: Click on your username to open your home project, then click on 'add package' in the packages tab. You should fill out the following three textfields: "Name" (mandatory), "Title" and "Description". Just use the package name as "Name", the package summary as "Title" and the package description as "Description".

After the package is created, go to the 'files' tab to add the files for your package. You need to upload the source code of your package and at least a spec file (see also openSUSE:Packaging_guidelines).

  • Commandline:
osc meta pkg -e home:<username> <packagename>

osc will open a template xml file in your favorite editor (based on the EDITOR environment variable) and you can just add the same things (Name, Title and Description) as described above.

Now call

osc up

and you'll get a new directory with the name of your new package. To add files via the command line, just 'cd in the new folder, copy the relevant files (typically a tar.gz and support files).

openSUSE RPM packages have their build instructions in a specfile. See the packaging guidelines how to create this. An easier approach is to copy and adapt a specfile from a similar package, or from within the tar-ball, if available. When the files are ready call

osc add *

this will mark the files in the directory for the next submit. To submit the files, call

osc commit

A commit automatically triggers the build process. You may want to delay the commit, until after you successfully built the package locally, see below.

Step Three - Choose build targets

Now you have to select for which distributions (eg. openSUSE 11.3, Ubunto 10.04 etc.) your packages should get built.

  • Webclient: Go to the 'repositories' tab on your project, and click on add repositories and choose one of the available Distributions and Architectures.
  • Commandline: first get a list of available repositories
osc ls

then edit your project metadata;

osc meta prj -e home:<username>

and add the repository like:

 <repository name="openSUSE_Factory">
   <path project="openSUSE:Factory" repository="standard" />
   <arch>x86_64</arch>
   <arch>i586</arch>
 </repository>

Note: the repository="standard" is just for future extensions (forks of a repository).

Step Four: Build your package

Your package is scheduled for build automatically after it is committed or some files have changed. If a required package is rebuilt, your package will automatically be rebuilt, too.

You can also manually trigger a rebuild if you need:

osc rebuildpac <project> <package> [<repo> [<arch>]]

'With the optional <repo> and <arch> arguments, the rebuild can be limited to a certain repository or architecture.'

Build your package locally

Sometimes it could be faster to build your package on your local machine instead of waiting for the results from the buildservice. osc supports local builds of your package, if your local hardware supports it (on x86_64 you can build for i586 and x86_64, on i586 only for i586).

 osc build <platform> <arch> <specfile> [--clean|--noinit]

If you start the build as normal user (good idea!), you will be asked for the root password of your local machine. You can avoid that if you add your user to /etc/sudoers and edit your ~/.oscrc :

su-wrapper = sudo

and with visudo add the line (as root):

<your login name> ALL = NOPASSWD: /usr/bin/build

to the sudo configuration (of course without '<' and '>'). osc will connect to the repository-server and download all needed RPMs to /var/tmp/osbuild-packagecache/<plattform>/<repository>/<arch> as cache directory. (So if you already have a complete repository, you can link the RPMs in this directory to avoid huge download traffic.)

For example for openSUSE_11.2 repository you can use Retail Box DVD iso as below:

 mount openSUSE-11.2.iso /mnt/openSUSE-11.2 -o loop
 mkdir -p /var/tmp/osbuild-packagecache/openSUSE\:11.2/standard
 ln -s /mnt/openSUSE-11.2/suse/* /var/tmp/osbuild-packagecache/openSUSE:11.2/standard/

The above will give you repositories for x86 as well as x86_64

Packages can be now built locally like this:

 osc build openSUSE_11.2 i586 <some-package-name>.spec

osc will create a chroot environment in /var/tmp/osc-build-root/ and start the build of your package. If you only have minor changes, you can avoid the re-creation of the build environment with the option --noinit. If you suspect that your chroot environment is broken, you can trigger a complete rebuild with the option --clean. You can configure the chroot directory; see the comments in your ~/.oscrc file.

After your packages are built in this chroot environment, you can find the resulting packages in /var/tmp/osc-build-root/usr/src/packages/RPMS/.

The complete log file of your local build is stored in /var/tmp/osc-build-root/.build.log.

Correct Errors in the Build Process

The main reason why you wanna compile a new package for openSUSE or any other distro is to assert compatibility if your package has not yet been compiled for your operating system version and release. However that way you may encounter new errors in the build process which need to be fixed. The easiest way to fix errors is to chroot to the build environment and create a fix there. You may want to use openroot instead of chroot in order to get X11 access and all the other necessary directories mounted.

 chroot /var/tmp/build-root/
 cd /usr/src/packages/BUILD/your-package-dir
 ls
 or:
 openroot /var/tmp/build-root/ 'cd /usr/src/packages/BUILD/your-package-dir; ls; bash'
 ...
 exit

If you plan to patch a file copy it before editing to .orig, retry the desired step in the build process until it succeeds and then create a patch for it. To make the build more verbose you may want to insert a "set -x" in your specfile making bash prompt all executed commands (set +x disables prompting afterwards).

 diff -Naur /var/tmp/build-root/usr/src/packages/BUILD/your-package-dir/Makefile.orig \
            /var/tmp/build-root/usr/src/packages/BUILD/your-package-dir/Makefile \
     >/osc/home:user/your-package-dir/my.patch

Now add the patch to the .spec-file by listing "Patch67: my.patch" in the header and then let it be applied at the appropriate position (usually %setup) in the build process by "%patch67 -p7" (-p7 strips seven directory levels if you have not manually edited the file directories in the header of the patch file.). You may find it easier to use a special program for automatic patch generation like Quilt.


Step Five: Check the logfiles

The buildservice produces one big logfile for each build of a package.

  • Webclient: Just click on the link [Build Log] in the package view.
  • Commandline: You have a few choices depending on your needs (packagedir is optional if you are in the package directory):
osc prjresults [packagedir]

Shows the aggregated build results of an entire project. Or you can do:

osc results [packagedir]

Shows the build results of a single package.

osc buildlog <platform> <arch>

Shows the log file from a package (you need to be inside a package directory).

Create Patterns

Patterns are files which contain a list of packages together with a description of what they are useful for. Additionally the Build Service creates .ymp files for each generated repository pattern. These .ymp files can be used for a One Click Install by the user.

In short, patterns are useful for installing a set of software for a typical need without creating dependencies between packages.

Submitting patterns is possible using the api directly, or using osc:

  • to open a pattern in $EDITOR (creating it if it doesn't exist yet)
osc meta pattern -e <project> <pattern>
  • to list existing patterns
osc meta pattern <project>
  • get an existing pattern
osc meta pattern <project> <pattern>
  • You can also submit an existing file as below:
osc meta pattern --file <local_file> <project> <pattern>

To test: clicking on the .ymp in konquerer should launch the installer, if you do not have konqueror installed, you can try launching from shell as normal user:

/sbin/yast2 MetaPackageHandler http://download.opensuse.org/repositories/<project>/<SUSE_Factory or openSUSE_10.2>/<pattern>.ymp

The following file is an example pattern file from the KDE:KDE4 project. You can see the generated .ymp file from it here.

<pattern
 xmlns="http://novell.com/package/metadata/suse/pattern"
 xmlns:rpm="http://linux.duke.edu/metadata/rpm"
>
    <name>KDE 4 Games</name>
    <summary>KDE 4 Games</summary>
    <description>A number of games for KDE 4.</description>
    <uservisible/>
    <category lang="en">Desktop Functions</category>
    <rpm:recommends>
      <rpm:entry name="kde4-kpat"/>
      <rpm:entry name="kde4-kmahjongg"/>
      <rpm:entry name="kde4-kmines"/>
      <rpm:entry name="kde4-kreversi"/>
      <rpm:entry name="kde4-ksudoku"/>
    </rpm:recommends>
    <rpm:suggests>
      <rpm:entry name="kde4-katomic"/>
      <rpm:entry name="kde4-kbattleship"/>
      <rpm:entry name="kde4-ksquares"/>
      <rpm:entry name="kde4-bovo"/>
      <rpm:entry name="kde4-kiriki"/>
      <rpm:entry name="kde4-kwin4"/>
      <rpm:entry name="kde4-kolf"/>
      <rpm:entry name="kde4-klines"/>
      <rpm:entry name="kde4-ksame"/>
      <rpm:entry name="kde4-lskat"/>
      <rpm:entry name="kde4-kgoldrunner"/>
      <rpm:entry name="kde4-kblackbox"/>
      <rpm:entry name="kde4-kbounce"/>
      <rpm:entry name="kde4-ktuberling"/>
      <rpm:entry name="kde4-knetwalk"/>
      <rpm:entry name="kde4-kjumpingcube"/>
      <rpm:entry name="kde4-kspaceduel"/>
      <rpm:entry name="kde4-konquest"/>
      <rpm:entry name="kde4-kshisen"/>
    </rpm:suggests>
</pattern>

Some Tag descriptions:

Tag Description
<rpm:requires>
<rpm:entry name="example" />
</rpm:requires>
Requires RPM example: this package must be installed - otherwise the pattern is not fulfilled.
<rpm:recommends>
<rpm:entry name="example" />
</rpm:recommends>
Recommends RPM example: if available and all dependencies of this package are fulfilled, the package would be installed. If the package is not available, there are not error messages. If the package dependencies are not met, the package would be visible but not installed.
<rpm:suggests>
<rpm:entry name="example" />
</rpm:suggests>
Suggests RPM example: would be shown in the pattern but not installed per default

</div>