openSUSE:Build Service Concept SourceService

Jump to: navigation, search
Open Build Service
End user - Packager - Development - Installations - Collaboration - Comparison


This page describes how source service could be implemented and used in the OBS. Source services are applications which process source files before the actual scheduling or building happens.

Examples for this are:

  • Download service - downloads a file (e.g. tarball) from given URL and stores it.
  • Checkout service - checks out from SVC system (svn, git, ...) and creates a tarball.
  • Validate service - validates a file with a given checksum (md5sum, gpg key).
  • Code generator - analyses a tarball and creates proper rpm and debian build descriptions (based on cmake, automake/conf, qmake or alike files).
  • Run pre_checkin.sh scripts via a service


Tool aspects

  • All of the given examples will need tools which are not 100% trustworthy, that means that they should run in a secure environment.
  • The tools should allow to run on server and client side.
  • The tools may be highly specific, e.g. a code generator which is based on cmake files. This code should not be part of the Build Service itself, but it should be inside a package which can be easily maintained. This allows contributors to fix, extend or add services via usual packaging methods.
  • The tools should be maintained in a project, building for all distributions to allow the executions on the developer workstations. However only one instance is needed on the server side.
  • As much as possible should be done within the packages to have a thin as possible layer for this within OBS code.
  • Tools can share functionality via common packages.


Workflow aspects

A meta file (_service file) as part of the package sources should trigger the services. It might run multiple of them in a given order. A change in this meta file should not direct trigger a build, but wait for the generator result.

The result of the services should become part of the source package and the history. However it should not possible to change these files directly to avoid lost work, when the service is generating a new file.

Server side service instance

The server should have a permanently running service instance. This is basically a usual build instance, but running all the time, except when new service tools get deployed (or build). This instance needs internet connection for its services, but we need to protect against random connections to our internal interfaces.

The same instance should keep running and not restarted/rebuild for each call, but it needs to get updated when new service packages appear.

Needed implementations

Tools

  • A set of tools which create files.
  • Store all of these tools in a project

Source server

  • On source commit, containing a _service file the server needs to trigger an event for updating the sources in package.

Dispatcher

  • Validate that a service instance is running, if not start one according to specification.
  • Send service event to service instance, if it is free (basically the _service file together with project and package information).

Scheduler

  • Needs to block a package, if a service event is waiting
  • Do not remove calculated buildinfo for service instance to keep it running. However it needs to get updated if service tool packages are changing and running instance need to get stopped as usual.

Build script and bs_worker

The build script needs a service mode which needs to

  1. build up a system according to a configured service repo and package list (TBD where to specify this, maybe just BSConfig.pm for now.)
  2. endless loop to process _service files, for each file
    1. run XEN/kvm with network enabled
    2. rename all _service:*:$name files to $name
    3. process all <service items by calling the specified tool in /opt/obs/lib/service/$name
    4. XEN/kvm session ends
    5. copy generate files with prefix to source server (_service:$tool:$filename)
    6. check if service instance got a kill event and die, otherwise loop back

Possible extensions

This framework might get extended later, to allow to create/remove complete new packages. This would allow us to use it for the productconverter and also for stuff like cpan2OBS generators.

This would also obsolete the oscupstream need.

Examples

Example 1: Input file commit

This is a real life example, stored as _service in the package sources. 

<services>
 <service name="download_url">
   <param name="protocol">http</param>
   <param name="host">download.kde.org</param>
   <param name="path">pub/kde/stuff/krabber-1.0.tar.gz</param>
 </service>
 <service name="verify_file">
   <param name="file">_service:download_url:krabber-1.0.tar.gz</param>
   <param name="verifier">sha256</param>
   <param name="checksum">7f535a96a834b31ba2201a90c4d365990785dead92be02d4cf846713be938b78</param>
 </service>
 <!--
  <service name="generate_automake_kde" />
 -->
</services>

This would download krabber-1.0.tar.gz and stores it as _service:download_url:krabber-1.0.tar.gz, validates it via a given sha256 checksum, and (commented out) generates a spec file using the generate_automake_kde tool.

This _service file is e.g. created by 

   osc add http://download.kde.org/pub/kde/stuff/krabber-1.0.tar.gz

The required packages (recent osc, obs-service-download_url, build) are available from openSUSE:Tools.


Example 2: GIT integration

Using OBS source services, you can create a package of the latest source code from a Git repository. We suppose that you know how to create/checkout/commit a package to an OBS project.

Requirements

You will need 4 additional packages on your OBS server:

  • obs-service-tar_scm: create an archive from a source code repository
  • obs-service-extract_file: extract files from an archive
  • obs-service-recompress: compress/recompress an archive
  • obs-service-set_version: update package version in spec/dsc files

You can check with service are available with the osc client: 

   osc api /service

Step by step

  • Create an empty package, either using the web interface or the commandline
  • In that package, create a file named _service with root tag <services> </services>
  • Insert a first service definition to create an archive from git: (FIXME tar_scm is discouraged in favor of obs_scm)
  <service name="tar_scm">
    <param name="scm">git</param>
    <param name="url">the URL of your Git repository (something like "git://gitorious.org/your-project.git")</param>
    <param name="subdir">the subdirectory of your repository you want to create an archive from</param>
    <param name="filename">the name of the file you want to create, without version</param>
    <param name="versionprefix">the first part of the version string (e.g. "0.4.git")</param>
    <param name="revision">optional: the commit hash, tag or branch you want to get</param>
  </service>

This will create a file called <filename>-<versionprefix>.<commit_timestamp>.tar, with <commit_timestamp> being the number of seconds between Jan 1st 1970 and the commit from which the archive is created.

  • Insert a second service definition to extract .spec/.dsc files from the archive. If you prefer manually adding these files in your OBS package, just skip this section.
  <service name="extract_file">
    <param name="archive">*.tar</param>
    <param name="files">the files you want to extract from the previously created archive, separated by space
    (notice that there is a top directory with a name that you don't know, so file names should start with "*/"</param>
  </service>
  • Insert a third service definition to compress the archive (Debian builds expect .tar.gz and it saves space on server)
  <service name="recompress">
    <param name="file">a pattern matching the archive name (e.g. "*git*.tar")</param>
    <param name="compression">the format in which you want to compress the archive: "gz", "bz2", "xz", "none"</param>
  </service>
  • Insert a fourth service definition to update package version in spec and dsc files, with the one of the generated archive
  <service name="set_version"/>
  • Now upload/commit the _service file on OBS
  • Each time a new commit is available in your Git repository, trigger a service run, either with the web UI or with osc service run

Complete _service example

FIXME tar_scm is discouraged in favor of obs_scm 

<services>
  <service name="tar_scm">
    <param name="scm">git</param>
    <param name="subdir">src</param>
    <param name="url">git://gitorious.org/meego-developer-tools/obs-light.git</param>
    <param name="versionprefix">0.4.git</param>
    <param name="filename">obslight</param>
  </service>
  <service name="extract_file">
    <param name="archive">*.tar</param>
    <param name="files">*/deb/* */rpm/obslight.changes */rpm/obslight.spec */rpm/obslight.yaml</param>
  </service>
  <service name="recompress">
    <param name="file">*git*.tar</param>
    <param name="compression">gz</param>
  </service>
  <service name="set_version"/>
</services>


All OBS services available

Note: You can check the parameters for the services with the osc client: 

   osc api /service

Or you can view the output form https://paste.opensuse.org/45284231

cpanspec


cargo


go_modules


download_files


download_src_package


download_url

extract_file


format_spec_file


generator_driver_update_disk


generator_pom


git_tarballs


github_tarballs


kiwi_import


node_modules


part2pkg


python_requires


product_converter


python_sdist


rearchive


recompress

  • none : No Compression
  • gz  : Gzip Compression
  • bz2  : Bzip2 Compression
  • xz  : XZ Compression
  • zstd : Zstd Compression

refresh_patches


regex_replace


renderspec


set_version


set_version.sle_11


source_validator


tar_scm


verify_file

Retrieved from "https://en.opensuse.org/index.php?title=openSUSE:Build_Service_Concept_SourceService&oldid=193668"