openSUSE:Build Service Concept SourceService

tagline: From openSUSE

Examples for this are:

• Download service, downloads a file (eg tar ball) from given URL and stores it.
• Checkout service, checks out from SVC system (svn, git, ...) and creates a tar ball.
• Validate service, validates a file with a given checksum (md5sum, gpg key)
• Code generator, analyses a tar ball and creates proper rpm and debian build descriptions (based on cmake, automake/conf, qmake or alike files).
• Run pre_checkin.sh scripts via a service

---

Contents 1 Tool aspects 2 Workflow aspects 3 Server side service instance 4 Examples 4.1 Example input file commit 4.2 Example GIT integration 5 Needed implementations 5.1 Tools 5.2 Source server 5.3 Dispatcher 5.4 Scheduler 5.5 Build script and bs_worker 6 Possible extensions

Tool aspects

• All of the given examples will need tools which are not 100% trustworthy, that means that they should run in a secure environment.
• The tools should allow to run on server and client side.
• The tools maybe highly specific, eg a code generator which is based on cmake files. This code should not be part of the Build Service itself, but inside a package which can be maintained inside of it. This allows contributors to fix, extend or add services via usual packaging methods.
• The tools should be maintained in a project, building for all distributions to allow the executions on the developer workstations. However only one instance is needed on the server side.
• As much as possible should be done within the packages to have a thin as possible layer for this within OBS code.
• Tools can share functionality via common packages.

---

Workflow aspects

A meta file as part of the package sources should trigger the services. It might run multiple of them in a given order. A change in this meta file should not direct trigger a build, but wait for the generator result.

The result of the services should become part of the source package and the history. However it should not possible to change these files directly to avoid lost work, when the service is generating a new file.

---

Server side service instance

The server should have a permanently running service instance. This is basically a usual build instance, but running all the time, except when new service tools get deployed (or build). This instance needs internet connection for its services, but we need to protect against random connections to our internal interfaces.

The same instance should keep running and not restarted/rebuild for each call, but it needs to get updated when new service packages appear.

---

Examples

Example input file commit

This is a real life example, stored as _service in the package sources.

<services>
 <service name="download_url">
   <param name="protocol">http</param>
   <param name="host">download.kde.org</param>
   <param name="path">pub/kde/stuff/krabber-1.0.tar.gz</param>
 </service>
 <service name="verify_file">
   <param name="file">_service:download_url:krabber-1.0.tar.gz</param>
   <param name="verifier">sha256</param>
   <param name="checksum">7f535a96a834b31ba2201a90c4d365990785dead92be02d4cf846713be938b78</param>
 </service>
 <!--
  <service name="generate_automake_kde" />
 -->
</services>

This would download krabber-1.0.tar.gz and stores it as _service:download_url:krabber-1.0.tar.gz, validates it via a given sha256 checksum, and (commented out) generates a spec file using the generate_automake_kde tool.

This _service file is e.g. created by

   osc add http://download.kde.org/pub/kde/stuff/krabber-1.0.tar.gz

The required packages (recent osc, obs-service-download_url, build) are available from openSUSE:Tools .

---

Example GIT integration

You can find a practical step by step GIT integration described here.

Needed implementations

Tools

• A set of tools which create files.
• store all of these tools in a project

Source server

• On source commit, containing a _service file the server needs to trigger an event for updating the sources in package.

Dispatcher

• validate that a service instance is running, if not start one according to specification.
• send service event to service instance, if it is free (basically the _service file together with project and package information).

Scheduler

• needs to block a package, if a service event is waiting
• do not remove calculated buildinfo for service instance to keep it running. However it needs to get updated if service tool packages are changing and running instance need to get stopped as usual.

Build script and bs_worker

The build script needs a service mode which needs to

1. build up a system according to a configured service repo and package list (TBD where to specify this, maybe just BSConfig.pm for now.)
2. endless loop to process _service files, for each file
   1. run XEN/kvm with network enabled
   2. rename all _service:*:$name files to $name
   3. process all <service items by calling the specified tool in /opt/obs/lib/service/$name
   4. XEN/kvm session ends
   5. copy generate files with prefix to source server (_service:$tool:$filename)
   6. check if service instance got a kill event and die, otherwise loop back

---

Possible extensions

This framework might get extended later, to allow to create/remove complete new packages. This would allow us to use it for the productconverter and also for stuff like cpan2OBS generators.

This would also obsolete the oscupstream need.