Build Service/Build Service Installation Tutorial/openSUSE 11.0

From openSUSE

Installing the Build Service is fairly painless on openSUSE 11.0, if you have internet access:

YaST Setup

Installation Source Setup

The first thing you want to do is tell YaST where to look for the packages.

1. Open YaST
2. Go to the Installation Source module.
3. Add an http source:
   • http://download.opensuse.org/repositories/openSUSE:/Tools/openSUSE_11.0/
     • Server Name: download.opensuse.org
     • Directory on Server: /repositories/openSUSE:/Tools/openSUSE_11.0/
4. If you want to run the front end, add an http source:
   • http://download.opensuse.org/repositories/openSUSE:/Tools:/Devel/openSUSE_11.0/
     • Server Name: download.opensuse.org
     • Directory on Server: /repositories/openSUSE:/Tools:/Devel/openSUSE_11.0/

Installation of Open Build Service

Next, you want to install the build service packages:

zypper install obs-api obs-server obs-worker

Configuration

(The following contents are copied shamelessly from /usr/share/doc/packages/obs-api/README.SETUP in the obs-api package).

By default all service run on the same system, but they can get distributed to increase reliability and do load balancing.

Run the backend

The backend hosts all sources, build packages and calculates the jobs. You need to install the "obs-server" package for this.

Configure the Scheduler

By default, the scheduler is configured to only build for the i586 platform. To change this, you'll need to edit the file /etc/sysconfig/obs-server. Simply add the required platforms to the OBS_SCHEDULER_ARCHITECTURES entry. For example

OBS_SCHEDULER_ARCHITECTURES="i586 x86_64"

If you change the set of build platforms after the scheduler is running, you'll need to restart the scheduler.

rcobsscheduler restart

Valid platforms are i586, x86_64, ppc, ppc64 and local

Start the Services

Start the following services in this order. WARNING: The following starts services which are accessible from the outside.

        Do not do this on a system connected to an untrusted network or be
        sure to block the ports via a firewall.

rcobsrepserver start
rcobssrcserver start
rcobsscheduler start
rcobsdispatcher start
rcobspublisher start

The data get hosted inside of /srv/obs directory by default.

Start the Workers

The workers ask the backend for open build jobs and do the build. You need to install the "obs-worker" package for this.

Local Worker

A worker can be started on the localhost by calling

# rcobsworker start

Multiple Workers on Remote Systems

A worker can be started on other machine in your network (in Internet too, but you should have really fast Internet connection ;-)). You must install package obs-worker on this machine and change few configuration files on both system.

• necessary changes on machine *main* machine

In file /usr/lib/obs/server/BSConfig.pm you should change variables $srcserver, $reposerver and
array @reposervers. After install there are http://localhost but in this case you should have there
output of hostname -f or IPs. (You should keep port number)

• necessary changes on worker machine

Install package obs-worker and in file /etc/sysconfig/obs-worker change variable

# OBS_STORAGE_SERVER to main.fqdn:6262

where main.fqdn is output of hostname -f or IP on *main* machine.

example: (assumed 192.168.0.10 is IP of *main* machine)

# OBS_STORAGE_SERVER="192.168.0.10:6262"

Run the Frontend

You need to install the "obs-api" package for this and a mysql server on this or a remote host.

MySQL Setup & Configuration

Initialize fresh Database

• start the MySQL database

 # rcmysql start    # use "insserv mysql" for permanent start

If you do this for the first time, MySQL will initialize itself and ask to create a root password in the database, like this:

 # /usr/bin/mysqladmin -u root password 'your-password'

• Create the empty production databases:

     # mysql -u root -p
     mysql> create database frontend_production;
     mysql> create database webclient_production;
     mysql> quit

• If you use the MySQL database for other services, too, then it's recommended to add a separate MySQL user, e.g.g 'obs', like

this:

     # mysql -u root -p'your-password'
     CREATE USER obs IDENTIFIED BY 'obs-password';

     GRANT all privileges
     ON frontend_production.*
     TO 'obs'@'%', 'obs'@'localhost';

     GRANT all privileges
     ON webclient_production.*
     TO 'obs'@'%', 'obs'@'localhost';

(note: on 11.2 oth the GRANT lines should have "IDENTIFIED BY 'obs-password';" appended)

• Configure your MySQL user and password in the "production_slave:" section of these files:

     # /srv/www/obs/frontend/config/database.yml
     # /srv/www/obs/webclient/config/database.yml

(on 11.2: note that the socket: line may be wrong and can be removed to cause the socket to be found automatically)

• populate the database

     # cd /srv/www/obs/frontend/
     # RAILS_ENV="production_slave" rake db:migrate

Update Database Schema

Updaters of build server versions before 0.9 needs to rename their databases from "development" to production. This can be performed by the following steps:

 # mysql -u root -p
   mysql> create database frontend_production;
   mysql> create database webclient_production;
   mysql> quit
 # mysqldump -p frontend_development > DUMP
 # mysql -u root -p frontend_production < DUMP
 # mysqldump -p webclient_development > DUMP
 # mysql -u root -p webclient_production < DUMP

All Updaters needs to update their database schema in any case by running the following commands:

 # cd /srv/www/obs/frontend/
 # RAILS_ENV="production_slave" rake db:migrate

Configure the Frontend and Web Client

After the configuration of the frontend (and webclient) rails interface is needed. A configuration for accessing via http://127.0.42.1 and http://127.0.42.2 gets automatically installed into /etc/lighttpd/vhosts.d/obs.conf

If you need other hostnames, edit that file /etc/lighttpd/vhosts.d/obs.conf:

   $HTTP["host"] =~ "buildserver.my.domain.example.org" {
     ...
   }

To enable these vhosts, make sure to uncomment the following is in the 'custom includes' section at the bottom of /etc/lighttpd/lighttpd.conf:

 ##
 ## custom includes like vhosts.
 ##
 #include "conf.d/config.conf"
 # following line uncommented as per
 # /usr/share/doc/packages/obs-api/README.SETUP
 include_shell "cat vhosts.d/*.conf"

Also, the modules "mod_magnet", "mod_rewrite" and fastcgi need to be enabled by uncommenting the corresponding lines in /etc/lighttpd/modules.conf:

   server.modules = (
     "mod_access",
   #  "mod_alias",
   #  "mod_auth",
   #  "mod_evasive",
   #  "mod_redirect",
     "mod_rewrite",
   #  "mod_setenv",
   #  "mod_usertrack",
   )

   ##
   ## mod_magnet
   ##
   include "conf.d/magnet.conf"

   ##
   ## FastCGI (mod_fastcgi)
   ##
   include "conf.d/fastcgi.conf"

Change /srv/www/obs/webclient/config/environments/production_slave.rb

If you changed the hostnames/ips away from 127.0.42.x, you need to configure also /srv/www/obs/webclient/config/environments/production_slave.rb (2 entries).

Restart lighttpd

Afterwards you can start the OBS web frontends via

   chown -R lighttpd:lighttpd /srv/www/obs/{frontend,webclient}/{log,tmp}
   rclighttpd start  # use "insserv lighttpd" for permanent start

If lighttpd does not start you need to enable/disable IPv6

Setup Initial Distributions

The OBS does need some base projects and packages to start the building. You can setup it for example in this way to build for openSUSE 10.3:

• visit http://127.0.42.1 or http://obs-client The default user is "admin" with the password "opensuse".

• Enter name: "openSUSE:11.0" and Title: "openSUSE 11.0 distribution". Click "Create Project"

• A resulting project xml file should have been created at /srv/obs/projects/openSUSE:11.0.xml and looks like this:

<project name="openSUSE:11.0">
  <title>openSUSE 11.0 distribution</title>
  <description></description>
  <person role="maintainer" userid="admin" />
  <person role="bugowner" userid="admin" />
  <person role="maintainer" userid="Admin" />
</project>

• Now you are in the project, click on '[Add Repository]' and on the next page on '[Advanced]' (underneath a bunch of distro buttons) in the name field, enter 'standard'

• The xml file now looks like:

<project name="openSUSE:11.0">
  <title>openSUSE 11.0 distribution</title>
  <description></description>
  <person role="maintainer" userid="admin" />
  <person role="bugowner" userid="admin" />
  <person role="maintainer" userid="Admin" />
  <repository name="standard">
    <arch>x86_64</arch>
    <arch>i586</arch>
  </repository>
</project>

• You can disable builds inside of this project, as it will be used as a base in other projects.

Afterwards you need to copy the rpms from the base distribution to /srv/obs/build/openSUSE:11.0/standard/i586/:full/ Simply create the directory, if it does not exist. In order to be sure that the directory structure has the proper rights set, execute:

chown -R obsrun.obsrun /srv/obs

Once you copied all rpms from the base distribution's i586 and noarch directory inside /srv/obs/build/openSUSE:11.0/standard/i586/:full/ you need to trigger the reading of the directory of the scheduler. The easiest way to achieve this is to restart it.

Last but not least you need to copy the distribution configuration to define how a base distribution needs to get setup. These configurations for SUSE are part of the build rpm.

# cp /usr/lib/obs/server/build/configs/sl11.0.conf /srv/obs/projects/openSUSE:11.0.conf

You may also want to read Build Service/Tips and Tricks#How is a buildenvironment defined? for a description of the configuration format.

The project is now ready to be used as a base for other projects.

Other distros can be added similarly, which makes the 'distro' buttons given when adding a repository work.

Names of projects to create for other distros:

(I'm sure there's others...)

Setup Download On Demand

Note this is broken in the new scheduler (Nov 2009)

This allows the obs service to avoid the preloading of packages if it has access to the internet.

How does it work (currently only debian metadata support is available): As debian stores all relevant information for dependency resolving in one big file (for instance http://ftp.de.debian.org/debian/dists/sid/main/binary-i386/Packages.bz2) the scheduler simply parses this file and builds up all the internal datastructures based on this single file. Now if a worker or an user makes a request for a certain package the repserver checks if the file was already downloaded. If it doesn't exist the requested package will be downloaded and forwarded to the requester. In case the requested file already exists it'll be directly delivered (without downloading it again).

Install instructions: Using this feature is quite simple: you have to

• setup the base distribution (for instance debian sid)
• download and install the Packages file
• fix the permissions

 mkdir -p /srv/obs/build/Debian:SID/standard/i586/:full
 curl http://ftp.de.debian.org/debian/dists/sid/main/binary-i386/Packages.bz2 | bunzip2 > /srv/obs/build/Debian:SID/standard/i586/:full/Packages
 chown -R obsrun:obsrun /srv/obs/build

install the project config etc. (we're reusing the Debian:Etch file)

 osc -A https://api.opensuse.org meta prjconf Debian:Etch > /srv/obs/projects/Debian:SID.conf
 echo "Preinstall: debconf" >> /srv/obs/projects/Debian:SID.conf
 # this is just a workaround...

 osc meta prj Debian:Etch > /srv/obs/projects/Debian:SID.xml
 echo '<download baseurl="http://ftp.de.debian.org/debian"; metafile="Packages" mtype="deb" />' >> /srv/obs/projects/Debian:SID.xml

Now edit the file and replace all occurrences of "Etch" with "SID"

fix the permissions

 chown -R obsrun:obsrun /srv/obs/projects

finally import the new base distribution into the database

 RAILS_ENV="production_slave" ruby /srv/www/obs/frontend/script/import

If the <download /> element is found in a project's metadata file the obs will perform the things described above.

Setup RemoteOBS

To setup a project that pulls from the offical OBS:

<project name="openSUSE.org">
  <title>openSUSE.org Project</title>
  <description>This project refers to projects hosted on the Build Service at the openSUSE.org project.
  This is important especially for the base projects which provides the distributions to build against
  by default. Your local Build Service instance will request, download and cache all needed sources or
  binary packages from the openSUSE.org project when you build against it.
  .
  Use openSUSE.org:openSUSE:10.3 for example to build against the openSUSE:10.3 project as specified on
  the opensuse.org Build Service.
  </description>
  <remoteurl>https://api.opensuse.org/public</remoteurl>
</project>

Access

By default you can access the api via using any browser with the URL:

 http://127.0.42.2

or, if setup:

 http://obs-frontend/

The web interface is accessable via:

 http://127.0.42.1

or, if setup:

 http://obs-client/

The default user is "Admin" with the password "opensuse".

Updating

When the obs-api package has been updated, it might be required to upgrade the database. Therefore, you should execute

cd /srv/www/obs/frontend/
rake db:migrate

after updating obs-api.

Command-Line Interface (osc)

In order to use the command line tool for the build service, change your ~/.oscrc to something like this:

[general]

# Downloaded packages are cached here. Must be writable by you.
#packagecachedir = /var/tmp/osbuild-packagecache

# Wrapper to call build as root (sudo, su -, ...)
#su-wrapper = su -c

# rootdir to setup the chroot environment
# can contain %(repo)s and/or %(arch)s for replacement
#build-root = /var/tmp/build-root

# use this API server
# (it needs a section [api.opensuse.org] with the credentials)
#apisrv = api.opensuse.org

# our local build service api:
apisrv = 127.0.42.2

# for local build service, use http (not https)
scheme = http

# local default admin account
[127.0.42.2]
user = Admin
pass = opensuse

# normal build service account
[api.opensuse.org]
user = foo
pass = bar