openSUSE:Osc Collab
OSC Collaboration edit
With the introduction of the collaboration features of the build service, everybody can now help with packaging in openSUSE. For GNOME, for instance, the development is done in the GNOME:Factory (G:F) project, and packages submitted there will then be submitted to openSUSE:Factory (oS:F) which is where the Factory distribution lives.
However, the command line for all this might be a bit hard for first-time users, and some of the usual workflow can actually be automated. That's why we created the osc collab plugin. This plugin provides three extremely useful commands: osc collab todo, osc collab update and osc collab setup. A few other commands are available.
A typical workflow example
First, let's look at what can be done (the output of the todo command is truncated in this example) for the GNOME:Factory project:
vuntz@lyon ~/work/opensuse/tmp/>osc collab todo --project GNOME:Factory Downloading data in a cache. It might take a few seconds... Package | openSUSE:Factory | GNOME:Factory | Upstream --------------------------------+------------------+------------------+----------------- anjuta | 2.23.91 | 2.23.91 | 2.24.0.1 (r) at-spi | 1.23.92 | 1.23.92 | 1.24.0 gdm | 2.23.92 | 2.23.92 (s) | 2.24.0 (s) gedit | 2.23.93 | 2.23.93 | 2.24.0
We can see that anjuta, at-spi, gdm and gedit all need an update in GNOME:Factory since there is a more recent version upstream. We notice that anjuta has already been reserved by somebody, so we can ignore it. Similarly, we notice that there is a submission for gdm waiting to be approved by a GNOME:Factory maintainer. So let's just ignore this one too. It's possible to automatically remove those lines with some arguments: osc collab todo --exclude-reserved --exclude-submitted (or osc collab t --xr --xs for people who prefer to not type long commands).
So let's work on updating gedit:
vuntz@lyon ~/work/opensuse/tmp/> osc collab update --project GNOME:Factory gedit
Package gedit has been reserved for 36 hours.
Do not forget to unreserve the package when done with it:
osc collab unreserve gedit
A /home/vuntz/work/opensuse/tmp/gedit
A /home/vuntz/work/opensuse/tmp/gedit/gedit-2.23.93.tar.bz2
A /home/vuntz/work/opensuse/tmp/gedit/gedit-desktop.patch
A /home/vuntz/work/opensuse/tmp/gedit/gedit.changes
A /home/vuntz/work/opensuse/tmp/gedit/gedit.spec
A /home/vuntz/work/opensuse/tmp/gedit/ready
Package gedit has been checked out.
gedit.spec has been prepared.
gedit.changes has been prepared.
Looking for the upstream tarball...
gedit-2.24.0.tar.bz2 has been downloaded.
Finding NEWS and ChangeLog information...
NEWS between gedit-2.23.93.tar.bz2 and gedit-2.24.0.tar.bz2 is available in osc-gnome.NEWS
ChangeLog between gedit-2.23.93.tar.bz2 and gedit-2.24.0.tar.bz2 is available in osc-gnome.ChangeLog
Running quilt...
Patches still apply.
gedit-2.23.93.tar.bz2 has been removed from the package.
gedit-2.24.0.tar.bz2 has been added to the package.
Package gedit has been prepared for the update.
The output of this command is a bit verbose, but this way, you can be sure of what's going on. In this specific case, everything went fine and so it seems you only need to update gedit.changes.
So, you first change the current directory to the checked out package with cd gedit. And then, you can now open gedit.changes in your favorite editor. You'll notice that it's already prefilled:
------------------------------------------------------------------- Tue Sep 30 16:37:35 CEST 2008 - vuntz@opensuse.org - Update to version 2.24.0: + -------------------------------------------------------------------
So you only have to mention what has changed. The osc-collab.NEWS and osc-collab.ChangeLog files should be of some help here.
Now that you've updated gedit.changes, you should make sure that all patches in the package are still relevant and that the build and runtime dependencies are up-to-date. There is no magic way to do this, so it's all manual work.
The next steps are typically to commit your changes, make sure the package builds correctly and if this is the case submit it. You can do all this in one command: osc collab buildsubmit -m 'Update to 2.24.0'.
After the successful osc collab buildsubmit, the package will automatically be unreserved by default, unless you used the --no-unreserve option. You can alternatively manually unreserve the package: osc collab unreserve --project GNOME:Factory gedit
Alternative to buildsubmit
If you don't want to use osc collab buildsubmit, you can use the following instructions which are the typical commands used for collaboration in the build service. However, osc collab buildsubmit is enough in most of the cases.
- First, you can commit your changes to the build service: osc commit -m 'Update to 2.24.0'
- Once this is done, make sure that the package builds correctly -- either by checking the result of the build in the build service (on the web page or via osc results home:vuntz:branches:GNOME:Factory gedit) or by using osc build.
- If the build is successful, submit your package to GNOME:Factory with the following command: osc submitreq create -m 'Update to 2.24.0'
What can go wrong
- if you try to update a package which is already reserved by somebody, you will get an error. You can use the --ignore-reserved option to ignore this. You should only do this after communicating with the person who reserved the package.
- sometimes, some patches don't apply anymore. In this case, you will see that the 'quilt' step failed. You will have to fix this first.
}}
User documentation
This section provides the basic steps to get your running with osc collab from installation to working with several projects.
Installation
The easy way to install this brilliant plugin is through the osc-plugin-collab package from the openSUSE:Tools project.
You can also install the latest version of the plugin from git:
- Install osc and git: zypper in osc git-core
- Checkout the osc-plugins repository: git clone https://github.com/openSUSE/osc-plugin-collab.git
- Copy the plugin: mkdir -p ~/.osc-plugins/ && cp osc-plugin-collab/osc-collab.py ~/.osc-plugins/
To make sure the plugin is installed, simply run osc collab --help and you should see the help summary of the plugin.
The first time you will run the plugin to do an actual command, it will ask you for an email address. This email address is saved in ~/.oscrc and will be used when updating the .changes files.
Configuration
The plugin has some default configuration that you can overwrite in ~/.oscrc (in the [general] section) or via some command-line options. Note that the command-line options overload the configuration from ~/.oscrc.
| Variable | Default value | Command line option | ~/.oscrc key to change it | Example |
|---|---|---|---|---|
| Projects | GNOME:Factory; | --project | collab_projects | collab_projects = GNOME:Factory;X11:common:Factory;GNOME:Contrib; |
| Repository | openSUSE_Factory | --repo | collab_repo | collab_repo = openSUSE_11.1 |
| Architectures | i586;x86_64; | --arch | collab_archs | collab_archs = i586; |
| Package tracking | 0 | collab_do_package_tracking | collab_do_package_tracking = 1 |
As an example, people working on Moblin on openSUSE 11.1 should probably use this configuration in ~/.oscrc:
collab_projects = Moblin:UI;Moblin:Base; collab_repo = openSUSE_11.1
while people interested in GNOME 2.26 on openSUSE 11.1 should use:
collab_projects = GNOME:STABLE:2.26;GNOME:Backports:2.26; collab_repo = openSUSE_11.1
A note about package tracking
If you do not know about package tracking, you can skip this :-)
Due to the way osc collab works, the package tracking feature is disabled when the setup and update commands are used: osc collab checks out the package in a flat directory, with no project structure which makes package tracking impossible to use. If, as a user, you want to enforce package tracking to be used, you will need to use osc collab appropriately (by calling it in the right directory for each project) and to set the collab_do_package_tracking option.
What can I do?
One issue in a big team is that often, people don't know what they can do: there are so many things to do that you get lost since you don't know what should really be done and what people are working on. The osc collab todo helps you here. It will display all the packages that need an update to a new upstream version. It's generally the easiest way to start working on the packages.
You can use the --exclude-reserved and --exclude-submitted options to hide packages which are already being worked on by someone else, or to hide packages which are already submitted. Those packages are identified with (r) and (s) in the normal output.
Reservation system
To avoid people duplicating work, we are using a really simple reservation system. When you run osc collab update or osc collab setup, you automatically reserve the package for work. To ignore this reservation step, you can use the --no-reserve option.
A reservation lasts 36 hours. After this time, the reservation expires and is automatically removed. A package is also automatically unreserved after a successuful osc collab buildsubmit, unless you used the --no-unreserve option. To manually remove a reservation, the osc collab unreserve command can be used..
Note that you can manually reserve a package for work with osc collab reserve. This command is provided for completeness only, since you would generally use osc collab update or osc collab setup.
It is possible to see who is working on what with the osc collab listreserved, and you can also check if a package is reserved with the osc collab isreserved command.
Updating a package to a new upstream version
The most common task during the early development cycle of a new version of openSUSE is to update packages to new upstream versions. The osc collab update command can be used for that.
This command will do the following:
- check if there is an update for the package. If this is not the case, the command stops.
- check that nobody is working on the package, unless the --ignore-reserved option is used.
- reserve the package for you, unless the --no-reserve option is used.
- branch the package from the devel project and check it out in the current directory.
- do the required updates that can be determined to the .spec file (usually, update the Version tag).
- prepend a new .changes entry that you will later fill.
- download the upstream tarball.
- extract the relevant NEWS and ChangeLog information for this new upstream tarball.
- check that patches in this package still apply with the new upstream tarball.
- remove the old tarball from the build service and add the new tarball to the build service.
In general, the only thing left to do after running this command is to fill the .changes file. You also have to make sure that all patches are still relevant and that all the build and runtime dependencies are correct.
Working on a package
The other common task when working on a package is to modify a package, to add a new patch or to fix a packaging issue. The osc collab setup command helps you prepare the package.
This command will do the following:
- check that nobody is working on the package, unless the --ignore-reserved option is used.
- reserve the package for you, unless the --no-reserve option is used.
- branch the package from the devel project and check it out in the current directory.
You can then easily do the work you wanted to do.
Building a package
Once a change is ready, it's important to make sure that the updated package still compiles. You can compile the package locally with osc build, or you can commit and wait for the build to succeed on the build service. The osc collab build command is a convenience command that will commit the changes on the build service, and will make sure that a build starts there. It will then monitor the build so that you can easily check the status of the build.
The --repo and --arch arguments can be used to specify the target of the build.
Building and submitting a package
In the usual workflow, you want to submit your change to the parent project if the package builds fine. The osc collab buildsubmit lets you do that: it works exactly like the osc collab build command, but in addition, if the build is successful, it will submit the build to the parent project.
Some people refer to this command as the "fire and forget" command, since you can just do your changes, and then run this command; if everything is fine, you won't have to worry about this package since it will get automatically submitted. You can even use the --forward option to automatically forward to the destination project (see below for a description of the forward feature).
Working with more than one project
In a simple use case, you only need to work on packages living in a single project. However, many users of osc collab will need to interact with packages living in various projects. This usually is completely intuitive since there's no intersection between the packages of those projects (eg, between GNOME:Factory and X11:common:Factory).
Sometimes, though, you need to work with projects that have packages with the same names. A good example of this is GNOME:Factory and GNOME:STABLE:2.26. In such cases, if you use --project GNOME:Factory --project GNOME:STABLE:2.26, all the actions on a package name that exists in both projects will take effect on the first project (here, GNOME:Factory). This should still stay intuitive.
Note that it's always possible to force on which project you want to work by using a single --project option.
Other useful commands
There are a few other commands in osc collab that might be of interest to you.
Tasks for administrators
osc collab todoadmin lets you do a sanity check of some project. It will display a list of things you might want to do to clean the project. For example, with GNOME:Factory:
- packages with a patch that don't apply anymore
- packages which are not links to openSUSE:Factory
- packages which shouldn't exist in GNOME:Factory (either because they're dead or because they're maintained elsewhere)
- packages which should exist in GNOME:Factory (generally, this is new packages that haven't been linked in GNOME:Factory yet)
- packages which should be submitted to openSUSE:Factory
- etc.
You can use the --exclude-submitted option to hide packages which are submitted to openSUSE:Factory but waiting for a review there.
Generally, the way to fix the issues are trivial to know for an administrator. For reference, the following commands are generally helpful:
- osc linkpac openSUSE:Factory $pkg GNOME:Factory: to link a package from openSUSE:Factory to GNOME:Factory
- osc checkout -u GNOME:Factory $pkg: to check out a package in unexpanded mode, to fix patches that don't apply anymore, eg.
Forwarding a package
osc collab forward lets you forward a submitted change directly from a user branch to the destination project, bypassing the step where the change stays in the devel project for a while. Internally, the submission gets accepted to the devel project, and then a new submission is created from the devel project to the destination project.
Here's an example: assuming that submission #12345 is a submission of gnome-panel from home:vuntz:branches:GNOME:Factory to GNOME:Factory, osc collab forward 12345 will accept this submission in GNOME:Factory, and will create a new submission from GNOME:Factory to openSUSE:Factory for the gnome-panel package.
Contributing
Everyone is invited to contribute to the development of the osc-plugin-collab plugin. Feel free to report bugs and send patches to the development mailing list:
- gnome@lists.opensuse.org - openSUSE GNOME Mailing List.
Subscribe - Unsubscribe - Help - Archives
The plugin is developed in a git repository on github. You can also clone your own git fork on github or elsewhere on the Internet to ease collaboration in the development process.
Backend details
Right now, the osc collab works thanks to a database that is created by various scripts. These scripts live in osc-plugin-collab/server/. Feel free to look at them and improve them too.
For Packagers: you can get hints about how to submit your packages to the tracker here: SDB:Upstream Tracker#Submit your package to the tracker. (This applies to all packages existing on OBS)