openSUSE:Release team processes

Tools/Repositories

• Git repository on github containing all verification tasks (repo-checker) and script that sents the reminder mails
• Repository for automatic submitter at gitorious

Taks for Factory maintenance

We work in various areas in order to keep Factory going and the ISO images published and installable.

Checking in submit requests

We are overseeing the queue and approving new updates for the Factory repository.

There is automation currently set up to do some tasks:

• checking in automatically changes from dev project to factory (obs-autosubmit) FIXME: should use proper locking instead of file lock
• submitting new cpan packages
• validating the build/install integrity of the submission (repo-checker)
• factoryauto script

These tasks are currently (2013-10-16) handled on opensuseqa.suse.de:2222 in a screen session for root:

• If this session is not running it needs to be restarted.
• The repo-checker part should be overwatched to close failing SRs which are still in stage "review".
• All cron jobs must be working (under all users) ; especially */10 * * * * /abuild/home_package-lists/update-lists.sh script FIXME: scrpits should use proper locking instead of file lock

If the repo-checker is failing for some packages and you don't want to force accept it you can force-skip it on the machine directly by stopping the repo-checker look and calling following command osc check_repo --skip <SR#> and resuming back the queue afterwards.

If the requests are related (eg. should be verified together for the install state) the osc group feature should be used to ensure the repo-checker can do its part:

• osc group is osc extension script provided in factory-auto repository which is quite straight forward to use. Currently only problem is that you can't approve the request as whole but by all the submissions.

After all the processes are finished the SR state gets changed to new and we should approve it in to the repository when the state monitor of OBS shows there is no big overload. Huge-rebuilds should be planned for Friday to build over the weekend.

If the package is not auto-reviewed correctly (eg fails to pass while it should) the SR can be accepted regardless, but it should be avoided and the tools should be fixed.

When submission is accepted to Factory the package gets rebuild and then all the reverse dependencies are rebuilt in Factory:Rebuild project. When this project finishes its task the repository gets republished and is ready for end-user consumption.

Ensuring Factory health

Factory health verification can be specified in two areas. Ensuring everything builds and all packages are resolvable and vadiating if the installed system is actually usable.

Package verification is done in two areas:

• checking the build state in OBS and fixing the most profilic issues and proding maintianers to fix
• checking the overall health with all reports

Apart from these two we also sent reports over mail to maintainers about their packages, which is personalised for their needs listing only those they are involved in. This script is stored at github with others and sents the announcement with coolo' mail identity.

Runtime verification is also two stages process:

• openQA testing to see at least basics are working
• live testing on our users (and ourselves as the development of Factory is done from Factory)

Releasing and maitnaining ISO releases

Generating ISO for release/snapshot is quite tricky process.

1. All builds must be completed and repoitory state is shipped on normal tree (as in all slots are marked with "truck" icon). This also builds the images packages for the install process as they are built after all other packages built.
2. Live tree must be enabled and build+shipped on all packages. The Live packages are updated by cronjob at the packagelists machine which updates gitorious repo which in return update the Live packages with fresh lists. Next step is again possible only when again all the packages are published again.
3. All products must be succeeded in order to provide all required ISO images.
4. Autobuild team needs to sign the ISO by hand and distribute it to mirrors.

Current problems with the task:

• Updating package results in multiple generated ISOs as the Live gets updated so all products get rebuild which generates new images which needs new Live update. My check seems that one package update renders 4 new ISO builds (eg. Build001 to Build004).
• Instead of having simple way how to put packages to ISOs we go over the autobot in cron that sorts this out (and hand editing to fit on DVDs/CDs whatever we agree on).
• Signing and mirror distribution should be automated to not abuse the autobuild team.