openSUSE:Development Process Details
openSUSE Development and Integration Process (Details)
From Roadmap to Beta1
Version 1.0.3 - 29/08/2013
Author: openSUSE Team at SUSE
- 1 openSUSE Development and Integration Process (Details)
- 1.1 1. Roadmap and Feature Planning
- 1.2 2. Package Development
- 1.3 3. Devel Project Integration
- 1.4 4. Review Process
- 1.5 5. Factory Integration
- 1.6 6. Quality Assurance Process
- 1.7 7. Release Process
- 1.8 8. Public Test
- 2 Project Organization
- 3 Glossary
- 4 References
1. Roadmap and Feature Planning
- To create the initial time-line for the release: the roadmap
- To identify the main technical goals for the new release
- E1.1 Previous release version of the distribution was released
- T1.2 Feature freeze calendar. Together with the roadmap calendar is the associated feature freeze calendar . There are three stages of freezing: toolchain freeze after M3, base system freeze after M4 and full feature freeze after B1. After that the maintenance process starts. Freezes are scheduled at the end of milestone development periods to give time for development after a milestone release. They should not be placed too close to the next release as the frozen components should get a little testing before the milestone is out.
- T1.3 Determine the new features. On the mailing list (as the primary channel for discussing ideas and features) and through other community communication channels the new features expected for this release are discussed and defined. Some of those features can be documented on openFATE  (not widely used) or the wiki . The features can be decided inside the community (in this moment this is done only by a really small number of contributors) or internally in SUSE when there is a relation between SLE and openSUSE: a developer can create a feature in FATE for SLE and then add openSUSE afterward (this kind of feature is not visible outside) At this moment we do not have a strict policy on using openFATE. Some features are also decided later without planning and documentation. If the feature is implemented after the freeze calendar, an explicit approval from the Factory Maintainer is needed for the integration.
- V1.1 Review the roadmap. The roadmap needs to be approved by the Product Manager and the Release Manager.
- V1.2 Roadmap maintenance. Small changes in the roadmap are not communicated, just executed. But if a delay threatens the release day, an announcement is expected, and if the delay is serious we need to communicate it widely  . In every case T5 is involved, update the wiki.
- X1.1 The roadmap and the freeze calendar are published in the mailing list
- D1.1 XML Document where the roadmap is described 
- D1.2 The features documented in openFATE  and in the wiki 
2. Package Development
- Create, update or fix packages in the devel projects or in Factory
- Be the starting point of the development process
- Develop and experiment without interfering with the devel projects and Factory packages
- E2.1 The developer is creating a new package that is not in Factory or in devel projects, but has the intention to integrate the package in the distribution
- E2.2 The developer wants to update a package that is currently in the distribution
- E2.3 The developer wants to fix a bug in a package that is currently in the distribution, in response to a bug report in Bugzilla or another channel 
- E2.4 There is a message from the messaging system (Hermes)   describing an action that needs to be done in a previous submit request
- T2.1 Create local package. If the package is new (does not yet exist in a devel project or in Factory), the developer creates a new package using the osc command line tool or the web interface. The creation of a package is a complex process well documented  and with policies which the developer must follow . If the new package is stored in CPAN, Pypi or is a Ruby gem, there are scripts that help during the package creation, but in either case it is the packager who needs to submit the package.
- T2.2 Branch Package. If the package currently exist in one of the devel projects and /or in Factory, the package developer branches the package from the original project to the local home. This process is similar to copying the files, but a relation between the new copy and the original package is retained (stored in some metadata files). The developer still needs to follow the general policies referenced in T2.1
- T2.3 Fix / Update Package. The real work of the developer is expressed with this task. The package developer now can fix the bug, create a new feature or update the package. After the work is done, the developer submits the changes to OBS to launche the process described in "V2.2 Automatic checks complete successfully".
- T2.4 Submit to Devel Project. Once the package is created or fixed/updated, and V2.2 is completed successfully, the package developer creates a submit request for the Devel Project (see Glossary to understand the relation with Devel Project and Factory) to get the package included there. If he / she doesn't know which Devel Project is related with the packages (maybe because is a new packages), the Package Developer need to ask in opensuse-factory mailing list. The Factory Maintainer can create a new Devel Project in case that there is not a good match for the package, or more frequently, point to the already existing Devel Project.
- T2.5 Automatic Submit. There are external scripts which update certain packages automatically. Those scripts are related to some very specific Devel Projects and are not generally used. When run they act like a Package Developer, updating the package and submitting directly to OBS (V2.2)
- V2.1 Is in Factory. The work flow of creating a new package for future integration into Factory is different from the work flows for fixing bugs or updating packages that are currently in Factory. With this verification point the developer makes an explicit decision to change the path of the work flow.
- V2.2 Automatic checks complete successfully. Every package needs to be committed in OBS which will build the package and run some standard tests. If compilation is successful and the package passes the tests, the developer can create a Submit Request.
- X2.1 The developer creates a submit request in OBS. This submit request will be received by the Package Maintainer or the Project Maintainer of the package.
- X2.2 The developer drops the change, aborting the development process.
- D2.1 Submit Request. This document is a digital document created by the Package Developer when the package is successfully created in OBS and deemed ready by the Package Developer. This document is used by the Package Maintainer and the Project Maintainer to integrate the changes of the new version of the package in the devel project.
3. Devel Project Integration
Before a package can be integrated into Factory, it needs to belong to a devel project. A devel project is a set of related packages that share certain features. For example, in devel:language:python are stored all the packages related to the programming language Python. LibreOffice:Stable is the devel project where are stored the packages of LibreOffice updated to the last stable version.
- Provide a space where the main development (integrating feature releases) takes place.
- To guarantee a certain level of quality and integration before the package is finally integrated into Factory. This integration process follows the cycle checkout – fix – commit – submit request, very similar to the process described in 2. Package Development.
- E3.1 A submit request from a Package Developer to a devel project.
- E3.2 A decision by the Project Maintainer to make changes to a package directly in the Devel Project
- T3.1 Integrate the submit request. The Project Maintainer can integrate the code coming from the submit request after the review is done in V1. Usually this integration is done using the OBS web interface or the osc command line tool. The Package Maintainer can only accept and integrate the submit request for his / her own packages.
- T3.2 Package checkout. The Project Maintainer has two options for working with the packages part of the devel project: one is use the exit criteria and branch the package in his/her own home project (changing roles into Package Developer). The other option is checkout the package directly from the devel project and submit a change there. The former option is more secure and clear, as it avoids collisions with new submit requests. The latter one is a faster approach more adequate for projects where the number of maintainer and developers is low. To help in this process, some devel projects use extensions to the OSC command line tool. For example, the GNOME project uses the plug-in  "collab"  to allow advanced workflows.
- T3.3 Fix / Update Package. The real work of the developer is expressed with this task. The Project Maintainer or the Package Maintainer can work, using as input the bugs from bugzilla , or the features planned for their own project.
- T3.4 Submit to Factory. Once the Package or Project Maintainer considers the package(s) in his project good enough he/she creates a submit request for Factory.
- V3.1 Review of the submit request. Before the integration, the Project Maintainer (or the Package Maintainer if the request is related with his / her own package) checks the request manually, to detect problems in the spec file, in the patches attached or in the compilation process. If no problems are found, the request is accepted and integration takes place (T3.1), otherwise the request is declined and Hermes sends a message to the creator of the submit request. Some devel projects have an internal review process, but there is not a shared policy by default.
- V3.2 Compile in OBS. Every package is committed to OBS. OBS will build the binaries according to the spec file . If the compilation is successful, the Project Maintainer can create a Submit Request and send it to Factory or continue the integration process with the rest of the packages.
- V3.3 Automatic Security Review. For certain packages that use SETUID binaries or offer D-BUS services or PAM modules, an automatic security review takes place. The check is executed in the rpmlint phase of the package creation and if one of the security policies applies  the package can't be compiled. To lift the restriction on building the package the package maintainer must contact (open a ticket) the Security Team for review. As long as the package is not in the white list, it can't be submitted to Factory.
- X3.1 The Package or Project Maintainer creates a submit request for Factory.
- X3.2 The Package Maintainer decides to change role to Package Developer and branch the package in his/her home project.
- D3.1 Submit request to Factory. A similar submit request can be created and sent to Factory. The Project Maintainer can group a set of submit requests into one single request which can be accepted or declined as a whole.
4. Review Process
The review process is only done for Factory.
- To improve the quality of the submit requests.
- Potentially simplify the integration of the request into Factory.
- To detect the most usual errors in: legal, technical (intro-package and inter-package) and security aspects.
- Avoid the problem of accepting interdependent packages. With the review you can assure that before you commit into Factory, all the related packages meet a certain level of quality and follow certain policies, preventing Factory breaking due to missing packages if some of them are not up to the quality standards.
Because this review process is a full verification step, this process is composed only by verification steps, not tasks
- V4.1 Legal Review. The legal review is split in two parts: (1) the script Factory-auto  checks if the license(s) in the package have changed or not. If there is no change and the license is in the license database this script will accept the request. (2) If the package contains a change of the license, or is a new package with a license not found in the license database, the legal team has to manually check the request and accept or decline.
- V4.2 Factory Auto Review. Factory-auto is a script  which does a shallow technical review, trying to find the most common problems usually found in requests. For example, this script checks if the associate package is currently successfully compiled in OBS. This check is fast and is used to decline the most evidently bad quality requests.
- V4.3 Technical Review. The review team checks the more complicated technical aspects of the request. The Technical Reviewer will check if the specification file follows the set of predefined policies  , look over quality of the code in patches and verify the content of the changelog file.
- V4.4 Check Repo Review. This review is also automatic. Is executed by the check-repo script  and checks for typical integration problems. For example, this bot checks if certain dependencies are present in the Factory project, that the dependencies from Base:build are all self-contained in this same special project, or that the new group of request doesn't create new problematic dependency cycles in Factory.
- X4.1 All the reviews are positive.
- X4.1 If there is a negative review from one of the verification points, the request is declined and Hermes sends a notification to the sender of the request. Some automatic checks flag the request if it fails but do not completely reject the package. If this is the case, an action is needed from the submitter so it can be considered as an exit criteria. OBS also allows comments on the request, some of which need an action to be taken by the submitter.
- D4.1 The review process changes the status of the request, showing at every moment what reviews are pending, accepted or declined.
5. Factory Integration
- Bring all packages together in one tree, creating a consistent whole
- To be used as the base for the new products ISO image
- Ensure packages flowing into Factory follow feature freeze and timing rules
- Ensure consistency of other branches of Factory (milestones, Beta's, RC's, Release)
- E5.1 A submit request has been submitted from a devel project and has gone through the review process
- E5.2 The Factory Maintainer decides to fix an issue inside Factory
- T5.1 Monitor Factory. On a periodic basis, the Factory Maintainer checks how Factory is doing, and if there are any problems (which solution is outside the scope of the Factory Maintainer role) he / she sends reminders to the correct roles, or reverts the packages that causes the problems. The Factory Maintainer uses three types of monitors: the Build Monitor showing the current stage of all process (failing packages)  , the Submit Request queues  and the Users.
- T5.2 Maintain XML files used by OBS to generate ISO images. The Release Maintainer updates and fixes the XML needed by KIWI  to generate the ISO images for the products. This process is not currently documented.
- T5.3 Integrate the submit request. The Factory Maintainer accepts and integrates the submit request from the devel project to Factory. In order to avoid an overload of the OBS servers, the Factory Maintainer accepts the request following some basic rules based on experience (as in, not documented) :
- Do not accept a SR when a new build is in process.
- If a leaf package is accepted, OBS will trigger an small subset of packages.
- Base or core packages can lead to a lot of re-compilation in Factory. These are usually delayed until the end of the week.
- T5.4 Revert or remove packages. If the maintainer of the package does not respond in time or if the package causes great pains, the change can be reverted or the package removed.
- T5.5 Issue notification. The Factory Maintainer detects and notifies the Packages or Project Maintainers when there is an issue related with the package in Factory.
- T5.6 Create staging projects. To simplify the integration process and issue detection the Factory Maintainer can create staging projects. A staging project usually contains a new important package (like GCC or Perl) and all the dependencies and otherwise related packages. In this way the maintainer can detect new issues related only to the new package, avoiding issues caused by other integration actions or updates taking place.
- T5.7 Create Patterns. The Factory Maintainer creates and update the patterns needed for the installation . The patterns format and purpose are described here  
- V5.1 Is Factory open for submissions? Check for freezes or other reasons not to accept SR's
- V5.2 Brief review. The review process filters low quality submit requests, so the Factory Maintainer does not verify quality. Instead he/she judges if this is the proper time to accept the request and start the build process. The Factory Maintainer usually does a simple review of the request, and can decline or put it in hold if:
- The submit request is (related to) a package which is currently frozen AND the package is not fixing an issue
- The package will break so many other packages that a staging project will be needed
- X5.1 The package is integrated in Factory
- X5.2 The package is declined and a event in reported in Hermes
- X5.3 A bug report is created in bugzilla
- D5.1 Package integrated into Factory
- D5.2 A working Factory repository
6. Quality Assurance Process
- Determine the quality of the product generated after T4. Factory Integration.
- Test ISO images which are candidates to become Milestones or Beta 1
- Try to identify real bugs (integration bug or package bugs) in the ISO build (not FIX problems)
- Quantify the final quality of build, detect errors and update Bugzilla with the new errors to be fixed.
- E6.1 There is a new automatic build generated by OBS
- E6.2 The Release Manager or the Factory Maintainer has decided to test a concrete build image, candidate as a final product (Milestone or Beta)
- E6.3 The QA team has decided to test a new feature or reproduce a bug reported in BNC 
- T6.1 Create a new test. If the QA Team decides that the current coverage of the test set is not enough, they need to write a test that covers the new feature / use case.
- T6.2 Test maintenance. The way to make tests for Factory is using openQA. This tool requires the maintenance of certain components named needles, used in the tests to assert outcome conditions . These needles need to be updated and adjusted when an important change in the artwork or other areas effecting the look and behavior of the distribution occurs.
- T6.3 Create bug reports. openQA detects problems in the distribution, but it does not specify the exact issue which affects the ISO image. The QA Reviewer will dig into the failed test, deduce what is wrong and document the findings in a bug report published in bugzilla. This bug is assigned to the correct developer to fix the bug in the T2. Package Development task or in T3. Devel Project Integration.
- V6.1 Test review. The tests are grouped according to small set of criteria: the phase of the test (installation process, text applications or graphical applications) and the relative importance (critical, important, normal). The tool can mark one ISO as failed if critical tests are not successful passed, or if certain tests fail in the installation process. If the final result in openQA is not green the ISO has to be debugged and a bug report be written (T6.3)
- X6.1 The important and critical test are green and the ISO is considered candidate for a Milestones or Beta release
- X6.2 An important bug is detected by the tool in some configuration and the ISO is rejected as a viable product candidate.
- D6.1 Test report. The application generates a test report with screen-shots and status of the test. This report is reviewed by the QA Reviewer.
- D6.2 Bug report. If a bug is found (critical or not), the QA Reviewer creates a bug report with the description of the bug and assigns it to the correct maintainer of the package.
7. Release Process
- E7.1 It is 2 days before the chosen release date
- T7.1 Update roadmap. The Release Manager adjusts and communicates new roadmap.
- T7.2 Build Images. Using KIWI, OBS creates ISO images that are specified in a XML document. Different products (ISO images) have different XML documents.
- T7.3 Generates signatures. The autobuild team generates signatures in the name of SUSE and generates a delta ISO against the last milestone
- T7.4 Calculate hashes. Release manager calculates the ISO MD5/SHA1 hashes of the images
- T7.5 Create list of changes for marketing. Release manager generates differences between current and previous milestone with dvddiff script and sends it to marketing together with other important notes. The resulting list is shortened to a more manageable number (between 10 and 30) based on the importance of the updates.
- T7.6 Branch sources for Beta. In case of Beta, release manager branches all the sources in Factory off. When the branch is made, Factory is open for development again.
- T7.7 Create announcement. Marketing manager creates announcement based on diff and the template on progress.o.o 
- T7.8 Sync image to mirrors. Release manager syncs the image and binary repository to the mirrors . Release manager notifies mirror admins about the new iso's and the proposed release date by mailing firstname.lastname@example.org. Release manager publishes the image on the mirrors (making files readable)
- T7.9 Update Staging. Release manager updates staging area, metalinks for bittorrent
- T7.10 Enable product in NPP. Release manager enables the new milestone as a product in the Novell Product Page in bugzilla
- T7.11 Publish announcements. Marketing manager publishes announcements
- V7.1 Pick image. Release manager decides, based on openQA output, that a potential release image is good enough. If the release is an early milestone, the Release Manager can decides to release using a different criteria than openQA output. The closer to the release, the more the Release Manager raises the threshold for releasing.
- V7.2 Discuss roadmap for Beta. If this is Beta, a meeting with the product manager and Management is set to discuss the state of Factory and the roadmap. There is a go / no-go meeting for the final release with the Release Manager, Controller, QA Reviewer, the Product Manager and management where it is decided to delay or not.
- V7.3 Check communication. The Release Manager checks if the delay (if any) is big enough that it has to be communicated
- V7.4 Check Sync state. The Release Manager checks if the image is synced and published to the mirrors
- X7.1 Release is available and announced
- D7.1 ISO images for all supported platforms and architectures
- D7.2 Release announcement text for news.o.o and mailing lists
- D7.3 In case of delay: announcement and new roadmap
8. Public Test
After the release process, a second QA is done by the community (developers and users). This test is not automatic and is usually done on real hardware
- Determine the quality of the product generated after the release
- Identify real bugs in real hardware and in real use case
- Detect errors and update Bugzilla with the new errors to be fixed.
- E8.1 A new release (Milestone or Beta) is published on sofware.o.o 
We can consider this action as a task or as a verification point. If this is a verification point, there are no real task assigned to this action.
- V8.1 Test in real machine. The user (or developer) can go to software.o.o  to download the release and install it in the real hardware. If he / she detect a bug a new bug report is created in bugzilla  or / and a mail is send to the Factory mailing list. Also, many users decide to test Factory directly updating the repositories , without installing the released media.
- X8.1 A new release is published
- D8.1 Bug report. If a bug is found, the tester creates a bug report with the description of the bug. The QA Reviewer can reproduce the bug and assign the it to the correct developer.
1. Roles and Responsibilities
There is a large number of people involved in Factory development. We have identified the following roles:
- Product Manager
- Responsible for planning features
- Contact: email@example.com
- Release Manager
- Sets up the release schedule and oversees all phases of release and makes adjustments where needed.
- Contact: firstname.lastname@example.org
- Factory Maintainers
- Act as backup for package maintainer and help with more complex integration issues.
- Contact: email@example.com
- Package Developer
- Creates a package, adds features and updates it to the newer version.
- Package Maintainer
- Fixes bugs in the package.
- By default the person who created the package.
- Existing maintainers (if not active, Devel project maintainers) can add new maintainers if they see sustained good contributions and care for the package.
- Devel Project Maintainer
- Reviews requests going to the devel project.
- Super Devel Project Maintainer
- Acts as backup for devel project maintainers. This role is implemented in OBS by having the "factory-maintainers" group in the role "maintainer" in all devel project.
- Contact: firstname.lastname@example.org
Following tools are involved in specific roles:
- Factory Auto
- Checks whether some basic packaging standards are met.
- Legal Auto
- Automatic check that evaluates whether manual review by Legal Team is necessary.
- Factory Repo Checker
- Does deeper checks (package builds, installs, doesn't colide, …) before letting package pass.
And Following teams are involved:
- Review Team
- Responsible for maintaining the quality of packages submitted to Factory
- Each submission to the Factory has to be reviewed by member of this team who checks whether software is packaged correctly and written according to our packaging guidelines.
- Description: https://en.opensuse.org/openSUSE:OpenSUSE_review_team
- Contact: email@example.com
- Legal Team
- Checks whether we can legally distribute a package.
- Reviews whether declared license is correct and looks for possible conflicting licenses
- Contact: firstname.lastname@example.org
- Security Team
- Autobuild Team
- Generates SUSE signatures and delta's between ISO's
- Documentation Team
- QA Team
- Marketing Team
When a contributor wants to work on a package, the first step is to create a copy of the original package. OBS will establish a relation between the original package and the copied (branched packaged). Both packages can evolve separately until a merge process occurs (the integration process).
Management tool used to collect, monitor, assign and update bug reports. The project is using the bugzilla installation from Novell . This site is also known as BNC.
When a developer (contributor) is working on a package, he / she needs to register the changes made. This registration process is known as committing, and is done in OBS using the OSC command line tool or automatically in the web interface.
Project which is a upstream part of Factory. Factory is composed with a set of different Devel Projects that are integrated to build the next release version. In  there is more formal description of what is a Devel Project. A Devel Project can have one or more maintainers who decides the features (roadmap) implemented for this package and work to integrate them in the project.
Formal methodology to describe a process developed by IBM in 1985. This methodology is classified as narrative, instead of graphical  (flowcharts).
Factory is a project stored in OBS that integrates the different Devel Projects with the goal of creating a future openSUSE release. The work in Factory is usually done by the Factory Maintainers (integrate the different project packages) and the Release Managers (create releases based on ISO images from Factory).
To make the integration of the devel project into Factory easier, the Release Manager establishes a calendar with periods during which certain packages are now allowed to be updated (are frozen). Only bug fixes (and no new versions or new features) are allowed until the Release Manager and the Factory Maintainer start a new release cycle. Exceptions can be made.
Hermes  is the central notification hub of the openSUSE Project. It gives users control over the kind of notifications they want to receive, in which way and when.
Every developer has a home project assigned which can be used to develop and experiment without interfering with the rest of the packages stored in the different projects. When a developer branches a package from a devel project or from Factory, the copied package is stored automatically in his or her home project.
The openSUSE KIWI  Image System provides a complete operating system image solution for Linux supported hardware platforms as well as for virtualisation systems like Xen, Qemu or Vmware.
The Open Build Service (OBS)  is a generic system to build and distribute binary packages from sources in an automatic, consistent and reproducible way. Is used to release packages as well as updates, add-ons, appliances and entire distributions for a wide range of operating systems and hardware architectures.
With openQA  we can automatically run tests over a distribution. Some tests are designed to make a full install of the distribution in a virtual (or real) machine, with different configurations of file systems, desktop environment, architectures, medias, etc.
Command line tool used to interact with OBS
A source package is a list of files (usually a tar file and resources like patch files) and a specification file that can be used to generate one or more binary packages
A predefined set of packages related with a specified configuration or functionality. Patterns are used to install a set of packages that can form a usual configuration (laptop, server, desktop, developers, office user, etc.)
Source package container.
Binary package container. Test of binary packages compiled for a specific architecture.
A plan that matches short-term and long-term goals with specific technology solutions to help meet those goals
Temporal projects used to test critical components before they are submitted to Factory. Staging Projects can be created and destroyed on demand by the Factory Maintainer, but some of them, like  are more general and are reused frequently.
Digital document established by a developer and managed by OBS that describe the set of changes done for a source package.
 openSUSE Roadmap: http://en.opensuse.org/openSUSE:Roadmap
 openSUSE Goals: http://en.opensuse.org/openSUSE:Goals_13.1
 openSUSE Features: https://features.opensuse.org/statistic/product/opensuse_131
 XML Roadmap: http://turing.suse.de/~coolo/opensuse_13.1/opensuse.xml
 SMILE view of the Roadmap: http://turing.suse.de/~coolo/opensuse_13.1/
 openFATE: https://features.opensuse.org/
 openFATE description: http://en.opensuse.org/openSUSE:Openfate
 Open Build Service (OBS): https://build.opensuse.org/
 Factory Release Process: https://en.opensuse.org/openSUSE:Release_team_processes
 Factory-Auto and other scripts: https://github.com/coolo/factory-auto
 openQA: http://openqa.opensuse.org/
 openQA (Internal link): http://opensuseqa.suse.de/
 ETVX Methodology: Radice, R.A.; Roth, N.K.; O'Hara, A.C., Jr; Ciarfella, W.A. " A Programming Process Architecture" IBM Systems Journal Vol.24, No 2 (1985), pp.79-90.
 Software openSUSE (devel): http://software.opensuse.org/developer/en
 Roadmap (openSUSE 12.2 report): https://wiki.innerweb.novell.com/index.php/OpenSUSE_team_report_12.2#Schedule
 OBS Tutorial: https://en.opensuse.org/openSUSE:Build_Service_Collaboration
 Delay announcement: http://news.opensuse.org/2012/06/14/where-is-my-12-2-my-kingdom-for-a-12-2/
 Analysis of the delay: http://lists.opensuse.org/opensuse-factory/2012-06/msg00468.html
 Bugzilla Novell: http://bugzilla.novell.com/
 Hermes: https://hermes.opensuse.org/
 Hermes description: http://en.opensuse.org/openSUSE:Hermes
 Create a package with OBS: http://en.opensuse.org/openSUSE:Build_Service_Tutorial
 Package policies and guidelines: http://en.opensuse.org/openSUSE:Packaging_guidelines
 OBS plugins: http://en.opensuse.org/openSUSE:OSC_plugins
 Some OBS plugins: http://en.opensuse.org/openSUSE:Build_Service_Tools
 Specification file: http://en.opensuse.org/openSUSE:Specfile_guidelines
 SUID bits: http://en.opensuse.org/openSUSE:Specfile_guidelines#SUID_bits
 Factory submissions: http://en.opensuse.org/openSUSE:Factory_submissions
 Review process: Factory Review Process
 Review team: http://en.opensuse.org/openSUSE:OpenSUSE_review_team
 Build monitor: https://build.opensuse.org/stage/project/status?project=openSUSE%3AFactory&filter_devel=All+Packages&ignore_pending=true&limit_to_fails=false&include_versions=false&commit=Filter+results
 Submit request queues: https://build.opensuse.org/project/requests/openSUSE:Factory
 KIWI: http://en.opensuse.org/Portal:KIWI
 Repository metadata: https://en.opensuse.org/openSUSE:Standards_YaST2_Repository_Metadata_patterns
 openSUSE Patterns: http://gitorious.org/opensuse/patterns
 Pattern description: https://lizards.opensuse.org/2009/10/07/about-patterns-versus-packages/
 Release process video (Part 1): http://streaming.nue.suse.com/i/lunch_learn/lunch_learn_opensuse_122_release-part1.webm
 Release process video (Part 2): http://streaming.nue.suse.com/i/lunch_learn/lunch_learn_opensuse_122_release-part2.webm
 Progress.o.o: http://progress.opensuse.org
 openSUSE mirrors: http://mirrors.opensuse.org/
 Package maintainers & developers: https://build.opensuse.org/package/users/[projectname/[packagename]]
 Devel Project Maintainers: https://build.opensuse.org/project/users/[projectname]
 Super Devel Project Maintainers: https://build.opensuse.org/group/show?group=factory-maintainers
 Review-team: https://build.opensuse.org/group/show?group=opensuse-review-team
 openQA Tutorial: https://github.com/openSUSE-Team/os-autoinst/blob/master/doc/tutorial.pdf
 openSUSE-Factory Mailing List: http://lists.opensuse.org/opensuse-factory/
 Devel Project Concept: http://en.opensuse.org/openSUSE:Build_Service_Concept_Devel_Project
 Factory Staging Project: https://build.opensuse.org/project/show/openSUSE:Factory:Staging
 How to install Factory: http://en.opensuse.org/openSUSE:Factory_installation