openSUSE:Factory-tested Proposal
Overview
Provide a single, well-known location with information about whether or not Factory is broken, and a link to the last known-good Factory tree.
Problem Space
Sometimes Factory is broken. Brokenness typically comes in one of these types:
- Installer
- Installer images missing
- Installer doesn't work
- Repo
- Metadata is missing
- Package/file conflicts
- Dependency problems
- Functional
- Major bugs with packages in the "Critical Path" (see openSUSE:Critical Path Packages Proposal)
When Factory is broken it's much harder for people to become Factory testers, harder for developers to get fixes into the hands of testers, and other such badness.
Proposed Solution
First, we should create test cases for each of the types of breakage described previously. These should be automated or scripted when possible.
These tests should be run regularly (daily if possible) by QA team members and other interested parties. The results should be submitted and displayed prominently on a public website.
When a tree is deemed good (i.e. when it passes all the tests) someone from QA should file a ticket with rel-eng to create a "known-good" symlink for that Factory tree. This link should be public and well-known. The name "Tumbleweed" (alias: factory-tested, factory-snapshot) is suggested for this known-good Factory.
Scope
- QA Team members (i.e. testers) will need to do regular (daily) testing
- rel-eng will need to move the symlink when requested
Active Ingredients
- Define requirements for Good vs. Broken
- Write test cases to check each requirement
- Automate all test cases
- Public website that shows results of daily Factory testing
- Set up a known-good Factory ("Tumbleweed") link based on those test results
- Publicize a link to the "Tumbleweed" known-good tree
Roadmap
...ToDo...
- openSUSE:QA Factory Acceptance Test Plan used to define Good vs. Broken.
- identify test cases in that test plan
- create a "isfactorybroken" page
Results
- Many important test cases have been automated on openqa
- code: [1]
- There is a snapshot repo available for updates and net-installs at http://download.opensuse.org/factory-tested/repo/ - this can be useful in cases when you would use Factory, but want to reduce the risk of having to recover your system (of course, the risk is still not zero). This repo gets updated with new packages from a Staging project, whenever the tests showed, that basic functions (installing, updating, X11) were still working.
Discussion Points
Need to list requirements / solicit mockups for the webapp.
Comments?
To leave a comment, use the Talk page for this proposal.