openSUSE:OpenQA/Partners
This is a landing page for companies who are interested in using openQA as part of their workflow. Companies, projects and/or startups are encouraged to share how they use the Open Build Service.
openQA is an automated test tool for operating systems and the engine at the heart of openSUSE's automated testing initiative.
Partner use cases
Fedora
Fedora uses the openQA (https://openqa.fedoraproject.org/) automated testing system as a significant part of the release validation testing process, and for testing updates. On this page you can find more information about openQA, how Fedora uses it, and how to install your own instance of openQA so you can try it out and contribute to test development.
How Fedora uses openQA
There are currently two official Fedora openQA deployments: production and staging.
At present, Fedora uses openQA for release validation type testing. We have developed a set of tests oriented around testing a complete Fedora distribution compose. These tests are run:
On every nightly compose of Rawhide or Branched On every 'candidate' compose (including release candidates)
A subset of tests is run on the Cloud images from the nightly Cloud stable release composes, and on each refresh of the semi-official post-release live respins.
We also run a subset of the compose tests, plus some other tests, on some of the updates under test for Fedora stable and Branched releases. The tests are run on all critical path updates, and for some other specific 'whitelisted' packages (where the testing seems specifically relevant and valuable) - they are not run on all updates as we do not have sufficient testing resources for this. The results of these tests are visible on the Automated Tests tab of the update page, for updates which are tested. Updates can optionally be 'gated' on these results.
The results of all openQA tests are forwarded to ResultsDB. When there is a release validation test event for a compose, openQA results are mapped to Wikitcms test cases and reported to the appropriate wiki pages: results you see from the user coconut are results derived from openQA testing.
An email summary of most of the compose runs is produced and mailed to test and devel by the check-compose tool.
The fedora_nightlies tool which produces the nightly compose finder page considers openQA results in determining the test pass/fail status of the images it tracks.
Future plans include 'gating' Rawhide composes on openQA test results (i.e. not pushing a Rawhide compose into the mirror system if it fails the openQA tests).
Qubes OS
Qubes OS is a free and open-source security-oriented operating system meant for single-user desktop computing. Qubes OS uses openQA to test two types of artifacts:
1. Installation image - install Qubes OS in various configurations using openQA native tests. This is included in https://openqa.qubes-os.org/group_overview/1. One of the test job there uploads installed system image to use as a base for other tests.
2. Package updates - this is basically about running integration tests with selected updates applied. This applies to packages in "current-testing" repository (candidate updates to stable releases), but also updates built from not merged yet pull requests.
The package updates testing is a quite complex setup: It takes a base disk image from "installation tests" group, configure extra repositories (based on test configuration), install them, then uploads another disk image with those updates applied. Each other test in the group takes the disk image and launch a subset of integration tests. Most of them are written in python unittest framework, then run using nose2 tests runner which can export results in junit xml format - to be imported back to openQA as external tests results. But there are also some tests as native openQA tests.
This type of tests is also integrated with github: There is a script handling AMQP events to post test results as github comments in relevant place (we use github issues also to track updates in testing) (for example https://github.com/QubesOS/updates-status/issues/1902#issuecomment-650144461) There is another script to look for pull requests with 'openqa-pending' label, download them, build and run tests from them. (In this case, results are posted as comments to relevant pull requests.)
More information about Qubes OS integration tests can be found at https://www.qubes-os.org/doc/automated-tests/#installation-tests-with-openqa
Currently all of the tests are scheduled manually on when-needed basis (for example after tagging a bunch of pull requests with 'openqa-pending' label). Future plans are to automate this step.
A challenge Qubes OS faces is that Qubes OS itself uses virtualization, so running it within a virtual machine is fragile. This basically means interpreting test results is non-trivial, as some fails just because the nested virtualization is not stable enough.
Debian & Kali
Debian is a distribution composed of free and open-source software that is developed by the community-supported Debian Project. Kali Linux is a distribution designed for digital forensics and penetration testing. An instance for testing aspects of these two distributions is available at https://openqa.debian.net/
GNOME & KDE
Both GNOME and KDE contributors benefit from the testing of nightly builds. openSUSE contributors to these Desktop Environments (DE) have their own workflow, but the tests feed into the each projects' development and bugs are reported and communicated to release managers of these DEs.