Testing Team

From openSUSE

Want to help test?

Basic Info about Testing Submitting Bug Reports Bug Reporting FAQ

Members

user

Testing team slides

See proposal

This article is a stub! This article needs to be expanded. You are welcome to help in line with the openSUSE:Wiki Style Guidelines.

The Problem

In earlier releases most people just started to test things during the late betas or the release candidate. Naturally this lead to the developers getting swamped with bugreports only during the last weeks of the development cycle when they were busy finishing their work anyways.

Quite understandably the developers weren't able to deal with all of those issues until release. Also, every update that gets released after the official release has must go through QA. This doesn't help to speed things up nor does it prevent the release of minor updates. Now if such a situation coincides with a holiday season like Christmas it leads to stuff not getting fixed for quite some time as we experienced with 11.1.

The Solution

To prevent something like this from happening again we obviously need thorough testing. And this must not only happen during the last weeks of the development cycle but throughout the whole development cycle, even for early Milestones.

During some discussions / brainstorming we realized that this would be best done by creating "Yet another web application" that lists all test cases and gives everyone who is interested the opportunity to contribute. That way it would be pretty self organizing instead of relying on people to coordinate it so people can come and go as they please.

The Webapp

The most important point is that the entry level should be as low as possible so even a newcomer is able to test (basic) stuff. This should also be reflected in the user interface. E.g. there should be a big fat "Gimme stuff to test" button which just selects a random testcase (based on the selected difficulty) but also a list / search function so one can easily find & test one's pet peeve.

This web app is also supposed to not only list testcases for newly implemented features, but also old & common features (like e.g. "Burn CD with K3B") so we are able to create a list of common use cases that need to be tested for every version.

To achieve this the application has to list every testcase with the following attributes:

• A description of the feature to test.
• The expected outcome / steps to correctly test the feature if it isn't plain obvious.
• The packages required to install to test the feature
• A link to the feature tracked in [open]FATE for easy tracking.
• A link to the bugzilla entry if the test failed.
• The date of the last successful test (Milestone X Version Y)

Ideally the whole list gets worked through for every single Milestone. Obviously this all depends on the number of people participating.

Further thoughts

• The testcases should be linked to the packages so they can be easily listed for the next openSUSE version or removed if the related package is dropped.
• The list should have some expressive background (e.g. green for successful tests, red for failed tests and something neutral for "to be done" tests so it is easy to get an overview of the current state by simply scanning through the list (besides the normal statistics).
• Testcases should get a rating - e.g. "Easy" for everyone testable, "Special" for not so common stuff and so on so a "newbie" can select to get only "easy" stuff to test which one actually is able to handle.
• Perhaps some ranking with number of tests run per person would encourage people to compete and thereby test more with some goodies like a geeko plushy for the top 10 (obviously besides the eternal fame one earns anyways).

Obvious stuff

• There need to be links to wiki pages explaining how to file useful bugreports in a highly visible area / frontpage - e.g. how to get a useful backtrace.
• A search function not only to do a text search but also to list e.g. all new features or just the "easy" ones or all failed tests and so on.
• The testing should be integrated with the internal Novell / SuSE QA. What integration points exist?

Implementation [Details]

• The idea is that the webapp is pretty much selforganizing the whole testing by distributing testcases to willing testers so we wont need people organizing anything but everything is handled automatically.
• A tester is presented a testcase which one can accept or not. If a test is accepted the testcase is locked and the tester has 24 hours to finish the test. If no feedback is given within that time the testcase is unlocked and open to all again (that's simply to avoid duplication by several people testing the same thing).

Final Thoughts

Obviously that approach wont catch every single bug straight up from the beginning. But it will allow us to create an exhaustive list of testcases and, with the help from the community, by going through that list we will be able to catch more and more bugs so we finally might achieve some "It just works" state (hopefully).

Nevertheless the whole thing stands & fails with people participating and, most importantly, starting early to test & file bugreports so developers wont get swamped a few weeks before release but have enough time to resolve those issues.

That's my 0,02$, every comment / feedback / enhancement is totally welcome.

Further / Related Ideas

• Cyberorg suggested to integrate it somehow with the Desktop - e.g. by putting a link on the Welcome page that is shown after a new install.
• Beineri suggested to have a look at Testopiawhich is a test case management extension for Bugzilla.

Related Links

• Testopia Homepage
• Fedora's Testopia Evaluation Site