Libzypp/Testsuite repo

From openSUSE

Testsuite 1 (aka pokus) for ZYpp

The goal of this testsuite is to verify the correct behaviour of the repository parsers and the SQLite interaction.

It is mostly targeted towards module tests by the libzypp testsuite. This testsuite is now included in the package libzypp-testsuite-tools.

For integration tests, see 'Testsuite 2' below.

Motivation

We already have (unfinished) parsers and writers for YUM and YaST repositories as well as (unfinished) bindings for Ruby. So a light framework will be created that allows easy generation of repositories and verifying of the resulting pool.

Partly implemented checks (for details look at the source):

• Verify correct architectures in pool.
  • For multi-arch repositories (like factory), only packages with a compatible architecture should appear in the pool (one cannot install ppc packages on x86 anyways, rpm will barf on that)

• Adding and removing of repositories (although very simple).

• Verify modifications the parsers make to the metadata
  • Self provides of packages. Every object must internally list its own name and version as a 'provides' dependency. The C++ constructor for objects within libzypp should ensure that.

• Verification of attributes:
  • Some provides of packages. Check if all metadata belonging to a package is actually parsed, stored in the cache database, and correctly retrieved again.

• Special check for bug 298746.

• Verify signature of repositories.

Planned checks:

• Veritfy that attributes of all resolvable kinds are handles correct.
  • Esp. when the metadata is only present once for a package that is present for different architectures. Background: In order to save metadata size, repeating metadata for packages appears only once and other packages refer to it. This is done by the Shr tag in susetags metadata. This sharing is esp. effective for multi-arch repositories where a (single) package appears in multiple architectures but shares a single summary, description, authors list and changelog.

• Refreshing of repositories.
  • Changes on the metadata repository must be detected and (only the changed portions of the) metadata newly downloaded, parsed and cached on the client.

Testsuite 2 for ZYpp

Goal of this testsuite is to verify appropriate handling of errors from the users perspective.

YaST and zypper must report

• what: What went wrong
• why: Why did the error occur
• how: To fix the problem / to proceed

Planned checks:

• for HTTP, FTP and NFS servers
  • Server does not respond at all
    • Bug 300527

 Purpose: check proper error reporting
 Setup: add an online repository, unplug network connection
 Run: zypper refresh -f
 Run: yast "software repositories", add <url of server repo>, click on "refresh now"
 Check: zypper error report

• • Server does not respond during download

 Purpose: check proper error reporting
 Setup: add an online repository
 Run: zypper refresh -f
      unplug network connection during download
 Run: yast "software repositories", add <url of server repo>, click on "refresh now"
      unplug network connection during download
 Check: zypper/yast error report

• • Server responds very slow (thus no hard timeout is triggered)
  • Auth failure (HTTP and FTP)

 Purpose: check proper error reporting
 Setup: ftp or http server requiring authorization
        copy packages to server
        run 'createrepo'
 Run: 'zypper ar <url of server repo> testing'
 Run: yast "software repositories", add <url of server repo>, enter wrong credentials
 Check: zypper/yast error report

• • Other failure (HTTP and FTP)
    • Bug 305113 server returns 403:Forbidden

 Purpose: check proper error reporting
 Setup: ftp or http server
        copy packages to server
        run 'createrepo'
        make the server return 403 on access
 Run: 'zypper ar <url of server repo> testing'
 Run: yast "software repositories", add <url of server repo>
 Check: zypper/yast error report

• Network proxy
  • wrong proxy
    • Bug 299775

 Purpose: check proper error reporting
 Setup: ftp or http proxy, without authorization
        configure this proxy on client; mistype proxy config on purpose and test again
        'zypper ar <any online repo> testing'
        Enter Url in yast "software repositories"
 Run: 'zypper refresh testing'
 Run: click on "refresh now" in yast
 Check: zypper/yast error report

• • wrong auth

 Purpose: check proper error reporting
 Setup: ftp or http proxy, with authorization
        configure this proxy on client, enter wrong proxy credentials
        'zypper ar <any online repo> testing'
        Enter Url in yast "software repositories"
 Run: 'zypper refresh testing'
 Run: click on "refresh now" in yast
 Check: zypper/yast error report  

• for CD/DVD media
  • IO errors

 Purpose: check proper error reporting
 Setup: Put scratches on install CD
 Run: 'zypper install <package from CD>'
 Run: yast "software management", view "by repo", choose a package from CD
 Check: zypper/yast error report  

• • wrong media

 Purpose: check proper error reporting
 Setup: Get a CD from a previous beta and insert it into CD drive
 Run: 'zypper install <package from CD>'
 Run: yast "software management", view "by repo", choose a package from CD
 Check: zypper/yast error report 

• URL incorrect
  • wrong scheme (e.g. htp instead of http)
    • Bug 300121

 Purpose: check proper error reporting
 Setup: -
 Run: 'zypper ar <mistyped url> testing'
 Run: yast "software repositories", add <mistyped url>
 Check: zypper/yast error report 

• • wrong format (e.g. http:/foo.bar.com, ftp:baz.bar.com)

 see above

• • non existing location
    • Bug 306667

 Purpose: check proper error reporting
 Setup: -
 Run: 'zypper ar http://does.not.exist/foo/bar testing'
 Run: yast "software repositories", add 'http://does.not.exist/foo/bar'
 Check: zypper/yast error report 

• • not a repository
    • Bug 298506

 see above

• • wrong product/version/architecture

• Repository corrupt

 Purpose: check proper error reporting
 Setup: ftp or http server
        copy packages to server
        run 'createrepo'
        run 'zypper ar <url of server> testing'
        run yast "software positories", add <url of server>
        manually edit files below 'repodata'
 Run: 'zypper refresh testing'
 Run: yast "software positories", choose <url of server>, click on "refresh now"
 Check: zypper/yast error report for various changed files

• • files truncated

 same as above, truncate files manually

• • files missing
    • Bug 301022
    • Comment #9 of Bug 302435

 same as above, remove/rename files

• • wrong file references
    • Bug 296332
  • parse errors in files

 same as above

• • inconsistent state (esp. for mirrors)
  • binary garbage
  • manipulated files (e.g. wrong or missing signature/checksum)
  • other errors, e.g. repository with unsatified dependencies (esp. for repositories created by users)
  • File not found
    • Bug 299095

• Repo signing
  • unknown key
  • incomplete signing (wrong repo setup)
    • Bug 302043
    • Bug 302059

• Misc errors
  • something is locked, e.g. ZYpp or RPM DB
  • unsufficient permissions (e.g. not root, read-only mounted filesystem or refusal by AppArmor)
  • disk full (e.g. for cache)
  • inconsistent local data (config or cache), e.g. after user abort or crash