Libzypp/Refactoring/Plan

From openSUSE

Plan

• Speed is #1 priority for ZYPP in 10.3
• Typical scenario: install one package via ZYPP

Suggested solution: Use a sqlite database for storing the metadata

Database requirements

• one table of each kind of resolvable (combining resolvables from all sources) The detailed requirements are to be investigated.
• raw cache will be preserved in order to detect changes (refresh), recreate/repair database if needed, etc.

1st Step

• create SqliteSourceImpl class
• move YUMSourceImpl and SuSETagsSourceImpl to a DB feeder

Prerequires

• create a DB schema
• create the DB layer, query + write API

Problems

Having one read-only database would break add-on products during distro installation. Possible solutions are using multiple databases as well as merging all databases into one in-memory database.

Other notes

• no changes to applications using ZYPP will be required, except changes in the source management (source add/edit/delete/initialize)

Benefits

• parsing at ZYPP startup is faster (preparsed data is read from the database);

 raw metadata is parsed only when needed (after source has changed).
 This should speed up the most common tasks
 Refresh/database update is separated from ZYPP startup

• large and seldomly used metadata (eg. package description/summaries) don't have to be stored in memory (can be read from database on request)

Further steps

1. Only create in-memory representations of objects on demand
   • be able to cope with hundreds of repositories
   • achieve better scalability (due to on-demand creation of objects)
   • dependency-based transations (let solver chose the best package which fits to the system)
     This is one possible implementation of "keep state information within the application". If the application keeps a 'pool' inside or tracks 'dependency-based transations' we cannot influence. However, we all agree that transactions will be represented as dependencies within zypp in the future.
   • any estimation cannot be done now; we will be continuously working on this in order to achieve reliable effort estimations
2. extend dependencies
   • conditional dependencies (kill atoms)
   • dependency expressions (X or Y)
   • needs detailed discussions with autobuild team and rpm maintainer
3. software management database log
   • keep track on package/patch/pattern/product installs/upgrades/removals by time, version, repository, and user (for role based access)
4. locking (package locking, vendor-based locking)
   • needs further evaluation
5. policies
   • enhance configurability
6. generic handling of patch/delta RPMs
   • currently patch/delta can be only a part of a patch
   • allow existence of patch/delta RPMs even outside patch
7. Move from Resolvable to Selectables
   • the current implementation treats each package offered by a repository as a separate object, called Resolvable
   • this does not properly account for multiple repositories offering identical packages
   • even the same package (same summary, description and probably other overlapping attributes) available in different versions is not properly handled internally.
   • all this directly affects libzypps ability to handle hundreds of repositories with thousands of packages.
   • instead, we want to create one object per package(-name), and track different versions, architecture and repositories within this object. YaST packagemanagement already has this notion (called Selectable), we want to make it a first-class object within libzypp.

Note: The order here doesn't necessarily mean the priority of the tasks.

Functionality requirements (Step 1)

From repo to database

• Source handling
  • add
  • edit
  • delete
• Key handling
  • need additional entry in database
  • think of role-based administration
• Metadata download
• Metadata refresh
  • need to be merged to the database after refresh
• Local metadata cache handling
  • Consistency check
  • cache size
• Metadata verification
• Metadata parsing
• Upgrade from previous versions
• Target store/RPM database: convert to sqlite later - not covered by the plan for Step 1

From database to resolvables:

• resolvable definition class
• NVRAD to be read at creation time
• other data to be read on demand from DB via query API
• this description should be extended on a wikippage

Proposed plan to achieve Step 1

Tasks

• collect requirements: 1d dmacvicar
• DB schema: 1w dmacvicar
• query + write API: 1w ma (incl. first implementation draft)
• investigate: w2 kkaempf
  • add-on
  • system installation
  • distro upgrade
• move MediaSet from SourceImpl incl. testsuite: 2w jkupec
• MediaSet is a class representing a set of medias (CD1, CD2,...)
  • belonging to one source
  • currently is tightened too strong to a source
• collect Source Management Requirements: 2w mvidner
  • think of CIM
  • figure out support for future versions of ZENworks
  • consider source priority / media preferrence (in case of same package available via multiple medias)

Notes:

• contains DB size checking (installation suitability)

Internal check point 1: 2w

Research

• Tambet's repomd stream parser
• MA's SuSEtags
• Create a prototype Source -> Database: 2w (incl. research) ma & duncan
• Create a prototype Download Source: 1w jkupec
• Create SQLiteSourceImpl prototype: 2w schubi
• Create Resolvables
• Fix a Crash

Internal check point 2: 2w

Tasks 3

• SqliteSourceImpl (extend work from CP2): 1w schubi & dmacivcar
• New SourceManager: 2w (incl. items below) ma & mvidner
  • Integrate Refresh/Download
  • Create Resolvables

Internal check point 3: 2w

Tasks 4

• Adapt Pkg bindings
• Adapt zypper

Internal check point 4: 2w

Tasks 5

• Adapt source editor UI
• Internal check point 5: 2w

Integration

• pkg bindings
• QT UI
• NCurses UI
• Gtk UI
• zypper

Internal check poing 6: 2w

Testing

Internal check poing 7: 2w

Documentation

Internal check poing 8: 2w

Commit policy (before initial submit)

• ZYPP: Develop in trunk, submit 10.2 branch to STABLE
• YaST: Develop in separate branch where needed (bindings, maybe UI), submit trunk to STABLE