openSUSE:ALP/Workgroups/Git-Packaging-Workflow/WorkPackages/Distro-Management

Mission

Build a declarative, templated way to define sets of package source assets, their build configuration and other metadata that is modular, extensible, reproducible, auditable, and long term maintainable

Goals

Exactly what problem will this solve?

• Lower the cost of of maintaining products over what we have today by making it easier to instanciate “tailored” variants that can (marginally) deviate for the common pool of available packages (“common codebase” today)
• Improve on our supply chain story by improving auditability and security. Any change to the project setup can be retroactively inspected, verified, and potentially reverted if necessary.
• Provide “bi-sectability” by tracking all packages, their metadata and the project metadata in one place.

How it is

There are different workflows currently for each distribution change:

• changes to the projects configuration (prjconf, repositories etc.) usually are sent by e-mail to a project maintainer, bypassing peer review, testing and change control
• adding or removing a package requires custom handling to ensure integrity of the package set (to prevent dropping of still required packages)
• setting up a new “similar enough” project requires cumbersome copying of packages and creating links to workaround quirks in the build service. manual work that is often error prone
• package updates are going through a full review, even though they went through the same review already in another project/distribution, leading to duplicated efforts
• larger reworks of package sets can not easily be batched together, as the current system works package/submit request based
• The project status is not tracked globally uniformally in the history of changes (floating links, project config, permissions)
• If packages uses git for their tracking, we should need to consolidate the split SCM / build at project level

Consequences

• setup of new distribution “incarnations” (SPs) is very cumbersome and labor-intensive, requires custom checklists and manual coordination, making the process slow, expensive and errorprone
• duplicated source reviews
• reviews only on package level, not on project level

How it should be

Setup of new distribution incarnations should be automatically instanciateable using a declarative description

Creation of customer/customer segment specific distributions from a common pool of package sources (potentially with local modifications/variations) should be very low effort.

It is also relevant that any change in any package version, is also tracked as a new commit in the history of the project. Defining a project in a way that can be tracked under git will allow the use of branches, merges, or cherry-picks into the project itself. Which makes it possible to have a branch of a distribution, with different set of packages or properties, but that the release manager can rebase back the changes from the main distribution or even different code bases.

At the same time provides the same symmetry than in the package version, as separating the SCM from the build server allows the creation of local distributions using pbuild, reusing all the tools for SCM. This model is more similar to others used by embedded developers, like buildroot or yocto, but at the same time fully support the OBS model.

A distribution it is more that an aggregate of packages. It is also compose of some metadata like the project config, that can change drastically the deliverable that it produces. There is also a list of architectures that it is supported, and the list of users and roles that can influence on it.

Can also contain information relatively to other services, like instructions about how to generate an ISO image or a public repository. But also includes some business level information, like what sections are supported or are in tech preview state, or the license attached to the final appliance.

All this information should be tracked in the history of changes. If a new change in the project config has been made, the git history should give me the diff, the user, the time and the chance to revert or change it, as part of the same git repository.

In that way any change that can deliver one or another final artifact will be completely represented by the set of commits of the project repository, giving us a set of properties that we can use later

Builds should be back-linking to one complete project SHA1 of the HEAD commit used to build the artifact. This makes it possible to compare if two systems are equivalanet by just comparing the SHA1. For image-based deployments this is an important feature, and it allows to efficiently compare two different builds.

For whom do we solve that problem?

Release managers, source reviewers and distribution maintenance

Why now?

ALP will as a new platform inspire the need to build multiple products out of the same source package pool.

What factors are critical to success?

[in progress]

File:Git workflow sketch.pdf