openSUSE:OBS with Git

Jump to: navigation, search

How to exercise Git-backed OBS on build.opensuse.org

Introduction

The Open Build Service (OBS) has its own internal source code management (SCM) system. Together with the command-line client osc, the user-facing behavior is roughly equivalent to Subversion (SVN), wherein a working copy (local directory where you do work and which contains a (partial) copy of the sources) traditionally contains only data from one chosen revision at any one time, necessiting the use of the network if one wishes to navigate forth and back in the commit history.

Synchronization to Git

There exists a mechanism which copies package sources from OBS to Git (just this direction). This synchronization service is not user-controllable, so just know that it exists. Packages in openSUSE:Factory get synchronized, somewhat infrequently, to https://src.opensuse.org/pool/pkgname. This is good news for the "offline browsing" usecase alluded to earlier. Packages which are already "git-ified" are exempt from the synchronization. The pool/pkgname repositories typically contain a single branch called factory. There are no permissions set. Altogether, the pool/pkgname Git repositories have overall limited use.

Synchronization from Git

Because the OBSSCM system is entrenched everywhere, it is not easy to directly replace. Instead, the SCM Bridge / obs_scm_bridge / scmsync mechanism was added which can copy package sources from an arbitrary Git repository to an OBS package entity (again, just this very direction).

The software running next to other OBS server processes is called obs_scm_bridge. Configuration happens via an OBS package entity's metadata (osc meta pkg -e): 

<package name="descent3" project="games">
        <title/>
        <description/>
        <scmsync>https://src.opensuse.org/jengelh/descent3#master</scmsync>
        <person userid="jengelh" role="maintainer"/>
</package>

The <scmsync> tag specifies the remote location. A branch may be specified using the URI fragment syntax, e.g. the "master" branch is referenced here. It is conceivable that an arbitrary Git ref may be specified.

As soon as the scmsync tag is added, the package is watched by obs_scm_bridge. Commit synchronization runs asynchronously in the background. Whenever obs_scm_bridge detects a change in the Git repository, a new OBSSCM commit is created from the files in the referenced Git revision. The Git history in its own right is not transferred; for example, forced pushes simply result in a new OBSSCM commit on top. (Hypothesis: the bridge program makes a --depth=1 shallow Git clone, and uses that result for the new OBSSCM commit.)

The synchronization is usually quick to complete (a few seconds), but slow networks or huge repositories might delay the completion. If and when it is done, the OBSSCM history will have gained a new commit with a terse commit message of the form [info=825f56ff6136e3c958676fb574b502d157b152c361e7dd43761752cb12e926d2] (Git commit hash). The OBSSCM history can be viewed by the well-known osc log command: 

$ osc log games/descent3
----------------------------------------------------------------------------
r4 | unknown | 2024-09-23 13:54:57 | 880e81b6fa73957de8f254d65997d709 | 1.6.0~g226.g616f921e |

[info=825f56ff6136e3c958676fb574b502d157b152c361e7dd43761752cb12e926d2]
…

880e81b6.. is the so-called srcmd5, kind of like an OBSSCM commit hash. Another artifact of Git-backed OBS packages is that the OBS revisions contains an extra file, _scmsync.obsinfo.

The games/descent3 package used here as an example is thus, for all intents and purposes, stored in, and to be managed with, Git. This has no immediate effect on openSUSE:Factory/descent3 or home:contributor/descent3, as sources can amicably be copied, because they all still have associated OBSSCM copies in any case.

Checking out git-backed OBS packages

Even after switching an OBS package to scmsync, the user should not stop using the osc command-line utility. Repositories downloaded with just Git are not osc-enabled and one cannot use commands like osc r without further arguments.

Furthermore, using osc up to update an existing working copy whose metadata has just been scmsync-enabled skips setting up the local Git repository. It is therefore recommended to discard working copies where a scmsync switch occurred and re-download with osc co. In git-backed working copies, the following commands no longer work, or have a replacement, or still work with caveats:

  • osc branch, osc bco: The command is deactivated in favor of git branch — more on that in the next section.
  • osc commit: Deactivated in favor of git commit and git push.
  • osc log: Shows the OBSSCM history. You mostly want git log with scmsync-enabled packages, unless you want to examine the "low-level" OBSSCM layer.
  • osc sr: Works as before for submissions to Git-disabled OBS package entities. Keep in mind that this will be submitting the OBSSCM state, and no Git history. You would use osc sr when submitting from e.g. games/descent3 (develpkg) to openSUSE:Factory/descent3. (As of 2024-09-30, openSUSE:Factory is not git-backed, so sr is exactly right.)

Branching/Contributing

Git repositories store just code/history. Unlike OBS package entities, there is no implicit build trigger and no conceptual build artifact storage defined. As a result, Git branches on their own do not lead to any implicit server-side builds being performed. But this is not necessarily a bad thing. Trivial changes like fixing up changelog spellos now no longer incur a (potentially unnecessary) implicit build in one's home: project. It is still possible to do a local build via osc build, and that may just turn out to be faster than workers from build.opensuse.org anyway, as public worker instances are often configured with just 4-way parallelism, whereas locally, one can exercise all umpteenth threads at once (e.g. 32).

CEFP: Checkout–Edit–Fork–Publish (TBD Title for a method 1)

“How would I go about branching/editing a (Git-enabled) package?” Like so:

  1. osc co games/descent3 && cd games/descent3
  2. edit descent3.spec
  3. Perform local build to test that it still works, osc build
  4. git add $changed_files, git commit -m "Whatever"
  5. Open the scmsync URL (e.g. https://src.opensuse.org/jengelh/descent3 in the example) in a browser and create a fork, e.g. at https://src.opensuse.org/contributor/descent3 .
  6. Because the default Git remote, called origin, still points to jengelh/descent3, another remote entry needs to be added for your own fork: git remote add my gitea@src.opensuse.org:contributor/descent3
  7. git push my
  8. If and only if you want to do a server-side build:
    • Vivify a dummy project to use for testing, e.g. osc meta prj -e home:contributor:test
    • Vivify a package entity to use for testing, e.g. osc meta pkg -e home:contributor:test/descent3 and add a <scmsync> line pointing to https://src.opensuse.org/contributor/descent3.
    • See the build to its completion
  9. Open a pull request via browser by navigating to https://src.opensuse.org/contributor/descent3 once more and using the Pull Requests tab to then select "New Pull Request".

The order is not particularly important; one might as well create the fork before the first build:

FCEP: Fork–Checkout–Edit–Publish (Title for a method 2)

  1. Fork at https://src.opensuse.org/jengelh/descent3
  2. Vivify a dummy project to use, e.g. osc meta prj -e home:contributor:test
  3. Vivify a package entity to use, e.g. osc meta pkg -e home:contributor:test/descent3; add a <scmsync> line
  4. osc co home:contributor:test/descent3 && cd home/contributor/test/descent3
  5. Examine git remote -v:
    • Sufficiently new versions of the osc command-line utility (seems like version 1.10 and above) will rewrite the remote URL from what is in the scmsync line https://src.opensuse.org/contributor/descent3 to gitea@src.opensuse.org:contributor/descent3, which is good.
    • On older osc, you need to perform this step manually, e.g. via git remote rm origin; git remote add origin gitea@src.opensuse.org:contributor/descent3.
  6. edit descent3.spec
  7. git add / git commit / git push my
  8. Open a pull request via browser

Notes for maintainers

src.opensuse.org uses git-lfs. This has some implications for pull request analysis. On the command line, wishing to examine repository-foreign branches needs some extra steps. Consider this workflow: 

$ osc co devel:tools/grpc && cd devel/tools/grpc/
$ git ls-remote origin
ed54753dfc5efcea1f4f678be907436e7b76449d377b1060a469da9d2ea7c084        refs/pull/1/head
$ git fetch origin refs/pull/1/head
* branch            refs/pull/1/head -> FETCH_HEAD
$ git checkout FETCH_HEAD
Downloading v1.68.2.tar.gz (17 MB)
Error downloading object: v1.68.2.tar.gz (afbc5d7): Smudge error: Error downloading v1.68.2.tar.gz
(afbc5d78d6ba6d509cc6e264de0d49dcd7304db435cbf2d630385bacf49e066c): missing protocol: "No such OID\n"

The act of making a merge request on collaborative forge software (such as GitHub, Gitea, etc.) copies the source-side commit into the target repository. This includes internal data structures like commits, trees and blobs. However, LFS objects seem to be exempt, thus the smudge error report. Anecdotally, this feels unusual, because if and when a merge request is accepted via the webinterface, Gitea ultimately does copy LFS objects. Anyway, to workaround the missing LFS objects prior to merge, one needs to explicitly retrieve LFS objects from the sending repository like so: 

$ git remote add r2 https://src.opensuse.org/badshah400/grpc
$ git lfs fetch r2 ed54753dfc5efcea1f4f678be907436e7b76449d377b1060a469da9d2ea7c084
$ git checkout FETCH_HEAD
HEAD is now at ed54753 Update to version 1.68.2.

Warnings about the scm-staging user guide

This section contains a diagnosis on contentious points of a scmsync guide created elsewhere. games/descent3 is used as a placeholder package like before.

Unwritable repository

That document asks the maintainer to change the OBS package entity metadata (osc meta pkg -e games/descent3) and employ <scmsync>https://src.opensuse.org/pool/descent3#factory</scmsync>.

Doind thag is problematic, because the /pool/descent3 repository does not have any write permissions. The maintainer is unable to perform git pushes. The maintainer is also unable to accept any pull requests made towards /pool/descent3.

Abolishment of development projects

Opening a pull request against /pool/descent3 creates an OBS project entity by the name of e.g. devel:Factory:git-workflow:staging:contributor:descent3:1 and creates an OBSSCM submit-typed request (SR) from …descent3:1/descent3 to openSUSE:Factory/descent3. This is the “will automatically get forwarded” part mentioned in the scm-staging user guide.

The maintainer(s) of games/descent3 are not added to this SR in any way and cannot intervene in case an evil/innocent user makes a malicious/stupid pull request. In essence, develprjs/develpkgs are abolished.

factory-auto

When the factory-auto bot rejects a submission from src-o-o-forwarder with e.g. 

devel project scmsync setting is https://src.opensuse.org/jengelh/descent3#master.
Must be https://src.opensuse.org/pool/descent3 instead.

Editing the scmsync line is generally not the right solution. The problem is that you created the pull request for the wrong target (pool/descent3) instead of jengelh/descent3. The bug for factory-auto making an improper suggestion is tracked as scm-staging issue 55.

Retrieved from "https://en.opensuse.org/index.php?title=openSUSE:OBS_with_Git&oldid=192655"