Rust

Jump to: navigation, search


How to package the Rust compiler in the openSUSE Build Service.

Packaging the Rust compiler

The following discussion applies to Rust 1.30 and later. For older versions, see the section "Rust 1.26 and earlier" below. For reasons for the missing versions in between, see that section as well.

Rust 1.30 and later

The Rust compiler needs a Rust compiler, i.e. it needs bootstrapping. Once a package is built for Rust version N, then that can be used to compile version N+1.

  1. Building with bootstrapping: The rust package contains prebuilt binaries from upstream, for various architectures. That is, it has binaries for rust-i686, rust-x86_64, rust-powerpc, etc. If the OBS project in which the Rust package lives is configured to use bootstrapping, these binaries will be used to build the compiler.
  2. Building without bootstrapping: This is done once the bootstrap build is done the first time, or in normal situations when the build service already has Rust version N and is being used to build Rust version N+1.

Terminology

  • rustc - the Rust compiler.
  • cargo - the Rust build tool and package manager.

The bootstrapping process

First we need binaries for rustc/cargo that have been built in the primordial soup upstream. We will use those to build openSUSE's own version, from fresh sources. The rust package with its rust.spec contains these prebuilt binaries, as well as the rustc/cargo sources (and Rust's standard library, documentation, etc.). This package lives in the devel:languages:rust project.

During bootstrapping (see the next section), the specfile unpacks the prebuilt rustc/cargo and uses those to build the "new" compiler. After bootstrapping, the "new" compiler can be used to rebuild itself, or to build the next version of Rust without the bootstrapping process.

Creating a project for the first compilation

For the purposes of this discussion, we will create a new project called initial and copy the rust package to it. Get that package from devel:languages:rust.

Next, set up the following prjconf for your initial project. You can use the "Advanced / Project config" tab at the top of your initial project's page in the build service.

Prefer: rust
Keep: rust

# Define the architectures in which Rust (and Rust crates) are available
# NOTE: Keep this in sync with rust-srpm-macros!
%define rust_arches x86_64 i586 i686 armv6hl armv7hl aarch64 ppc64 powerpc64 ppc64le powerpc64le s390x

# Uncomment to enable bootstrapping rust and its sub packages
Macros:
%rust_arches x86_64 i586 i686 armv6hl armv7hl aarch64 ppc64 powerpc64 ppc64le powerpc64le s390x
# Uncomment to enable bootstrapping, comment to disable
%_with_rust_bootstrap --with-rust_bootstrap
:Macros

This prjconf does a few things:

  • "Prefer: rust" indicates that the build service should prefer the "rust" package when it has a choice among several packages that provide the same capability.
  • "Keep: rust" is used for packages that require themselves to build.
  • Defines the %_with_rust_bootstrap macro. We will activate this the very first time we compile the package, and comment it out afterwards.

You can see the final prjconf from devel:languages:rust. Keep in mind that that one has the %_with_rust_bootstrap macro commented out! You need it to be defined (without the "#" at the beginning of the line) for the very first build.

At this point, the build service will churn for some time and compile rust. This takes time, around three hours. You may go watch a movie or something.

What just happened?

The build service took the prebuilt bootstrapping binaries, and built the Rust sources into rust/cargo/etc. RPMS with them. We now have usable RPMS for those, but we would like to build the next versions of Rust/Cargo from the binaries we built ourselves, not with bootstrapping packages.

Building non-bootstrapped (normal) packages

Create a project called second, and copy your rust package to it.

At this point, that package cannot be built because it requires a previously-built version of itself. We will use the aggregatepac trick to do this.

Do this:

osc aggregatepac initial rust second

This tells the build service to make the binaries from the rust package from the initial project, to be copied into the second project. The immediate effect of these commands is that a file called _aggregate is placed in your second/rust project.

At this point you must wait a few minutes for the build service to copy the binaries. Watch the build status of the second project. When it succeeds, you must then delete the _aggregate files from the second/rust package.

At this point, the build service will start churning again, and will use the binaries you copied from initial to build second.

This takes a long time, around three hours. You may go bake a cake or something.

Updating to the next version of Rust/Cargo

Make sure your destination project has the "Keep" line in its prjconf as described above.

You can put new versions of the tarballs for rust in its corresponding package. It should get rebuilt using the binaries from the previous version, as per the "Keep" line in the project configuration.

Note that you need Rust version N to build Rust N+1. This means that you need to update Rust sequentially; if you miss a few versions between updates, you must build the chain of versions one by one.

Summary of the bootstrapping steps

  1. Create a new project
  2. Copy the prjconf described above to have the Prefer/Keep lines, and uncomment the bootstrapping macro.
  3. Copy the rust package from devel:languages:rust into the project; wait for it to build.
  4. Alternative 1: Create a second project
  5. Copy the prjconf; disable the bootstrapping macros
  6. Copy the rust package into the second project.
  7. Aggregatepac.
  8. Wait for the aggregated binaries to be copied.
  9. Delete _aggregate from the rust package.
  10. Wait for rust to build (slow).
  11. Alternative 2: Comment out the bootstrapping macro.
  12. Wait for the rust package to rebuild (slow).
  13. In either case, your packages are done. You can update to new tarball versions as they come out.


Rust 1.26 and earlier

The following refers to Rust 1.26.x and earlier, which used to be packaged differently than 1.30.x and later.

Why did this change? Because older Rusts used to be distributed in separate source tarballs for Rust and Cargo, whereas in new ones they are distributed in a single rust.tar.xz. This means that the bootstrapping is different.

Why is there a gap between 1.26.x and 1.30.0? No one packaged those intermediate versions!

The Rust compiler for versions 1.26.x and earlier comes in two SRPMS, one for rust and one for cargo. However, both Rust and Cargo need each other to compile themselves! The following section describes how bootstrapping is done.

The bootstrapping process

Rust's bootstrapping process needs three things:

  • rustc - the Rust compiler itself
  • rust-std - the Rust language's standard library
  • cargo - the build tool for Rust

The bootstrapping packages

First we need the binaries for rust, rust-std, and cargo that have been built in the primordial soup upstream. We will use those to build openSUSE's own version, from fresh sources. We create rustc-bootstrap, rust-std-bootstrap, and cargo-bootstrap packages out of these binaries from upstream. These -bootstrap packages live in the devel:languages:rust:previous-versions project. Each package contains precompiled binaries for a bunch of architectures, and a specfile that just puts the precompiled binary in the correct place in the filesystem.

Creating a project for the first compilation

For the purposes of this discussion, we will create a new project called initial and copy the rustc-bootstrap, rust-std-bootstrap, and cargo-bootstrap packages to it.

At this point, the build service will create RPMS out of rustc-bootstrap, rust-std-bootstrap, and cargo-bootstrap.

Next, set up the following prjconf for your initial project. You can use the "Advanced / Project config" tab at the top of your initial project's page in the build service.

Prefer: rust
Keep: rust

# Uncomment to enable bootstrapping both rust and cargo.
# Rust will then compile with rust-bootstrap and cargo-bootstrap.
# Cargo will compile with the built rust, and cargo-bootstrap.
Macros:
%_with_rust_bootstrap --with-rust_bootstrap
%_with_cargo_bootstrap --with-cargo_bootstrap
:Macros

This prjconf does a few things:

  • "Prefer: rust" indicates that the build service should prefer the "rust" package when it has a choice among several packages that provide the same capability.
  • "Keep: rust" is used for packages that require themselves to build.
  • Defines %_with_rust_bootstrap and %_with_cargo_bootstrap macros. We will activate these the very first time we compile the packages, and comment them out afterwards.

You can see the final prjconf from devel:languages:rust. Keep in mind that that one has the macros commented out! You need them to be defined (without comments) for the very first build.

Building the first packages

Copy the the rust and cargo packages from devel:languages:rust into your initial project.

At this point, the build service will churn for some time and compile rust first, and cargo second. The rust package will provide rust-std as well as a result of its compilation. This takes time, around two or three hours. You may go watch a movie or something.

What just happened?

The build service took the prebuilt bootstrapping packages, and made RPMS from them. Then, it used the prebuilt compiler, standard library, and cargo from those RPMS to build the Rust compiler and Cargo from real sources. We now have usable RPMS for those, but we would like to build the next versions of Rust/Cargo from the binaries we built ourselves, not with bootstrapping packages.

Building non-bootstrapped (normal) packages

Create a project called second, and copy your rust and cargo packages to it.

At this point, those packages cannot be built because they require previously-built versions of themselves. We will use the aggregatepac trick to do this.

Do this:

osc aggregatepac initial rust second
osc aggregatepac initial cargo second

This tells the build service to make the binaries from the rust and cargo packages from the initial project, to be copied respectively into the rust and cargo packages in the second project. The immediate effect of these commands is that a file called _aggregate is placed in each of your destination projects.

At this point you must wait a few minutes for the build service to copy the binaries. Watch the build status of the second project. When it succeeds, you must then delete the _aggregate files from both rust and cargo in the second project.

At this point, the build service will start churning again:

  • First, it will build the rust package in the second project, using the binaries from the initial project.
  • Then, it will build the cargo package in the second project, using the rust binary from second and the cargo binary from initial.

This takes a long time, around two or three hours. You may go bake a cake or something.

Updating to the next version of Rust/Cargo

Make sure your destination project has the "Keep" line in its prjconf as described above.

You can put new versions of the tarballs for rust/cargo in their respective packages. They should get rebuilt using the binaries from the previous versions, as per the "Keep" line in the project configuration.

Note that you need Rust version N to build Rust N+1. This means that you need to update Rust and Cargo sequentially; if you miss a few versions between updates, you must build the chain of versions one by one.

Summary of the bootstrapping steps

  1. Create a new project
  2. Copy the prjconf described above to have the Prefer/Keep lines, and uncomment the bootstrapping macros.
  3. Copy the rustc-bootstrap, rust-std-bootstrap, cargo-bootstrap packages into the project; wait for them to build (more or less fast).
  4. Copy the rust and cargo packages. Wait for them to build (slow).
  5. Create a second project
  6. Copy the prjconf; disable the bootstrapping macros
  7. Copy the rust and cargo packages into the second project.
  8. Aggregatepac.
  9. Wait for the aggregated binaries to be copied.
  10. Delete _aggregate from the rust and cargo packages.
  11. Wait for rust and cargo to build (slow).
  12. Your packages are done. You can update to new tarball versions as they come out.