openSUSE:Reproducible openSUSE/Part1

Jump to: navigation, search

This is a report of Part 1 of my journey: building 100% bit-reproducible packages for every package that makes up our minimalVM image.

This target was chosen as the smallest useful result/artifact. The larger package-sets get, the more disk-space and build-power is required to build/verify all of them.

Design

The many sources are kept in https://build.opensuse.org/project/show/home:bmwiedemann:reproducible:distribution:ring0 . As usual, we use the original upstream tarball and apply patches as needed in the .spec file (build description).

It also keeps the scripts in the 000pbuildconf pseudo-package. With a checkout of these, you can verify the source checksums and start building locally using pbuild from the build package (obs-build on Debian). This results in 81 GB of binaries for ring0 and 448 GB of binaries for ring1. And calculating a hash over all these binaries produces a constant result in builds on different machines on different days.

What I did and How to check

https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/2XMBAI6G4ZFAS7LFWM2XPYHZYMG5C2GF/ already documents some details of this.

However, after that I found a few more issues: perl embeds the build machine kernel version. This showed up twice. Once when I ran pbuild without --kvm so that the build host kernel version went in there. And the second time when I switched the base snapshot in the _pbuild config and that came with a different kernel. pbuild did not trigger a re-build of perl, likely because the kernel-obs-build package counts as "Support" and not "BuildRequires". I resolved this by explicitly adding kernel-source into ring0. We need a kernel for minimalVM anyway.

Then, the next issue that showed up was that kernel-docs was unreproducible. Some experiments led to a workaround of running sphinx without -jauto . There is some discussion in https://lore.kernel.org/linux-doc/33018311-0bdf-4258-b0c0-428a548c710d@suse.de/T/#t and some renewed activity in the related Sphinx issue tracker after 5 years.

Another issue was noticed with kernel-vanilla. I had analyzed this earlier and found that I had to add the _projectcert.crt from OBS so that the build process would not create + embed a random pubkey.

Some additional notes: the ML post suggested to use 'rbk' for test-builds. However, this will only build the main package. If there is a _multibuild file involved, 'multibuildrbkall' will loop through all of them and stop at the first unreproducible result. The 'nachbau' script can only verify the main package so far.

However, the easiest is to use pbuild via

osc co home:bmwiedemann:reproducible:distribution:ring0 && cd $_
# verify source hashes
sh 000pbuildconf/sha256sums.src
#=> 9e0dd066b2c741290688b017bf069a0980df9aa8e0dae5b48addd53ac9a4a400  000pbuildconf/sha256sums.src.out
ln -sf 000pbuildconf/_* .
sh 000pbuildconf/build.sh
# if more than 2 failed builds occur (e.g. see util-linux below), retry it with:
sh 000pbuildconf/build.sh --rebuild failed

It needs around 150GB in /var/tmp and ~80 GB for the ring0 binaries. And after a day it should compute the hash of all binary hashes as

25714238c6bdf4518a671fcd5d61e3decd255794dd9420b8a80db60c30fd28cf  .sha256sums

If you get a different hash, Bernhard is very interested in getting a diff with http://rb.zq1.de/rbos/ring0-sha256sums (send mail to bwiedemann+rbos at suse de)


Merged fixes:

Pending/WIP fixes + open discussions: