openSUSE:Build Service Concept Windows Support

Open Build Service

End user - Packager - Development - Installations - Collaboration - Comparison

This page discusses a DRAFT CONCEPT for the openSUSE Build Service: Input is very welcome; discussions should take place on the Build Service mailinglist or on the mingw specific list.

buildservice@lists.opensuse.org - .
Subscribe - Unsubscribe - Help - Archives
mingw@lists.opensuse.org - .
Subscribe - Unsubscribe - Help - Archives

Windows support

If Windows support were to be offered by the OpenSUSE Build Service, then the goal would be to produce easily-installable packages of free/open source software for end-users of the Windows platform.

In general, in the context of OBS, the goal is cross-platform porting of Linux-based software across to Windows, rather than the provision of a big fat build farm for generic Windows-only projects from the broader Windows community :-)

Building for Windows would bring some important advantages such as

more platforms to build on, thus higher attraction of free software projects
help spreading free software on Windows platforms
a potential much more larger community of developers working on free software in general and on the openSUSE projects infrastructure.
the power of OBS for building large complex interdependent sets of packages and programs
some kind of dependency-handling mechanism that could be exposed to end-user when installing new packages.

TODO: It needs to be considered how the licensing question for operating systems, compilers etc. is to be solved for the Build Service. If build hosts are volunteered by the community, then perhaps this is not a problem.

For the toolchain there are (at least cost-) free solutions listed on this page. Still, even if they are freely downloadable, licensing terms have to be checked nicely.

Building for Windows

To integrate building for Windows seamlessly into the Build Service, the following scenarios should be considered:

1. Cross-compiling

Current binutils and GCC support building for PE targets.

In OBS, we have complete builds of GTK+, including GIMP, Evince, Evolution, KDE applications like Umbrello and KMyMoney and lots more. The produced binaries work well on Windows, but are currently packaged as Linux RPMs.

Building ZIP/MSI/NSIS packages/installers is possible within limits. Non-RPMs may not yet propagate to /srv/obs/repos, depending on the BuildService software version running.

2. Virtualised builds

Another chance to produce native Windows bins is to compile in a virtualized environment running a windows host. In the Windows virtualization must run a native Windows compiler executable, some sort of base libs and a build script that controls the building and covers the communication with the Build Service.

Virtualisation of Windows doesn't seem to be a problem. It works well for me using KVM and it is also understood to be fine with Xen too (although I haven't tried myself). I understand that this is an important part of being able to set up a reliable 'clean' system for building each new package.

3. Native building

This scenario has real non virtual windows boxes. Building is done in a sandbox on the windows installation. Compiler and base libs are installed in the system.

Build description

There would need to be some meta-info or build description related to each package for Windows. In the same way that you have debian.rules, debian.changelog and debian.control, you might have windows.build and windows.package:

The first file, windows.build would be some kind of file containing information telling you how to compile and package a program. The Windows batch-file syntax could be used for these steps, or a Python file might be preferred (it could then be used cross-platform with OS X, potentially?). The data this file would contain would be:

how to unpack the sources,
how to initiate the build,
a list of the resulting 'target' files, and
a list of package build-time prerequisites and their required versions.

As with RPM .spec file and Debian 'control' file, this stuff would be (could be?) external to the sources.

The second file, windows.package would be meta-information related to actually installing and uninstalling the resulting package(s). These files might be produced as a result of the above build steps, or might be manually written separately. Data contained would include:

how it can be installed (in silent/unattended mode, specifically),
how it can be uninstalled, and
how it can be detected on the system
how it's version number can be identified
how it can checked (post-install checks) (needed?)
a list of package run-time prerequisites and their required versions.
where the package can be downloaded from (if not available in a standard 'repository')

We should be able to write a .package file for any already-released installer, so that we don't need to repackage things to suit the OBS system. If someone has written code for a nice installer for some particular FOSS program, we shouldn't have to repackage the program, we should just add the metadata that allows OBS to work out how to install/uninstall that package on a build machine.

Package manager

The Package Manager is an essential companion to the automated build system of OBS, because there needs to be a way for prerequisites to be quickly and efficiently installed before the main build proceeds. Also, this package manager could act as a 'consulstant' during user-initiated installs, if packages were set up to speak to such a package manager.

A difference of the Windows platform, compared to Linux, is that users are accustomed to seeking out 'installer files' directly, rather that using a package manager in the first instance. This mindset can be integrated with the package-manager mindset but creating special installers that 'talk' with the package manager before proceeding: requesting that all necessary pre-requisites be installed first.

A good package manager has the advantage that it allows developers to escape from the inefficient situation of having to bundle all prerequisites with their installer package, but there has to be a sensible common system for working this, and it has to be stable (e.g. in some cases, for example, multiple versions of same package must be able to coexist).

In the case of Windows, we need the current mechanisms to continue to work as expected. This includes the 'add/remove programs' tool, which can be made to work find if packages are modified so that they 'talk' to the package manager before proceeding to remove themselves -- 'pre-remove' and 'post-remove' hooks embeddded in the uninstaller. Making calls like this is pretty straightforward with tools like NSIS, still need to see how that works with MSI.

There are also mechanisms in place for 'signed' packages in Windows. Need to see if those mechanisms can be adopted within a Package Manager to assure authentic packages, even if user first has to install an additional CA certificate.

To satisfy our needs, the package manager would need a commandline interface or API for use by the power user and by the Build Manager, as well an API that could be accessed by package-manager-aware installers.

Anticipated commands for the package manager:

install package
install package-VERSION
remove package
register-dependencies package-VERSION deppackage1>=VERSION deppackage2==VERSION (allow a package to report its dependencies to the Package Manager)
depending-on package (are dependent packages installed?), return list of packages and versions.
cacheclean (remove local copies of installers)
detect (determine if a package is currently installed on the system)
detect-all (search for all recognisably installed packages)

Automated upgrade/update of packages could be be added as a later feature, as it's not needed as part of OBS integration.

Although the Package Manager could be implemented as a a windows service, this doesn't seem necessary. It's more important for the Build Manager to be written in that way (for external control); the Package Manager would be fine as a command-line program.

Detect

Steps requires in a detect action:

use the 'how it can be detected on the system' script for each registered package to determine if packages are still installed
- the 'how it can be detected on the system' script might be a simple Python script that could check the Windows registry, or check for specific installed programs/DLLs, etc. Or it could be possible to make it even simpler than that.
- what to do if previous version installed? keep fetching previous .package files from repo?
update registry for any vanished packages
if detected, return version number corresponding to value in .package file.

This action is used to detect vanished packages, because some uninstallers won't be package-manager-aware, so they won't notify when they are being removed.

It would be proposed that the detect action should be run before all other actions, at least in early-stage designs. For this reason, detection of packages should be minimal, eg does a certain registry key or file currently exist.

Detect-all

Steps required in detect action (optional):

download all .package files from repository
use the 'how it can be detected on the system' script for each one to determine if package has beed installed
- if installed, record .package details in registry.

This action is used to detect packages that have been installed from outside the control of the package manager. This is necessary because some installers won't be package-manager-aware.

Register-dependencies

Steps required in a register-dependencies action:

for each package in the list
- check that the package is present in the repository
  - if not, warning to user.
  - if so, check that version requirement can be satisfied
- obtain the .package file for the depended version
- record the .package details in the registry

This action is required because we want package-manager-aware installers to be able to specify their own dependencies. The dependency database will be built up locally and may include packages not available from repositories.

Install

Steps required in an install action:

obtain .package file from repository, or as embedded in package)
check that the actual package file can be located (in cache, or downloadable: in repository, on website)
check dependencies of package (using 'check' action), optionally ask user if they should be installed...
record package dependencies in Package Manager (windows registry?)
obtain the actual package file
run the installer
- optionally, the installer will call-back to the package manager to register its dependencies. they must agree with above.
move the installer to a cache, if automatically downloaded.

This action would be called either from the command line, or via the API exposed to the Build Manager, or during installation of other packages (satisfying dependencies).

Convenient command-line use of the install action would rely on the concept of a repository. The repository would need to contain

.package files/data
copies of binary installer files for those packages being hosted locally (not necessary all packages)

Depending-upon

Steps requires in a depending-on action:

determine if package is installed, work out version number (using 'check' action)
use locally recorded .package details to determine whether there are dependent packages installed
- recurse?
optionally, use repository to check for other packages that might have been installed 'outside the package manager', but not yet detected?
return amassed list of dependent packages

This action would be exposed to package-manager-aware uninstallers, so that they could warn the user of potential problems.

Remove

Steps required in a remove action:

use recorded .package details on local system to determine whether there are dependent packages installed.
ask for confirmation if deps are to be removed
precedence order the dependency list, work backwards through it (call 'remove' action, with no dependency checks)
run the 'how it can be uninstalled' script as per .package details.

This action would be initiated from the command-line, but would also be called recursively during removal of dependencies.

Cacheclean

Steps required in cacheclean action (optional):

remove all cache installer files

This action would be initiated from the command-line only (initially).

Build manager

The Build Manager would make calls to the Package Manager in order to satisfy the build-time dependencies. It would interface with OBS in a similar way to the current Linux scripts.

The build manager would work on top of the Package Manager. Using the .build file, it would

download the source code, if required (could that be done using the package manager, possibly?)
talk to the Package Manager in order to obtain all necessary prerequisites. Need ability for this to work non-interactively.
unpack the source code (using instructions from the .build file)
apply any necessary patches (perhaps this could be part of the previous step)
initiate the build, and watch/listen for errors (using instructions from the .build file)
check that all expected target files were created, and
send the target files to the master server, if requested.

The Build Manager would be implemented as a command-line program.

For integration with OBS, a Build Slave would be need. An approach like that used in BuildBot would be a good idea: a service that starts up on boot, and attempts to create a connection with a central server. The central server would be configurable at the time of setting up the slave. The Build Slave would await commands from the Build Master.

This architecture would be suitable for 'native' build nodes as well as for virtualised nodes. The Build Slave could possibly even run via Wine on a Linux system.

TODO We need to work out to what extent this overlaps the functionality currently provided by the osc commandline tool.

Initial 'seed' packages

Although we'd hope for all packages to be buildable using the Build Manager, there will have to be some initial 'seed' packages. These will be created as binary-only packages by simply creating a .package file and adding it to the repository. All the necessary meta-data would be held outside the binary installer; these binary installers would not be package-manager-aware; successful installation of such packages would be manually detected.

Examples of these packages would include MinGW as a minimum, and possibly others such as OpenGL libraries, etc.

Some of these 'binary-only' packages could be 'virtual' packages, for detecting components of Windows that may or may not be present, eg MSSQL, MS Visual Studio, etc. In such cases, the installer would be set to always fail (the Package Manager being unable to install such packages itself), but the detection of already-installed components would still work correctly.

One might consider whether the Package Manager itself would be considered as a package. Removing the package manager should remove all .package data from the local system, because this could be re-created later using the detect-all action. The user could be given the choice of whether or not to remove all the assocated packages. If that were possible, the hooks from package-manager-aware installers would have to be fail-safe for case where Package Manager was absent (perhaps 'do you want to reinstall the package manager?')

Interop with RPM?

OBS is currently being used to build RPMs for MinGW using cross-compilation on Linux. Could it be possible to include support for installing such RPMs using the above Package Manager, perhaps with a reduced feature-set?

Development plan

Development of the above system could be done in stages.

Constructing initial 'seed' packages for MinGW and related tools.
Constructing a repository for these seed packages
Writing the 'detect' action.
Writing the 'install' action, without automated dependency installation.
Writing the 'remove' action, without automated dependency removal.
Adding support for automated dependency handling.
Adding support for 'register-dependencies' action.
Adding support for 'depended-upon' action.
Creating a 'package-manager-aware' package that talks to the package manager during installation.
Creating the Build Manager.