openSUSE:Packaging Python

Jump to: navigation, search



The Packaging Python is a step-by-step introduction on how to build Python software packages for openSUSE and others using the openSUSE Build Service.

The fast and automated way via py2pack and osc

Extracting the package name

Let's suppose you want to package zope interface and you do not know how it is named exactly or where to download from.

First of all, search for the exact name on PyPI. The name of a package is the in the URL and in the pip install instruction at the top of the page (e.g. pip install zope.interface).

Now download the source tarball automatically with py2pack:

$ py2pack fetch zope.interface
 downloading package zope.interface-5.4.0...
 from https://files.pythonhosted.org/packages/source/z/zope.interface/zope.interface-5.4.0.tar.gz

Generate the spec file

As a next step, you may want to generate a package recipe for your distribution. For RPM-based distributions, you want to generate a spec file named python-zopeinterface.spec. The metadata for the spec file is read from the source archive you have just downloaded:

$ py2pack generate zope.interface -f python-zope.interface.spec

The source tarball and package recipe is all you need to generate the RPM file. This final step may depend on which distribution you use.

Complete recipe

Again, for openSUSE (and by using the openSUSE Build Service), the complete recipe becomes:

$ osc mkpac python-zope.interface
$ cd python-zope.interface
$ py2pack fetch zope.interface
$ py2pack generate zope.interface -f python-zope.interface.spec
$ vi *.spec
BuildRequires: python-setuptools
ZZ
$ osc build
$ osc vc
$ osc add *
$ osc commit

The first line uses osc, the Build Service command line tool, to generate a new package (preferably in your Build Service home project).

It is always a good idea to review all the dependencies specified in the spec file accordingly to the upstream manifest and requirements, e.g. dropping the %{python_module devel} dependency in a case where the package does not compile anything, or adding %{python_module setuptools} if needed.

Sometimes, the test requirements are specified in upstream's setup.py in an extra [test] section or similar, which are converted to Recommends: python-foo tags by py2pack. You need to change them to BuildRequires: %{python_module foo} to be able to actually run the tests at build time.

Most of the time, we do not need static linters or code coverage tools despite being specified upstream, such as mypy, coverage, pytest-cov, flake8, or black. Just omit installing and running those.

Finally, the package is tested (built locally), a .changes (package changelog) file is generated (with osc vc) and the result is sent back to the Build Service for public consumption.

Depending on the Python module, you may have to adapt the generated spec file slightly. Py2pack is quite clever and tries to auto-generate as much as it can, but it depends on the metadata that the module provides. Thus, bad metadata implies mediocre results.

To get further help about py2pack usage, issue the following command:

$ py2pack help

Often, the first run of osc build will fail with build error messages, due to missing python modules. Look for the keyword "import"; this should give you a hint what needs to be added to the list of BuildRequires (and possibly Requires, if the import also happens in the installed package) of your spec file.


Hints on how to package Python modules manually

Manual packaging from scratch is discouraged since we have tools to do that (see above). If you insist on having fun, please consider any package spec file in devel:languages:python. python-autobahn, python-black or python-Twisted are a few examples.

In general, we are using single-spec packaging for any Distribution package which provides modules importable by other packages.

What is single-spec?

"Single-spec" is an approach for packaging multiple variants of a Python module from a single source spec file. The variants are based on the available Python "flavors". These used to be python2 and python3, but have been extended to multiple variants of Python 3. Tumbleweed by default builds for the flavors python39, python310, python311', but not python2 or python36 anymore. SLE15 Leap 15.X have python2 and python3 as their default build set. One of the python3X flavors, currently python38 is also at the same time the primary python3 flavor.

The set of available flavors is defined by python-rpm-macros. You need to define

BuildRequires:  python-rpm-macros

Compatibility shim

If your packages are targeted for anything older than Leap 15, the spec file must include a redefinition of %python_module macro.

%{?!python_module:%define python_module() python-%{**} python3-%{**}}

Build set and naming policy

SUSE has a policy for names of Python module packages. A module is to Python what shared libraries are to C 鈥 a piece of code that does not work by itself, but provides functionality to other Python programs.

All Python module packages, whether pure Python or C-based, should be called python-modulename. modulename should be the name of this module on the Python Package Index.

Previously, Python packages have been named after directories in the site-packages hierarchy. Often, this was an arbitrary choice, as several modules can install more than one directory there. The Python universe also includes modules that share the same directory name (search for "daemon" on PyPI) and it was not always easy to find out the right package name. Furthermore, the new approach has several advantages:

This policy does not apply to end-user applications that are not designed and planned to be used as libraries 鈥 so if you are packaging something that is going to have an icon in the application menu, you should just call the package by its normal name (as found on the Python Package Index), and do not generate subpackages for multiple flavors.

There are some corner cases as to what is an application and what is a module 鈥 for example, many modules come with simple command-line tools that allow you to use a subset of their functionality directly.

The rule of thumb is this:

  • If this package is going to be a dependency of some other Python application, use single-spec and apply the naming policy,
  • otherwise keep it just as the module name and specify all dependencies to come from the primary python flavor (python3).

Similarly, this policy does not apply to Jupyter kernels, extensions, and similar packages that are not designed and planned to be used as libraries. If they are meant to be used exclusively within the confines of a Jupyter environment, they should be called jupyter-modulename instead of python-modulename, where modulename is still the the name of this module on the Python Package Index. Such packages must also provide the python3-modulename package if they contain Python code. Conversely, packages that are primarily Python packages but provide Jupyter interfaces should either have the python3 flavor provide jupyter-modulename, or split out the components in the Jupyter data directories into their own jupyter-modulename subpackage.

Source URLs

For packages that are available from PyPI, the correct Source URL is the following: <code>https://files.pythonhosted.org/packages/source/t/tox/tox-%{version}.tar.gz</code>

The preferred hostname is files.pythonhosted.org. All other hostnames (like pypi.io, pypi.org, pypi.python.org) eventually redirect to it. In order to ease this transition, you can put in the link mentioned below and it will be automatically converted to the proper format by spec-cleaner tool.

The link you get from PyPI (by going to the download section of a package) will point to some sort of hash, e.g.

https://files.pythonhosted.org/packages/99/20/2fd208e75c05975bf6436258f469aa233045b93d195cdb442045de0923cc/tox-2.7.0.tar.gz

We do not want that. The "semantic URL" is still available:

/packages/source/<first letter>/<full name>/<full name>-<version>.<extension>

In some cases, packages are only distributed through PyPI as wheels. These are files with a .whl extension. In these cases, either the wheels can be packaged manually, or another upstream source can be used if available (such as from github, see below).

Icon-warning.png
Warning: Do not package pre-built binaries! See the packaging guidelines for more info.

The path to the wheel file is of the form:

/packages/<python version>/<first letter>/<full name>/<full name>-<version>-<python version>-<abi>-<platform>.whl
  • <python version> is the specific version of Python the wheel supports, such as py3 for python3-only and py2.py3 for both. Note that this appears twice in the URL, and they must match.
  • <abi> is the architecture-specific ABI. Only wheels with the none abi can be used, since others contain pre-compiled libraries.
  • <platform> is the particular operating system supported. This must be any, since, again, anything else contains pre-compiled libraries.

Some example of full source URLs for wheels are:

https://files.pythonhosted.org/packages/py2.py3/b/bqplot/bqplot-%{version}-py2.py3-none-any.whl
 https://files.pythonhosted.org/packages/py3/i/imatlab/imatlab-%{version}-py3-none-any.whl

For some packages, particularly Jupyter kernels or extensions, wheels contain important files which the source distribution is missing. In these cases, the use of wheels is preferred. You can check for such files by opening the wheel file (it is just a ZIP file; if your archive reader cannot open it, change the extension) and look for files in the etc or usr directory.

However, in some situations, important files are missing from the PyPI archive or wheel. For example, they may be missing tests or license files. Such issues should be reported upstream. If it is a single file, like a license file for licenses that require one, and upstream has a source such as GitHub that includes that file, it is possible to include it and set the source URL to be the "raw" version of the file while waiting for upstream to provide a proper fix. For example,

https://raw.githubusercontent.com/imatlab/imatlab/v%{version}/LICENSE.txt

However, for packages where many files or whole directories are missing, particularly tests, the upstream source should be used. For github, that source is of the form:

https://github.com/numpy/numpy/archive/v1.16.2.tar.gz#/numpy-1.16.2.tar.gz

The #/numpy-1.16.2.tar.gz part at the end is important. If that is left off, the openSUSE:Factory source validator will download a file named v1.16.2.tar.gz instead of numpy-1.16.2.tar.gz which will cause the source validation to fail.

BuildRequires

BuildRequires are not conditional and apply to the whole spec file. There is no such thing as 鈥渋f I build for Python 2, I require package foo鈥; you simply require all packages for all variants.

To automatically require a Python package for all flavors of Python within the build set, use the %python_module macro:

BuildRequires: %{python_module setuptools}
BuildRequires: %{python_module py >= 1.4}

Note that the version requirement goes inside the %python_module call.

This is not intended to be used for the Requires tag. The %python_module macro must only be used for BuildRequires, as it pulls in all flavors at once. Requires: should only specify packages for the particular flavor and is rewritten by %python_subpackages.

If you only need the package for one flavor of Python, simply do not use the macro and state the dependencies directly.

BuildRequires: python2-enum34
BuildRequires: python3-astroid

openSUSE Tumbleweed has removed the majority of Python 2.x packages and does not include python2 in the buildset anymore. It defines the equivalent of osc build --without python2 by default. So, if you still need to support building python2 packages for older distributions, you can exclude the requirements for Tumbleweed by

%bcond_without python2
%if %{with python2}
BuildRequires: python2-enum34
%endif

If you need to BuildRequire python (the interpreter binary), you can use %pythons:

BuildRequires: %{pythons}

If you need a package under certain conditions, use RPM boolean dependencies. You can reference the Python version of the expanded flavor itself by using the pseudo %python macro (you must not define %python anywhere else):

BuildRequires: %{python_module aiocontextvars >= 0.2.2 if %python-base < 3.7}
Icon-warning.png
Warning: %pythons and %python_module do not only require a recent python-rpm-macros package, but also need the corresponding definition in the OBS project configuration, either directly or by inheritance.
  • %pythons is currently defined for openSUSE Leap 15.0 and newer.
  • The boolean dependency support for %python_module is currently defined in openSUSE:Factory, devel:languages:python:backports and devel:languages:python for 15.4+, but not in plain SLE/Leap projects. Thus, the syntax is generally only available for Tumbleweed-only builds

Automatic Runtime Requirements Determination

Some other Linux distributions (namely, Fedora) prefer to use automatic dependencies generators and using constructs like:

BuildRequires: python-rpm-generators
%{?python_enable_dependency_generator}

Please, do not use such constructs in packages which should be included in Factory!

Unfortunately, dependency generators require perfectly maintained upstream metadata (which is by far not the case), and even then blind reliance on those generators leads to hundreds of mutually exclusive situations (especially, when < version constructs are used), which would lead to the maintenance nightmare we want to avoid.

Moreover, even if this worked, any build time and test dependencies would still have to be specified manually with BuildRequires.

The package python-rpm-generators offers some more useful macros, see project page on Github.

Requires, Provides and similar

In many cases, you do not need to do anything. The single-spec rewriter will convert your Requires to match the generated package. Only make sure the Requires content really matches setup.py or requirements.txt and there is nothing missing.

Icon-warning.png
Warning: In particular, do not use %python_module for Requires.

Python package names in Requires, Requires(pre) and all other scriptlet runtime requirements, Provides, Recommends, Suggests, Obsoletes, Conflicts, Supplements and Enhances are automatically converted to the correct flavor in the subpackage declarations autogenerated by %python_subpackages.

If the requirement or capability name starts with python-, with the default python flavor (python311- in Tumbleweed, python3- in older distributions), or with the same python prefix as your package name (so python3-bar in package python3-foo), then the python name is changed to match that of the generated package. This also works for python itself.

If you want to prevent rewriting of the name, mask it behind a macro, e.g. with the use of %oldpython as shown below.

The converter takes into account the packageand expression, and %requires_ge and %requires_eq macros. However, support for these must be coded explicitly. If you find an expression that is not converted, please either post it on the opensuse-packaging mailing list, or file a bug against python-rpm-macros. As of python-rpm-macros version 20210628.eccf3f2, RPM boolean requirements are supported as well.

Requirements specific for one python flavor

You can specify that some packages should only be included for some Python flavors, by wrapping them in conditionals using the %python_flavor variable:

Requires: python-idna
%if "%{python_flavor}" == "python2"
Requires: python2-enum34
%endif

As a shortcut, for every flavor, there is a %ifpython macro: %ifpython2, %ifpython3 or %ifpypy3.

%ifpython2
Requires: python2-enum34
%endif

Note that the shortcuts must not be nested in other conditionals, otherwise you can get %endif without %if error messages. If you need to nest conditionals, use the %python_flavor conditional.

Icon-warning.png
Warning: This works by copying the %ifpython sections to the generated subpackage definitions, where they are evaluated for conditional application. Subpackages have their own runtime requirements, file lists and %pre/%post/etc. scriptlets. The scripts from the %prep, %build, %install, %check sections and global tags such as BuildRequires are not subpackage specific. Do not use %ifpython for these.

Testing for the python3 flavor through %ifpython3 or %if "%{python_flavor}" == "python3" is deprecated. These conditionals do not work for repositories with multple flavors of Python 3, i.e. they will not apply for any flavor named python38, python39, etc. If you have a section that should only apply for one of the generated Python 3 packages, use

%if "%{python_flavor}" == "python310"
Requires: python310-specialfuturepackage
%endif

If a tag is only applicable for the primary python3 flavor, directly in SLE/Leap and indirectly through e.g. python38 in TW, use:

%if "%{python_flavor}" == "python3" || "%{python_provides}" == "python3"
Provides: foo
%endif

See also Conditionals on Python versions.

Obsoleting and Providing old symbols

The following sequence:

Obsoletes: python-distribute < %{version}
Provides:  python-distribute = %{version}

would mean that your python2-package will obsolete/provide python2-distribute and your python3-package will obsolete/provide python3-distribute and your pypy3 package will obsolete/provide pypy3-distribute. Often, this is not what you want.

First of all, in many cases, this is only applicable for Python 2, so the sequence should be wrapped in a %ifpython2 conditional block.

Second, you will note that neither package obsoletes/provides python-distribute.
Here is how you do that:

%define oldpython python
%ifpython2
Obsoletes: %{oldpython}-distribute < %{version}
Provides:  %{oldpython}-distribute = %{version}
%endif

%python_module in Provides

If you are creating a subpackage that will be common for all flavors (could be a -doc subpackage), sometimes you need to provide a symbol for all flavors. For example, your package python-foo-doc should also provide python2-foo-doc, python3-foo-doc etc. In such cases, you use the %python_module macro:

%package -n python-foo-doc
Provides: %{python_module foo-doc = %{version}}

This is also the one case where you could use %python_module in Requires.

(See below on declaring packages with -n to prevent autogeneration.)


%python_subpackages

Easy enough: place the %python_subpackages macro on a separate line at the end of the spec preamble (the part that ends where your package's %description begins).

Provides: pylint
BuildArch: noarch

%python_subpackages

%description

This macro emits all the subpackage descriptions, %files and scriptlet sections for the autogenerated parts.


Subpackage declarations

Subpackages are converted automatically. If you have a subpackage %package foo, singlespec will create python3-yourpackage-foo and all the rest from it.

If you want to prevent this, use %package -n %{name}-foo, or use the full name, %package -n python-yourpackage-foo. This will ensure that the subpackage will be skipped.

This also means that packages named %package -n yourpackage-python will not be processed. There will be a mechanism for these in the future.

Common documentation packages

It is very common that the shipped documentation or examples are quite large and should be in a separate subpackage. The package could have these sections for example:

%package -n %{name}-doc
Summary:        Documentation files for %name
Group:          Documentation/Other

%description -n %{name}-doc
HTML Documentation and examples for %name.

%files -n %{name}-doc
%doc examples docs/_build/html/

Build macros

To build a package the modern PEP517/PEP518 way, use

%build
%pyproject_wheel

%install
%pyproject_install

These macros depend on pip, and may need an installer backend such as the setuptools/wheel pair, flit-core, poetry-core or hatchling. The backend is specified in the build-system.requires table of the pyproject.toml.

For packages not having a pyproject.toml file yet, do use the setuptools, wheel backend.

Legacy setup.py

For building the packages the old way with a setup.py-based build, use %python_build. In the %install section, use %python_install.

These macros already contain the usual options (--root, --prefix), so, in the typical case, you do not need to supply any options. If you have something specific, you can add it like so: %python_build --enable-specific-feature.

This set of macros is deprecated.

Generic

If you set environment variables, export them first:

export CFLAGS="-fwrapv"
%pyproject_wheel

For any other commands, you can use %python_exec and %python_expand. That is also a good way to run pythonic executables. For example:

%install
%python_exec setup.py extracommand
%python_expand ./make PYTHON=$python
%python_expand PYTHONPATH=%{buildroot}/%{$python_sitelib} my-alternative-cloned-entrypoint-%{$python_version}

%python_expand

For anything more complicated than executing all the interpreters, use the %python_expand macro. This will repeatedly expand the %{$python}, %{$python_sitelib} etc. strings to the currently used flavor.

%python_expand rm -r %{buildroot}/%{$python_sitelib}/file.txt

results in (apart from some build dir manipulations):

rm -r %{buildroot}/%{python2_sitelib}/file.txt
rm -r %{buildroot}/%{python3_sitelib}/file.txt
rm -r %{buildroot}/%{pypy3_sitelib}/file.txt

Important: You can use %python_expand to replace macro definitions, but make sure you use %{$python_sitelib}, i.e. with a $ sign. If you use plain %{python_sitelib}, the macro will be expanded before %python_expand can modify it.

A common use for %python_expand is with fdupes, as in:

%python_expand %fdupes %{buildroot}/%{$python_sitelib}/mymodule

You can use multiline %python_expand if you enclose the lines in {}. The only technical limitation is that the first line must not be empty (but can be a comment starting with #):

%{python_expand # this will expand the following section
        %$python_install
        mv %{buildroot}/%{_bindir}/exename %{buildroot}/%{_bindir}/exename-%{$python_bin_suffix}
}

This is useful when running tests and some prior step is needed:

%check
%{python_expand #
        rm -rf .testrepository
        $python -m unittest discover -v                                                                                                                                                                                                                                                               
}

Naming flavor-specific files

Executables specific to a particular flavor should use %python_bin_suffix instead of %python_version for names. This expands to %python_version for CPython, pp%{python_version} for PyPy. If we support Jython, it will get a specific bin_suffix too.

Build directories specific to a particular flavor should use %python_prefix. This expands to the flavor name.


Filelists

If your package is called python-something (that is, the name prefix is python and not a specific flavor), you must mark your %files sections with %{python_files} macro.

%files %{python_files}
%{python_sitelib}/foo
%{python_sitelib}/foo-%{version}*-info

%files %{python_files plugins}
%{python_sitelib}/foo-plugins

You can use %ifpython2 and similar macros to conditionally include some files only in some flavors. In addition, you can use shorthands:

%files %{python_files}
%{python_sitelib}/foo
%{python_sitelib}/foorun.py*
%pycache_only %{python_sitelib}/__pycache__

Use %pycache_only or %ifpycache to mark __pycache__ directories.


Executables

For multi-flavor packages which carry executables (Python setuptools calls them entry-points), you need to ensure that they do not create file conflicts. This can be achieved by using update-alternatives as described below. In that context, the %python_clone macro is needed for the 鈥渞ight鈥 Python version.

Some packages still rely on the %python3_only syntax which ensured the binary was carried over only to the specific version. This approach, albeit shorter, has a drawback of not working in multi-Python-interpreter environment and, as such, we deprecated its use.

update-alternatives

Sometimes, it is useful to let the user switch the unversioned executable name to one version of the package or another.

update-alternatives allows you to switch other things (usually man pages) along with executables.

As a prerequisite, you need to have version-specific file names available, that is, for every file you provide, file-%python_bin_suffix should exist for all flavors. (For manpage.1, the appropriate name is manpage-%python_bin_suffix.1)

First, you need to set up the alternatives in %install section.

  • If you are using %python_clone to create the executable, pass the -a option: %python_clone -a %{buildroot}/%{_bindir}/executable
  • If not, and the file in question is %{_bindir}/something, call %prepare_alternative something
  • If the file is not in %{_bindir}, you have to specify the path: %prepare_alternative -t /path/to/file file. The path is without %buildroot.

Second, create the appropriate %post and %postun sections. Make sure that you have the proper requirements:

Requires(post):   update-alternatives
Requires(postun):  update-alternatives

Then use %python_install_alternative and %python_uninstall_alternative, respectively.

%post
%python_install_alternative exename

%postun
%python_uninstall_alternative exename

Third, mention the alternative, unversioned, in the file list:

%python_alternative %{_bindir}/exename

You can examine a full spec file with executables and manpages at [1].

Grouped alternatives

The update-alternatives system allows for multiple files in the same group to be switched together. This is useful if you want to install an executable along with its manpage, or multiple executables belonging to the same function group.

To make use of this, specify the files as multiple arguments to %python_install_alternative:

%{python_install_alternative pylint pylint.1 epylint epylint.1}

The macro can recognize manpage names and handle them correctly, but the first arguments always needs to be an executable. Alternatively, you can specify a full path to the file in question.

The first argument to %python_install_alternative is the group name. This is the only argument for %python_uninstall_alternative; you uninstall the whole group by the one name.


Decision whether to build for single or multiple Python versions

Python2 only (Leap)

Non-singlespec packages that only exist for Python 2 should be left as-is. New packages should be called python2-foo. Old packages should be left as python-foo, but you should add Provides: python2-foo. You should also make sure that all BuildRequires and Requires are for python2 and python2-foo.

Python 3 (Leap and Tumbleweed)

Packages that only exist for a single Python 3 flavor (apps) can omit the single-spec system, or they can use single-spec with a single flavor in the build set. Modules should be single-spec with all possible flavors. See #Build set and naming policy for a distinction between apps and modules. The definitions should be made at the top of the specfile.

For apps:

  • %define pythons python3

For modules:

  • Use %define skip_python2 1 so that Leap does not try to build python2 flavors.
  • Skip all flavors which are not supported, e.g. %define skip_python310 1.
  • Make sure at least one flavor is left for building.

Python 3 (Leap Future)

There is a proposal to provide a recent version of Python packages and the Python interpreter available on Leap. These packages will be modern versions and, to avoid collision with the current versions, will use a macro to override the default pythons macro and build just for the modern version, Python 3.11 in this case.

Use this macro to make the modern python package build-able for Leap, just with the latest python interpreter (3.11) and not for the old one (3.6):

%{?sle15_python_module_pythons}

The macro is defined in the project setup in OBS.

To avoid source package name conflicts, packages that build for the primary python (3.6) should be renamed to be python3-foo and will continue providing the binaries with the python3- prefix.

These are the macros defined in the project setup in OBS:

%sle15_python_module_pythons() %global pythons python311
%sle15allpythons() %global pythons python311 python3

Conditionals on Python versions

If you need to differentiate between Python versions, you can use the %python_version_nodots macro, which expands to the Python major and minor version of the currently auto-generated flavor package, with the dot(s) removed.

For example:

%if %python_version_nodots < 39
Requires: python-importlib_resources
%endif

If you need to determine the version of a specific general provider, use the flavored variant of the macro, e.g.:

Macros Example
%python2_version_nodots 27
%python3_version_nodots 38

Common gotchas

Things to look out for (which can also get your submission declined):

  1. If %python_module is re-defined, check that it has python-%{**} and not python-%1
  2. use %python_module for BuildRequires
  3. do NOT use %python_module for Requires
  4. make sure %python_subpackages macro is present
  5. %files sections must be marked as %files %{python_files}
  6. in %python_expand, make sure you use %{$python_} macros instead of %python_ macros
  7. use %python_bin_suffix instead of %python_version for distinguishing executables or build directories
  8. do not use %ifpython3 or %python3_only, if the line or code block should apply for the primary Python 3 flavor (exlusively or also for possible additional flavors).

Running tests

The Python code is not really compiled like C applications. As such, it is really easy to distribute something that does not work at all. In order to prevent this, all applications must run and pass their tests unless upstream is providing none themselves.

The easiest way to run tests is using the unittest discover mode for which we have macros (when using a recent python-rpm-macros package): %pyunittest -v for noarch packages and %pyunittest_arch -v for platform-dependent packages.

This of course only works if the package supports regular Python unittests. If this call does not work, one has to dig around and check various files in the development repository, such as .github/workflows, .travis.yml, or tox.ini.

To run pytest, the macros %pytest and %pytest_arch can be used.

The macros support supplying additional command flags, e.g. for deselecting tests which should not be run.

More macros

The full documentation for all macros defined for single-spec can be found at the GitHub page for python-rpm-macros package.