openSUSE:Packaging Python
Build Service Tutorial · Tips & Tricks · Cross Distribution Howto · Packaging checks
Desktop menu categories · RPM Macros · Scriptlets · Init scripts · How to write good changes
Contents
- 1 The fast and automated way
- 2 Hints on how to package Python modules manually
- 2.1 What is single-spec?
- 2.2 Compatibility shims
- 2.3 Naming policy
- 2.4 Source URLs
- 2.5 BuildRequires
- 2.6 Requires, Provides and similar
- 2.7 %python_subpackages
- 2.8 Subpackage declarations
- 2.9 Build macros
- 2.10 %python_expand
- 2.11 Naming flavor-specific files
- 2.12 Filelists
- 2.13 Executables
- 2.14 Packages for single Python version
- 2.15 Conditionals on Python versions
- 2.16 Common Gotchas
- 2.17 Running tests
- 3 All Macros
The fast and automated way
Lets suppose you want to package zope.interface and you don't know how it is named exactly or where to download from. First of all, you can search for it with py2pack, which can be found in the python-py2pack package and download the source tarball automatically with it if you found the correct module:
$ py2pack search zope.interface searching for module zope.interface... found zope.interface-3.6.1
$ py2pack fetch zope.interface downloading package zope.interface-3.6.1... from http://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.6.1.tar.gz
As a next step you may want to generate a package recipe for your distribution. For RPM-based distributions, you want to generate a spec file named python-zopeinterface.spec:
$ py2pack generate zope.interface -f python-zope.interface.spec
The source tarball and package recipe is all you need to generate the RPM file. This final step may depend on which distribution you use. Again, for openSUSE (and by using the openSUSE Build Service), the complete recipe becomes:
$ osc mkpac python-zope.interface $ cd python-zope.interface $ py2pack fetch zope.interface $ py2pack generate zope.interface -f python-zope.interface.spec $ vi *.spec BuildRequires: python-setuptools ZZ $ osc build $ osc vc $ osc add * $ osc commit
The first line uses osc, the Build Service command line tool to generate a new package (preferrably in your Build Service home project). The py2pack steps are known already.
It is always a good idea to review all the dependencies specified in the spec file accordingly to upstream manifest and requirements. Ie. dropping devel dependency in a case where package does not compile anything or adding python-setuptools if needed. Finally, the package is tested (built locally), a changes (package changelog) file is generated (with ‘osc vc’) and the result is sent back to the Build Service for public consumption. However, depending on the Python module, you may have to adapt the generated spec file slightly. Py2pack is quite clever and tries to auto-generate as much as it can, but it depends on the metadata that the module provides. Thus, bad metadata implies mediocre results. To get further help about py2pack usage, issue the following command:
$ py2pack help
Often, the first run of 'osc build' will fail with build error messages, due to missing python modules. Look for the keyword 'import', this should give you a hint, what needs to be added to the 'BuildRequires:' of you spec-file.
Hints on how to package Python modules manually
Manual packaging is discouraged since we have tools to do that. If you insist on having fun, please consider any package spec file in devel:languages:python. python-pbr, python-mock or python-Twisted are few examples.
In general we are using python single-spec packaging.
What is single-spec?
'Single-spec' (or 'Singlespec') is an approach for packaging both python2 and python3 variants of a Python module from a single source spec file.
Compatibility shims
If your packages are targeted for anything other than Leap 15 or newer, the spec file must include two things:
Redefinition of
%python_module
macro.%{?!python_module:%define python_module() python-%{**} python3-%{**}}
Build requirement on
python-rpm-macros
:BuildRequires: python-rpm-macros
Naming policy
SUSE has a policy for names of Python module packages. A module is to Python what shared libraries are to C - a piece of code that doesn't work by itself, but provides functionality to other Python programs.
All Python module packages, whether pure Python or C-based, should be called python-modulename. modulename should be the name of this module on the Python Package Index.
Previously, Python packages have been named after directories in the site-packages hierarchy. Often, this was an arbitrary choice, several modules install more than one directory there. Also, the Python universe includes modules that share the same directory name (search for "daemon" on PyPI) and it was not always easy to find out the right package name. Furthermore, the new approach has several advantages:
- Scripts (like py2pack) know where to fetch metadata and (new) release tarballs, PyPI has a stable API
- Package names match install_required, etc. in setup.py files and pip requirements files making it easy to find out (Build)Requires in your RPM specs
- Users know where to look for API or usage documentation: http://pypi.python.org/pypi/$PYPYNAME
This policy doesn't apply to end-user applications that are not designed and planned to be used as libraries - so if you're packaging something that is going to have an icon in the application menu, you should just call the package by its normal name (as found on the Python Package Index).
There are some corner cases as to what is an application and what is a module - for example, many modules come with simple command-line tools that allow you to use a subset of their functionality directly. The rule of thumb is this: if this package is going to be a dependency of some other Python application, apply the naming policy, otherwise keep it just as the module name.
Source URLs
For packages that are available from PyPI, the correct Source URL is the following:
https://files.pythonhosted.org/packages/source/t/tox-%{version}.tar.gz
The preferred hostname is files.pythonhosted.org. All other hostnames (like pypi.io, pypi.org, pypi.python.org) eventually redirect to it. In order to ease this transition you can put in the bellow mentioned link and it will be automatically converted to the proper format by spec-cleaner tool.
The link you get from PyPI will point to some sort of hash, e.g., https://files.pythonhosted.org/packages/99/20/2fd208e75c05975bf6436258f469aa233045b93d195cdb442045de0923cc/tox-2.7.0.tar.gz. We don't want that. The "semantic URL" is still available:
/packages/source/<first letter>/<full name>/<full name>-<version>.<extension>
BuildRequires
BuildRequires are not conditional and apply for the whole spec file. There is no such thing as "if I build for Python 2, I require package foo"; you simply require package foo always.
If you need a package for all available flavors of python, use the %python_module
macro. To ilustrate converting from direct dependencies to %python_module
:
BuildRequires: python-setuptools BuildRequires: python-py >= 1.4
to this:
BuildRequires: %{python_module setuptools} BuildRequires: %{python_module py >= 1.4}
Note that the version requirement goes inside the %python_module
call.
This is not intended for Requires tag. The %python_module
macro is only for BuildRequires
If you only need the package for one flavor of python, simply don't use the macro and state the dependencies directly:
BuildRequires: python2-enum34 BuildRequires: python3-astroid
If you need to BuildRequire "python", you can instead use %pythons
:
BuildRequires: %{pythons}
Requires, Provides and similar
In many cases, you don't need to do anything. The single-spec rewriter will convert your Requires to match the generated package. Only make sure the Requires content really matches up setup.py or requirements.txt and there is nothing missing.
%python_module
for Requires
.Package names in tags Requires (as well as "Requires(pre)" and all the others), Provides, Recommends, Suggests, Obsoletes, Conflicts, Supplements, and Enhances are automatically converted.
The converter takes into account the packageand
expression, and %requires_ge
and %requires_eq
macros. However, support for these must be coded explicitly. If you find an expression that is not converted, please either post it on the packaging ML or file a bug against python-rpm-macros.
If the requirement name starts with "python-
", or with the same name as your package (so "python3-bar
" in package "python3-foo
"), the python name is changed to match that of the generated package.
This also works for "python
" itself.
Packaged listed in packageand
expression are also converted.
Requirements specific for one python flavor
You can specify that some packages should only be included for some python flavors, by wrapping them in
conditionals using the %python_flavor
variable
Requires: python-idna %if "%{python_flavor}" == "python2" Requires: python2-enum34 %endif
As a shortcut, for every flavor, there is a %ifpython
macro: %ifpython2
, %ifpython3
or %ifpypy3
.
%ifpython2 Requires: python2-enum34 %endif
Note that the shortcuts must not be nested in other conditionals, otherwise you can get "%endif without %if" error message. If you need to nest conditionals, use the %python_flavor
conditional.
%prep, %build, %install, %check
.The
%ifpython
sections are copied (or not) to subpackage definitions. Subpackages have their own requirements, file lists and %pre
/%post
/etc. scriptlets. The scripts from %prep
and others is shared. %ifpython
will do the wrong thing.Obsoleting and Providing old symbols
The following sequence:
Obsoletes: python-distribute < %{version} Provides: python-distribute = %{version}
would mean that your python2-package
will obsolete/provide python2-distribute
and your python3-package
will obsolete/provide python3-distribute
and your pypy3 package will obsolete/provide pypy3-distribute
.
Often, this is not what you want.
First of all, in many cases, this is only applicable for Python 2, so the sequence should be wrapped in %ifpython2
conditional.
Second, you will note that neither package obsoletes/provides python-distribute
.
Here is how you do that:
%define oldpython python %ifpython2 Obsoletes: %{oldpython}-distribute < %{version} Provides: %{oldpython}-distribute = %{version} %endif
%python_module
in Provides
If you are creating a subpackage that will be common for all flavors (could be a -doc
subpackage), sometimes you need to provide a symbol for all flavors.
E.g., your package python-foo-doc
should also provide python2-foo-doc
, python3-foo-doc
etc.
In such case, you use the %python_module
macro:
%package -n python-foo-doc Provides: %{python_module foo-doc = %{version}}
This is also the one case where you could use %python_module
in Requires.
(See below on declaring packages with -n
to prevent autogeneration.)
%python_subpackages
Easy enough: place the %python_subpackages
macro on a separate line at the end of the spec preamble (the part that ends where your package's %description
begins).
Provides: pylint BuildRoot: %{tmproot}/blabla BuildArch: noarch %python_subpackages %description
This macro emits all the subpackage descriptions, %files
and scriptlet sections for the autogenerated parts.
Subpackage declarations
Subpackages are converted automatically. If you have a subpackage %package foo
, singlespec will create python3-yourpackage-foo
and all the rest from it.
If you want to prevent this, use %package -n %{name}-foo
. Or the full name, %package -n python-yourpackage-foo
. This will ensure that the subpackage will be skipped.
This also means that packages named %package -n yourpackage-python
will not be processed. There will be a mechanism for these in the future.
Common documentation packages
It is very common that the shipped documentation and probably included examples are quire big and should be in a separate sub-package. The package could have - for example - these sections:
%package -n %{name}-docs Summary: Documentation files for %name Group: Documentation/Other %description -n %{name}-docs HTML Documentation and examples for %name. %files -n %{name}-docs %doc examples docs/_build/html/
Build macros
For building the packages use %python_build
.
For installing the packages use %python_install
.
These macros already contain the usual options (--root
, --prefix
), so in the typical case, you don't need to supply any options. If you have something specific, you can add it: %python_build --enable-specific-feature
.
If you set environment variables, export them first:
export CFLAGS="-fwrapv" %python_build
For any other commands, you can use %python_exec
. That is also a good way to run pythonic executables. For example:
%check %python_exec setup.py test %python_exec %{_bindir}/nosetests %python_expand PYTHONPATH=%{buildroot}%{$python_sitelib} py.test-%{$python_version}
%python_expand
For anything more complicated than executing all the interpreters, use the %python_expand
macro. This will repeatedly expand the %{$python}
, %{$python_sitelib}
etc. strings to the currently used flavor.
%python_expand rm -r %{buildroot}%{$python_sitelib}/file.txt
results in (apart from some build dir manipulations):
rm -r %{buildroot}%{python2_sitelib}/file.txt rm -r %{buildroot}%{python3_sitelib}/file.txt rm -r %{buildroot}%{pypy3_sitelib}/file.txt
IMPORTANT: you can use %python_expand
to replace macro definitions, but make sure you use, e.g., %{
$python
_sitelib}
, with the $
sign.
If you use plain %{python_sitelib}
, the macro will be expanded before %python_expand
can modify it.
A common use for %python_expand
is with fdupes
, as in:
%python_expand %fdupes %{buildroot}%{$python_sitelib}/mymodule
You can use multiline %python_expand
if you enclose the lines in {}
. The only technical limitation is that the first line must not be empty (but can be a comment starting with #
):
%{python_expand # this will expand the following section %$python_install mv %{buildroot}%{_bindir}/exename %{buildroot}%{_bindir}/exename-%{$python_bin_suffix} }
This is eg useful when running tests and some pre-step is needed:
%check %{python_expand rm -rf .testrepository $python setup.py test }
Naming flavor-specific files
Executables specific to a particular flavor should use %python_bin_suffix
instead of %python_version
for names. This expands to %python_version
for CPython, pp%{python_version}
for PyPy. If we support Jython, it will get a specific bin_suffix
too.
Build directories specific to a particular flavor should use %python_prefix
. This expands to the flavor name.
Filelists
If your package is called python-something
(that is, the name prefix is "python" and not a specific flavor), you must mark your %files
sections with %{python_files}
macro.
%files %{python_files} %{python_sitelib}/foo %files %{python_files plugins} %{python_sitelib}/foo-plugins
You can use %ifpython2
and similar to conditionally include some files only in some flavors. In addition, you can use shorthands:
%files %{python_files} %{python_sitelib}/foo %{python_sitelib}/foorun.py* %python3_only %{_bindir}/foorun %pycache_only %{python_sitelib}/__pycache__
Use %pycache_only
or %ifpycache
to mark __pycache__
directories.
Executables
For new packages that carry executables, it is preferred that these are marked %python3_only
. That ensures that the executables are only installed for the preferred python flavor, which is Python 3.
If your package functionality depends on version of Python (e.g., test harnesses, source code processors, installers, etc.), you should package executables for all flavors. This is easy if the package itself installs versioned executables. If not, use the %python_clone
macro.
%python_install %python_clone %{buildroot}%{_bindir}/executable
This macro will copy the executable
as executable-%python_bin_suffix
for all flavors, and change the shebang line to the appropriate interpreter.
%python_clone
works for all kinds of files. If it recognizes a man page extension, it will correctly rename manpage.1
to manpage-%python_bin_suffix.1
.
When using %python_clone
, you need to list both the original executable *and* the copied executables in your filelist:
%{_bindir}/executable-%{python_bin_suffix} %python3_only %{_bindir}/executable
If changing the shebang line is not appropriate or sufficient, you will need to catch the executable after every installation, like this:
%{python_expand %$python_install mv %{buildroot}%{_bindir}/executable %{buildroot}%{_bindir}/executable-%{$python_bin_suffix} }
Python 2 stacks
The recommendation for preferring Python 3 is intended for new code.
Existing packages that are parts of Python 2 stacks (such as the current (2017-03) OpenStack Cloud) need executables for Python 2 as well. For these, it is preferable to use the alternatives approach outlined below.
update-alternatives
Sometimes it is useful to let the user switch the unversioned executable name to one version of the package or another.
Also, if you can't be sure that the user has a Python 3 stack, and want to provide unversioned executable names regardless, update-alternatives is the way to go.
update-alternatives allows you to switch other things (usually man pages) along with executables.
As a prerequisite, you need to have version-specific file names available -- that is, for every file
you provide, file-%python_bin_suffix
should exist for all flavors.
(for manpage.1
, the appropriate name is manpage-%python_bin_suffix.1
)
First, you need to set up the alternatives in %install
section.
- If you're using
%python_clone
to create the executable, simply pass -a option:%python_clone -a %{buildroot}%{_bindir}/executable
- If not, and the file in question is
%{_bindir}/something
, call%prepare_alternative something
- If the file is not in
%{_bindir}
, you have to specify the path:%prepare_alternative -t /path/to/file file
. The path is without%buildroot
Second, create the appropriate %post
and %postun
sections. Make sure that you have the proper requirements:
Requires(post): update-alternatives Requires(postun): update-alternatives
Then use %python_install_alternative
and %python_uninstall_alternative
respectively.
%post %python_install_alternative exename %postun %python_uninstall_alternative exename
Third, mention the alternative, unversioned, in file list
%python_alternative %{_bindir}/exename
You can examine a full spec file at [1].
Grouped alternatives
The update-alternatives system allows for multiple files in the same group, to be switched together. This is useful if you want to install an executable along with its manpage, or multiple executables belonging to the same function group.
To make use of this, simply specify the files as multiple arguments to %python_install_alternative
:
%{python_install_alternative pylint pylint.1 epylint epylint.1}
The macro can recognize manpage names and handle them correctly, but the first arguments needs always to the an executable. Or you can specify a full path to the file in question.
The first argument to %python_install_alternative
is the group name. This is the only argument for %python_uninstall_alternative
; you uninstall the whole group by the one name.
Packages for single Python version
Packages that only exist for Python 2 should be left as is. New packages should be called python2-foo
. Old packages should be left as python-foo
, but you should add Provides: python2-foo
. You should also make sure that all BuildRequires and Requires are for "python2" and "python2-foo".
Packages that only exist for Python 3 can be left as is, or they can be converted to singlespec, with %define skip_python2 1
. This is to ensure that they can build for PyPy or other flavors in the future. The definition should be made at the top of the spec-file.
Conditionals on Python versions
If you need to differentiate between python versions you can use the following macros:
Macros | Example |
%python2_version_nodots |
27 |
%python3_version_nodots |
36 |
For example:
%if %python3_version_nodots > 34 ... %endif
Common Gotchas
Things to look out for (which can also get your submission declined):
- If the
%python_module
redefinition is present, check that it has"python-%{**}"
and not"python-%1"
- use
%python_module
for BuildRequires - do NOT use
%python_module
for Requires - make sure
%python_subpackages
macro is present -
%files sections
should be marked as"%files %{python_files}"
- in
%python_expand
, make sure you use%$python_
macros instead of%python_
macros - use
%python_bin_suffix
instead of%python_version
for distinguishing executables or build directories
Running tests
The python code is not really compiled like C applications. As such it is really easy to distribute something that does not work at all. In order to prevent this all applications must run and pass their tests unless upstream is providing none themselves.
The easiest way to run tests is using setuptools test subcommand:
%python_exec setup.py test
This of course only works, if test_suite or cmd_class is properly defined in your setup.py. If the setup.py call fails or executes no tests one has to dig around checing various files in the development repository like .travis.yml or tox.ini.
All Macros
The full documentation for all macros defined for singlespec can be found at the GitHub page for python-rpm-macros package.