Home Wiki > openSUSE:Packaging Python
Sign up | Login

openSUSE:Packaging Python

tagline: From openSUSE


The Packaging Python is a step by step introduction on how to build Python software packages for openSUSE and others using the openSUSE Build Service.

python single-spec

Warning Recently the maintainers of devel:languages:python, the main development project for python packages within openSUSE, changed to a 'single-spec' approach for providing python2 + python3 variants from a single source spec file. The wiki has not yet been fully updated, see the separate page openSUSE:Packaging Python Singlespec

The fast and automated way

Lets suppose you want to package zope.interface and you don't know how it is named exactly or where to download from. First of all, you can search for it with py2pack, which can be found in the python-py2pack package and download the source tarball automatically with it if you found the correct module:

$ py2pack search zope.interface
searching for module zope.interface...
found zope.interface-3.6.1
$ py2pack fetch zope.interface
downloading package zope.interface-3.6.1...
from http://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.6.1.tar.gz

As a next step you may want to generate a package recipe for your distribution. For RPM-based distributions, you want to generate a spec file named python-zopeinterface.spec:

$ py2pack generate zope.interface -f python-zopeinterface.spec

The source tarball and package recipe is all you need to generate the RPM file. This final step may depend on which distribution you use. Again, for openSUSE (and by using the openSUSE Build Service), the complete recipe becomes:

$ osc mkpac python-zopeinterface
$ cd python-zopeinterface
$ py2pack fetch zope.interface
$ py2pack generate zope.interface -f python-zopeinterface.spec
$ vi *.spec
BuildRequires: python-setuptools
ZZ
$ osc build
$ osc vc
$ osc add *
$ osc commit

The first line uses osc, the Build Service command line tool to generate a new package (preferrably in your Build Service home project). The py2pack steps are known already. It is always good to review the BuildRequires (with 'vi *.spec') and e.g. add python-setuptools if needed. Finally, the package is tested (built locally), a changes (package changelog) file is generated (with ‘osc vc’) and the result is sent back to the Build Service for public consumption. However, depending on the Python module, you may have to adapt the generated spec file slightly. Py2pack is quite clever and tries to auto-generate as much as it can, but it depends on the metadata that the module provides. Thus, bad metadata implies mediocre results. To get further help about py2pack usage, issue the following command:

$ py2pack help

Often, the first run of 'osc build' will fail with build error messages, due to missing python modules. Look for the keyword 'import', this should give you a hint, what needs to be added to the 'BuildRequires:' of you spec-file.


Hints on how to package Python modules manually

Manual packaging is discouraged since we have tools to do that. If you insist on having fun, please consider any package spec file in devel:languages:python3. python-nose, python-pip or python-Sphinx are good examples.

Naming policy

SUSE has a policy for names of Python module packages. A module is to Python what shared libraries are to C - a piece of code that doesn't work by itself, but provides functionality to other Python programs.

All Python module packages, whether pure Python or C-based, should be called python-modulename. modulename should be the name of this module on the Python Package Index, the official third-party software repository for the Python programming language.

Previously, Python packages have been named after directories in the site-packages hierarchy. Often, this was an arbitrary choice, several modules install more than one directory there. Also, the Python universe includes modules that share the same directory name (search for "daemon" on PyPI) and it was not always easy to find out the right package name. Furthermore, the new approach has several advantages:

This policy doesn't apply to end-user applications - so if you're packaging something that is going to have an icon in the application menu, you should just call the package by its normal name (as found on the Python Package Index).

There are some corner cases as to what is an application and what is a module - for example, many modules come with simple command-line tools that allow you to use a subset of their functionality directly. The rule of thumb is this: if you think that the users will install your package to use the command line tool, call it by its normal name. If this package is going to be a dependency of some other Python application, apply the naming policy.

Source URLs

For packages that are available from PyPI, the correct Source URL is the following:

https://files.pythonhosted.org/packages/source/t/tox-%{version}.tar.gz

The preferred hostname is files.pythonhosted.org. All other hostnames (like pypi.io, pypi.org, pypi.python.org) eventually redirect to it.

The link you get from PyPI will point to some sort of hash, e.g., https://files.pythonhosted.org/packages/99/20/2fd208e75c05975bf6436258f469aa233045b93d195cdb442045de0923cc/tox-2.7.0.tar.gz. We don't want that. The "semantic URL" is still available:
/packages/source/<first letter>/<full name>/<full name>-<version>.<extension>

BuildRequires

In all cases, use BuildRequires: python-devel. Technically BuildRequires: python-base is sufficient for Python-only modules (i.e. no C code), but it increases consistency and doesn't add much overhead.

Some modules require setuptools to build. In this case add the package BuildRequires: python-setuptools.

Due to how the Python interpreter is spread across the packages "python" and "python-base", you sometimes need to say BuildRequires: python in your spec files. This is especially true if you need one of the following Python modules at build time: ssl, _ssl, md5, sha.

Python version

In openSUSE, even version-agnostic python packages need to depend on a specific python series. A python series is denoted by python's major and minor version. For example, the current (as of 3.4.2013) python series is 2.7 for the legacy line and 3.3 for Python 3.
This is because you are installing into version-specific directory /usr/lib(64)/pythonX.Y/site-packages, where X.Y is the series.

File locations

All python source and bytecode files should go into /usr/lib(64)/pythonX.Y/site-packages, or maybe /usr/lib(64)/yourapp. FHS says that /usr/share hierarchy should only contain data, so don't install sources in there.
This is not a strict requirement, though, only a recommendation. If your package's upstream is bent on installing to /usr/share, please try to convince them of the error of their ways, but don't feel obliged to patch the package to death just so it installs into /usr/lib.

File lists

There are two useful macros to list files:

  • %python_sitelib expands to /usr/lib/pythonX.Y/site-packages. This is the install location for platform-independent modules.
  • %python_sitearch expands to %{_libdir}/pythonX.Y/site-packages, that is, either /usr/lib or /usr/lib64, depending on your architecture. This is the install location for platform-dependent modules.

So, for platform-independent Python packages, the simplest example would look like this:

%install
python setup.py install --prefix=%{_prefix} --root=%{buildroot}

%files
%defattr(-,root,root)
%{python_sitelib}/*

To avoid file conflicts with packages for other Python interpreters, binaries and/or man-pages should be suffixed with the Interpreter versions, e.g.:

%{_bindir}/foo

should become

%{_bindir}/foo-%{py_ver}

Please check the Python3 / parallel installation section below.

System Architecture

If your package only contains python sources (.py), bytecode files (.pyc and .pyo) and platform-independent data, you should mark it as noarch. Include BuildArchitectures: noarch at the start of your spec file. Such module should install entirely into %python_sitelib.

Otherwise, the whole module installs into %python_sitearch. Note that it is not possible to have part of a module in %python_sitelib and another part in %python_sitearch. And in most cases, even if package contains more than one module, all of them should be in one prefix.

This is important so i'm going to stress it: the entire module is either platform-independent, or it isn't. Even one binary library or platform-specific config file in otherwise pure python module will mark the entire module as platform-dependent and you won't be able to use it as noarch. This is usually handled by distutils, so you shouldn't need to worry most of the time.

Certain packages might install parts of themselves into %python_sitelib and parts into %python_sitearch. Such setup is a common source of bugs and problems. Unless you know exactly what you're doing, you should modify such packages so that everything goes into %python_sitearch.

Byte Compiled Files

Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled python) or .pyo (optimized compiled python). These files are a byte code that is portable across OSes. If you do not include them in your packages, python will try to create them when the user runs the program. If the system administrator uses them, then the files will be successfully written. Later, when the package is removed, the .pyc and .pyo files will be left behind on the filesystem. To prevent that you need to byte compile the files when building your package and include the files in the %files section.

Many packages install byte-compiled .pycs and .pyos by themselves. If your package doesn't do that, you should use the macro %{py_compile} in this way: %py_compile <directory> to create .pyc and %py_compile -O <directory> for .pyo files.

Most of the time, .pyo files are exactly the same as .pyc. You can save space by running fdupes to hardlink them together.

BuildRequires: fdupes
(...)

%install
(...)
%fdupes $RPM_BUILD_ROOT%{py_sitedir}

Summary of useful rpm macros

  • %py_ver - python series denomination (major.minor version number)
  • %py_compile - byte-compiles python sources from the specified directory. Use %{py_compile -O} to produce optimized bytecode (.pyo)
  • %python_sitelib - site-packages directory for platform-independent modules. Expands to /usr/lib/python%{py_ver}/site-packages
  • %python_sitearch - site-packages directory for platform-dependent modules. Expands to %{_libdir}/python%{py_ver}/site-packages

Compatibility with older distributions

In 11.1, split python package was introduced. If you want to build a package for older distribution, you cannot require python-base. Note that python-base was introduced because it has almost no build-time or run-time dependencies. That means that it is unblocked sooner in the rebuild cycle, and conversely, if your package doesn't buildrequire python, it can be build sooner.

The possibility to build noarch packages, along with %python_sitelib and %python_sitearch, was introduced in 11.2. If you need compatibility with older distributions, you must define the macros you are using. Place this at the start of your spec:

%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
%{!?python_sitearch: %global python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}

If your package is noarch, you must wrap the noarch declaration in a conditional sequence.

This would build your package as noarch only on openSUSE 11.2 and higher:
%if 0%{?suse_version} >= 1120
BuildArchitectures: noarch
%endif
And this would build your package as noarch on openSUSE 11.2 and higher, plus on any non-SUSE distro. (Useful when building packages for Fedora)
%if %{?suse_version: %{suse_version} > 1110} %{!?suse_version:1}
BuildArchitectures: noarch
%endif

Running tests

The easiest way to run tests is using setuptools test subcommand:

python setup.py test

This of course only works, if test_suite or cmd_class is properly defined in your setup.py. E.g. if you use external modules to find the tests:

  • cmdclass = {'test': pytest}
  • test_suite='nose.collector'

Or the tests can be run by directly calling a module in `tests/` or `packagename/tests`:

  • test_suite='tests'
  • test_suite='packagename.tests'

This also works for packages with compiled components.