openSUSE:Packaging Python

tagline: From openSUSE

Contents 1 The fast and automated way 2 Hints on how to package Python modules manually 2.1 A simplest case 2.2 Naming policy 2.3 BuildRequires 2.4 Python version 2.5 File locations 2.6 File lists 2.7 System Architecture 2.8 setuptools/eggs 2.9 Byte Compiled Files 2.10 Summary of useful rpm macros 2.11 Compatibility with older distributions 3 Hints on how to Package Python3 modules 3.1 Building Python2 And Python3 Modules together

The fast and automated way

Lets suppose you want to package zope.interface and you don’t know how it is named exactly or where to download from. First of all, you can search for it with py2pack, which can be found in the python-py2pack package and download the source tarball automatically with it if you found the correct module:

$ py2pack search zope.interface
searching for module zope.interface...
found zope.interface-3.6.1
$ py2pack fetch zope.interface
downloading package zope.interface-3.6.1...
from http://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.6.1.tar.gz

As a next step you may want to generate a package recipe for your distribution. For RPM-based distributions, you want to generate a spec file named python-zopeinterface.spec:

$ py2pack generate zope.interface -t opensuse.spec -f python-zopeinterface.spec

The source tarball and package recipe is all you need to generate the RPM file. This final step may depend on which distribution you use. Again, for openSUSE (and by using the openSUSE Build Service), the complete recipe becomes:

$ osc mkpac python-zopeinterface
$ cd python-zopeinterface
$ py2pack fetch zope.interface
$ py2pack generate zope.interface -f python-zopeinterface.spec
$ osc build
$ osc vc
$ osc commit

The first line uses osc, the Build Service command line tool to generate a new package (preferrably in your Build Service home project). The py2pack steps are known already. Finally, the package is tested (built locally), a changes (package changelog) file is generated (with ‘osc vc’) and the result is sent back to the Build Service for public consumption. However, depending on the Python module, you may have to adapt the generated spec file slightly. Py2pack is quite clever and tries to auto-generate as much as it can, but it depends on the metadata that the module provides. Thus, bad metadata implies mediocre results. To get further help about py2pack usage, issue the following command:

$ py2pack help

---

Hints on how to package Python modules manually

A simplest case

Here's a simple example spec file (from package python-Jinja2). You probably don't need to depend on 'python-distribute'

 #
 # spec file for package python-Jinja2
 #
 # Copyright (c) 2012 SUSE LINUX Products GmbH, Nuernberg, Germany.
 #
 # All modifications and additions to the file contributed by third parties
 # remain the property of their copyright owners, unless otherwise agreed
 # upon. The license for this file, and modifications and additions to the
 # file, is the same license as for the pristine package itself (unless the
 # license for the pristine package is not an Open Source License, in which
 # case the license is the MIT License). An "Open Source License" is a
 # license that conforms to the Open Source Definition (Version 1.9)
 # published by the Open Source Initiative.
 
 # Please submit bugfixes or comments via http://bugs.opensuse.org/
 #
 
 
 Name:           python-Jinja2
 Version:        2.6
 Release:        0
 Summary:        A fast and easy to use template engine written in pure Python
 License:        BSD-3-Clause
 Group:          Development/Languages/Python
 Url:            http://jinja.pocoo.org/
 Source:         http://pypi.python.org/packages/source/J/Jinja2/Jinja2-%{version}.tar.gz
 BuildRequires:  fdupes
 BuildRequires:  python-devel
 BuildRequires:  python-distribute
 Provides:       python-jinja2 = %{version}
 Obsoletes:      python-jinja2 < %{version}
 BuildRoot:      %{_tmppath}/%{name}-%{version}-build
 %if 0%{?suse_version} && 0%{?suse_version} <= 1110
 %{!?python_sitelib: %global python_sitelib %(python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")}
 %else
 BuildArch:      noarch
 %endif
 
 %description
 Jinja2 is a template engine written in pure Python.  It provides a Django
 inspired non-XML syntax but supports inline expressions and an optional
 sandboxed environment.
 
 %package vim
 Summary:        Jinja2 syntax files for Vim
 License:        BSD-3-Clause
 Group:          Productivity/Text/Editors
 Requires:       %{name} = %{version}
 
 %description vim
 Vim syntax highlighting scheme for Jinja2 templates.
 
 %package emacs
 Summary:        Jinja2 syntax files for Emacs
 License:        GPL-2.0+
 Group:          Productivity/Text/Editors
 Requires:       %{name} = %{version}
 
 %description emacs
 Emacs syntax highlighting scheme for Jinja2 templates.
 
 %prep
 %setup -q -n Jinja2-%{version}
 
 %build
 python setup.py build
 sed -i 's/\r$//' LICENSE # Fix wrong EOL encoding
 
 %install
 python setup.py install --prefix=%{_prefix} --root=%{buildroot}
 install -Dm644 ext/Vim/jinja.vim %{buildroot}%{_datadir}/vim/site/syntax/jinja.vim # Install VIM syntax file
 install -Dm644 ext/jinja.el %{buildroot}%{_datadir}/emacs/site-lisp/jinja.el # Install Emacs syntax file
 %if 0%{?suse_version} > 1010
 %fdupes %{buildroot}%{_prefix}
 %endif
 
 %files
 %defattr(-,root,root,-)
 %doc AUTHORS CHANGES LICENSE artwork examples ext/JinjaTemplates.tmbundle.tar.gz
 %{python_sitelib}/*
 
 %files vim
 %defattr(-,root,root,-)
 %{_datadir}/vim
 
 %files emacs
 %defattr(-,root,root,-)
 %{_datadir}/emacs/site-lisp/jinja.el
 
 %changelog

Naming policy

SUSE has a policy for names of Python module packages. A module is to Python what shared libraries are to C - a piece of code that doesn't work by itself, but provides functionality to other Python programs.

All Python module packages, whether pure Python or C-based, should be called python-modulename. modulename should be the name of this module on the Python Package Index, the official third-party software repository for the Python programming language.

Previously, Python packages have been named after directories in the site-packages hierarchy. Often, this was an arbitrary choice, several modules install more than one directory there. Also, the Python universe includes modules that share the same directory name (search for "daemon" on PyPI) and it was not always easy to find out which one was packages. Furthermore, the new approach has several advantages:

• Scripts (like py2pack) know where to fetch metadata and (new) release tarballs, PyPI has a stable API
• Package names match install_required, etc. in setup.py files and pip requirements files making it easy to find out (Build)Requires in your RPM specs
• Users know where to look for API or usage documentation: http://pypi.python.org/pypi/$PYPYNAME

This policy doesn't apply to end-user applications - so if you're packaging something that is going to have an icon in the application menu, you should just call the package by its normal name (as found on the Python Package Index).

There are some corner cases as to what is an application and what is a module - for example, many modules come with simple command-line tools that allow you to use a subset of their functionality directly. The rule of thumb is this: if you think that the users will install your package to use the command line tool, call it by its normal name. If this package is going to be a dependency of some other Python application, apply the naming policy.

BuildRequires

In all cases, use BuildRequires: python-devel. Technically BuildRequires: python-base is sufficient for Python-only modules (i.e. no C code), but it increases consistency and doesn't add much overhead.

Some modules require setuptools to build. In this case add the package python-setuptools or preferably it's succcessor python-distribute to BuildRequires.

Python version

In openSUSE, even version-agnostic python packages need to depend on a specific python series. A python series is denoted by python's major and minor version. For example, the current (as of 3.4.2013) python series is 2.7 for the legacy line and 3.3 for Python 3.
This is because you are installing into version-specific directory /usr/lib(64)/pythonX.Y/site-packages, where X.Y is the series.

File locations

All python source and bytecode files should go into /usr/lib(64)/pythonX.Y/site-packages, or maybe /usr/lib(64)/yourapp. FHS says that /usr/share hierarchy should only contain data, so don't install sources in there.
This is not a strict requirement, though, only a recommendation. If your package's upstream is bent on installing to /usr/share, please try to convince them of the error of their ways, but don't feel obliged to patch the package to death just so it installs into /usr/lib.

File lists

There are two useful macros to list files:

• %python_sitelib expands to /usr/lib/pythonX.Y/site-packages. This is the install location for platform-independent modules.
• %python_sitearch expands to %{_libdir}/pythonX.Y/site-packages, that is, either /usr/lib or /usr/lib64, depending on your architecture. This is the install location for platform-dependent modules.

So, for platform-independent Python packages, the simplest example would look like this:

%install
python setup.py install --prefix=%{_prefix} --root=%{buildroot}

%files
%defattr(-,root,root)
%{python_sitelib}/*

System Architecture

If your package only contains python sources (.py), bytecode files (.pyc and .pyo) and platform-independent data, you should mark it as noarch. Include BuildArchitectures: noarch at the start of your spec file. Such module should install entirely into %python_sitelib.

Otherwise, the whole module installs into %python_sitearch. Note that it is not possible to have part of a module in %python_sitelib and another part in %python_sitearch. And in most cases, even if package contains more than one module, all of them should be in one prefix.

This is important so i'm going to stress it: the entire module is either platform-independent, or it isn't. Even one binary library or platform-specific config file in otherwise pure python module will mark the entire module as platform-dependent and you won't be able to use it as noarch. This is usually handled by distutils, so you shouldn't need to worry most of the time.

Certain packages might install parts of themselves into %python_sitelib and parts into %python_sitearch. Such setup is a common source of bugs and problems. Unless you know exactly what you're doing, you should modify such packages so that everything goes into %python_sitearch.

setuptools/eggs

In the past, Fedora has done the minimal amount to support eggs for upstream distros. As eggs are being adopted more widely upstream we need to have more comprehensive documentation on how to handle this. The following are a summary of the guidelines for reviewers to go over. The complete policy includes examples and rationale for the way we do things.

• Must: Python eggs must be built from source. They cannot simply drop an egg from upstream into the proper directory.
• Must: Python eggs must not download any dependencies during the build process.
• Must: If egg-info files are generated by the modules build scripts they must be included in the package.
• Must: When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
• Must: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
• Should: A package which is used by another package via an egg interface should provide egg info.

Byte Compiled Files

Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled python) or .pyo (optimized compiled python). These files are a byte code that is portable across OSes. If you do not include them in your packages, python will try to create them when the user runs the program. If the system administrator uses them, then the files will be successfully written. Later, when the package is removed, the .pyc and .pyo files will be left behind on the filesystem. To prevent that you need to byte compile the files when building your package and include the files in the %files section.

Many packages install byte-compiled .pycs and .pyos by themselves. If your package doesn't do that, you should use the macro %{py_compile} in this way: %py_compile <directory> to create .pyc and %py_compile -O <directory> for .pyo files.

Most of the time, .pyo files are exactly the same as .pyc. You can save space by running fdupes to hardlink them together.

BuildRequires: fdupes
(...)

%install
(...)
%fdupes $RPM_BUILD_ROOT%{py_sitedir}

Summary of useful rpm macros

• %py_ver - python series denomination (major.minor version number)
• %py_compile - byte-compiles python sources from the specified directory. Use %{py_compile -O} to produce optimized bytecode (.pyo)
• %py_requires - specifies a BuildRequires for the python interpreter and Requires (actually PreReq) for specific python series. Use %py_requires -d to also require python-devel
• %python_sitelib - site-packages directory for platform-independent modules. Expands to /usr/lib/python%{py_ver}/site-packages
• %python_sitearch - site-packages directory for platform-dependent modules. Expands to %{_libdir}/python%{py_ver}/site-packages

Compatibility with older distributions

In 11.1, split python package was introduced. If you want to build a package for older distribution, you cannot require python-base. Note that python-base was introduced because it has almost no build-time or run-time dependencies. That means that it is unblocked sooner in the rebuild cycle, and conversely, if your package doesn't buildrequire python, it can be build sooner.

The possibility to build noarch packages, along with %python_sitelib and %python_sitearch, was introduced in 11.2. If you need compatibility with older distributions, you must define the macros you are using. Place this at the start of your spec:

%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
%{!?python_sitearch: %global python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}

If your package is noarch, you must wrap the noarch declaration in a conditional sequence.

This would build your package as noarch only on openSUSE 11.2 and higher:

%if 0%{?suse_version} >= 1120
BuildArchitectures: noarch
%endif

And this would build your package as noarch on openSUSE 11.2 and higher, plus on any non-SUSE distro. (Useful when building packages for Fedora)

%if %{?suse_version: %{suse_version} > 1110} %{!?suse_version:1}
BuildArchitectures: noarch
%endif

Hints on how to Package Python3 modules

This text is work in progress. See the thread on the opensuse-packaging mailinglist for details. Use at your own risk. The hints listed in this section are subject to change in the future.

The same principles applies for Python3 modules. There are some changes though:

Building Python2 And Python3 Modules together

• Python 2.6 is still default
• Python runtime packages provide Python(abi) symbols (in versions 2.6 and 3.1 at the moment), modules require the respective symbols (via rpm auto-dependency-generator)
• To build a package containing pythonX module you need to add pythonX-devel to BuildRequires (python2-devel/python3-devel or both when package provides module for both versions)

If your Python source code is compatible for both versions, use the following procedure:

1. Start with python-foo.
2. Create python-foo.spec.
3. Build it and fix any errors.
4. Create a new project with osc linkpac:
   osc linkpac home:YOUR_REPO python-foo home:YOUR_REPO python3-foo
5. Create python3-foo.spec in the new project.
6. Apply any patches you may have for python3.

See openSUSE:Python 3 Status for a list of Python packages and their Python3 status on openSUSE.