Python Roadmap

Status

• Created: 2005-10-25 by MatthiasKloseBR

• People: MatthiasKloseBR

• Packages: python*BR

Introduction

Support different Python implementations and different versions of CPython (needed, when not all software can run with the latest/default python version when an Ubuntu release is going to happen).

Rationale

• The current approach to Python packaging allows different python versions to coexist on the system, but adds more packages to the system than needed. Forward conflicts in the python default/dependency packages make it hard to upgrade to the next python upstream version.
• A new CPython release is incorporated into Ubuntu: all existing pure Python modules should be made available to it, and native code modules built for it, within a short time (or immediately)
• A new Python implementation is added to the system: all installed modules (provided they are compatible with this new version) are available without user interaction.

Scope and Use Cases

• A new python module is installed. It is made available for all installed compatible python versions/implementations.
• A new python version/implementation is installed. All installed modules are byte-compiled for this version.
• A new python version becomes the default. All python modules installed outside of sys.path are byte-compiled for the new python version.
• A python extension has to be rebuilt for a new python version / dropped for an old python version. Do this with no or minimal changes in the source package.

Implementation Plan

Module packaging

• For architecture independent packages:
  • build one binary package and specify which python implementations it works with
  • TODO later: if different versions are required to run with packaged python implementations, package different versions of the module in different source packages.
    • Example: the latest version of foo uses generator expressions, which cannot be used with jython.
• Architecture-dependent packages:
  • Small extension modules (.so file smaller than some threshold value, 50k?):
    • The extensions for all compatible python implementations/versions are put into the same binary package.
  • Large extension modules:
    • Do as before: put them in a separate binary package (python2.y-foo), add a dependency package.
• Add two packages 'python-all' and 'python-all-dev'.
  • These packages depend on all supported runtime / development versions supported by the distribution. Use these packages as build dependencies for python modules, to determine for which python implementations/versions a module package is supposed to build. Using this information, a Python module package can extract the default Python version, and all Python versions it needs to build for, from the build environment. When generating the control file, the module's package does not need to be modified when the default Python version is changed or support for a new version and/or implementation is added.
• Add a new field to the control file for binary packages, X-Python-Package. The field should list all python versions which the module/extension is supposed to install to.
  • A special value of "all" causes installation to all Python versions (so that architecture-independent modules do not have to be re-uploaded when a new Python implementation/version enters the distribution). We have to use a new field, because we cannot determine all installed python modules when installing a new Python version.
• Provide a template (dh_make, debhelper) to package a Python module / a python application.

Python Registry

• The registry provides a set of scripts used in the installation and removal processes of packages.
• Python modules are installed in a directory tree which is managed by the registry.
  • No directory of the registry is part of sys.path in any python implementation.
• On package installation: modules are made available to every compatible installed Python implementation on the system (technically: copy the directory tree and symlink every file).
  • The files are then byte-compiled.
• On package removal: remove files/directories created during installation.
  • Delete byte-compiled files.
• When installing a new python implementation, all installed python modules are made available to it.
  • Then, byte-compile the files.

Pre-Work

An implementation for the registry supports handling of new modules and new python versions.