##(see the SpecSpec for an explanation) * '''Launchpad Entry''': UbuntuSpec:foundations-lucid-robust-python-packaging * '''Created''': MichaelVogt * '''Contributors''': * '''Packages affected''': python-central == Summary == We do a lot of magic currently in postinst/preinst scripts for Python packaging. This has caused problems before. It is also hard to debug: unpack/configure ordering makes it hard to reproduce. Users have a hard time diagnosing, too. == Release Note == The python packaging system is now more robust, easier to inspect and more user-friendly. == Rationale == Design guidelines for the proposed changes: * do building/compiling on the buildds * catch errors on the buildds (and not on the user machines) * let dpkg do the shuffling of files * let dpkg do (file) conflicts checking * as little complexity in the maintainer scripts as possible * don't force people learn new tools if something is wrong (dpkg -L python-foo should be enough) * as little work for the archive-admins as possible * python application keep working during unpack/configure == User stories == Jane perform a upgrade from 8.10 to 9.04 and wants to play a game during the (boring) upgrade. This does not work because right now and she is disappointed. With the 9.04 to 9.10 upgrade it does work and she is much happier. == Design == There are currently two python packaging system implementations. For pycentral we need to tweak the way its used. For python-support we need to start a dialog with the maintainer and see if we can agree on a good way forward. In detail: * python-central: use DH_PYCENTRAL=include-links by default (downside: pkg needs a rebuild on a new python version) * python-support: talk to python-support upstream about a patch that has a similar as semantic DH_PYCENTRAL=include-links (and provide that patch if needed) * tackle the problem of relative imports (and ask doko to provide more details) * tackle the problem that pysupport adds a additional distro site-packages dir (and find out the details why that is problematic) * clarify what supported, old-versions and unsupported in /usr/share/python/debian_defaults means == Implementation == For python-central the default is switched to "include-links". No further code changes are required. We need to also add (regression)tests for "pycentral pkginstall foo" and "pycentral pkgremove foo". For python-support we need to start a dialog with the upstream maintainer and suggest to have "include-links" like semantics by default. With the new karmic version of python-support packages will keep working between the --unpack and --configure phase. === Migration === The migration will happen automatically as python packages get rebuild. For each new python major version (e.g. 2.7) a no-changes rebuild is required with the include-links schema. == Test/Demo Plan == Testing this change should be part of the regular upgrade testing process. We need to check for upgrade failures (that may be caused by this) and do some functional testing after the upgrade. == BoF agenda and discussion == === mvo === ==== NOTES ==== * keep in mind to be in sync with debian * debian uses python-support ==== history why we have the current policy ==== * rebuild packages from source when python version changed * plus the packaging had to be changed in debian/control (python2.5-foo) on new python versions * between versions python byte code changes * C ABI changes with new versions of python * we used to have "python-foo" source package, that created python2.4-foo, python2.5-foo * rebuilds required when new python is released or dropped * archive-admins had to process those packages in the NEW queue * (debian-release-managers had problems with the testing distro) * GOAL: get rid of version information in the package source (i.e. apps just depend on python-foo instead of python2.4-foo) * simplify the job of the resolver (because there is just python-foo not python2.3-foo -> python2.4-foo) * some of the original requirements (problems with NEW processing, testing distro, ...) have disappreared ==== problem space ==== * keep packages working during the upgrade - symlinks got created during preinst so that they were available during unpack (around ~interpid) - problematic for: update-manager, apport, hplip, python cron jobs - no longer used * in intrepid we have DH_PYCOMPAT=nomove (multipiles the size of the package by the number of python versions) * in jaunty we have DH_PYCOMAPT=include-links (just include the symlinks, not the .pyc files) ==== problems ==== * two packaging helper tools (python-support, python-central) * python-gtk (extension) depends on python-gobject (extension) - now if python-gtk builds against 2.4,2.5 but python-gobject just builds against 2.4 then python-gtk breaks, but the dependencies still correct - python-gobject uses: Provides: python2.5-gobject, python2.6-gobject - and python-gtk depends on python2.6-gobject one solution is to do the uploads in reverse order, but apparently this is not understood and no common practice. * when adding a new version: upload new python, rebuild packages with dependency chain order bottom up * when removing a version: rebuild packages from top down without the version and then remove the version * going back python2.X-foo might be problematic for the ftp-master team (NEW queue processing overhead, soyuz overwrite handling, debian/control needs to be touched) - all of this could be automated * RELATIVE IMPORTS ==== possible solutions/improvements ==== * include DH_PYCENTRAL=include-links by default - requires rebuild when a new python version comes out (or one runs out of support) - just changelog entry change is enough - does not work with python-support so far (needs discussion with python-support upstream) - *.pyc building is still done in maintainer scripts ==== action items ==== * use DH_PYCENTRAL=include-links by default (downside: pkg needs a rebuild on a new python version; slangasek objected to this, probably should be only done for packages in main/on the CD) * talk to python-support about a patch that has a similar as semantic DH_PYCENTRAL=include-links * tackle the problem of relative imports (and ask doko to provide more details) - pysupport adds a additional distro site-packages dir that needs to be dropped relative imports break, if a python package (in the python sense) is packaged with different packaging helpers, resulting in files splitted on two different locations on sys.path. To avoid this situation all public modules must be installed in one location. ==== liw ==== We do a lot of magic currently in postinst/preinst scripts for Python packaging. This has caused problems before. It is also hard to debug: unpack/configure ordering makes it hard to reproduce. Users have a hard time diagnosing, too. mvo would like to simplify Python packaging. Recap of why the current code is in place (to avoid flip-flopping between problems): - in intrepid: tried to make time when packages are non-working as short as possible: create symlinks in preinst script, so they're there when dpkg unpacks package (preinst is run almost immediately before unpacking) giving python modules and extensions being immediately available; this is useful so that packages like update-manager and other packages used during an upgrade are usable - hplip, apport can also break during an upgrade - originally (before Ubuntu!) Debian had to rebuild python modules when a new python version was added to the archive - also required changing the source package (at least debian/control) - python is ugly when it comes to (system) upgrades - upstream changes bytecode format and other things between releases - abi for extensions - thus, have to rebuild with new python release - also, when dropping a python release from a release - we packaged a module as python2.2-foo - sabdfl always listens! - 2006: the goal was to get remove all hardcoded python versions in the source package - also removing around some processing in Debian and Ubuntu (NEW queue) - also simplified apt's computation for upgrades - Debian now no longer has problem with packages moving into testing, or NEW queue, but other problems - there is no real formal Python policy that applies to this time/space continuum - building for old and new versions of python eases transitions to Debian testing but also for users dist-upgrading - impractical to have newest (default?) python version from previous (LTS) release in new release - creating symlinks in preinst is rejected by Debian and is pretty fragile - in debconf6 @ mexico some design was whiteboarded which used depends+provides - is this current policy? - slangasek: an app that has to depend on non-default version of python should depend on the modules it requires (pythonX.Y-foo), but only apps, not libs/modules/extensions - rejected by Manoj @ Debian - some controversy about this, in the room and in Debian - I'm quite confused by this discussion - few actual improvements to the situation, we ran out of time - using the same package names for real and virtual packages causes problems - but splitting name spaces causes problems too Python-team in Debian has (practically) decided to standardize on python-support, and to avoid python-central. proposal for just one packaging tool; emerged from discussion at Debconf9: http://lists.debian.org/debian-python/2009/08/msg00003.html ---- CategorySpec