##(see the SpecSpec for an explanation)

 * '''Launchpad Entry''': UbuntuSpec:desktop-karmic-automagic-python-build-system
 * '''Created''': 2009-05-23
 * '''Contributors''': MartinPitt, RickSpencer
 * '''Packages affected''': `python-distutils-extra`, `quickly`

== Summary ==

In this blueprint we propose extending python-distutils-extra by a "do what I
mean" mode where it automatically chooses the right destination path and action
for various known file types such as Python modules, exectuable scripts, icons,
desktop files, etc. The goal is that the typical setup.py only has to
contain meta information like description, version, and author.

We also discuss Debian packaging generation.

== Release Note ==

Ubuntu 9.10 now provides a fully automatic Python distutils based build system
for Python applications in the `python-distutils-extra` package. Following a
"convention over configuration" approach, this automatically determines most
build and install actions for gettext, GtkBuilder, Qt UI, Python
packages/modules, etc. For simple cases, `setup.py` only needs to contain
package metadata, and `POTFILES.in`, `setup.cfg`, and `MANIFEST.in` are not
required at all.

== Rationale ==

Python's native and very popular build systems is "distutils" which comes
shipped with Python itself. It is fairly limited, though, and requires the user
to learn a lot about build systems. python-distutils-extra adds some
functionality for i18n, but there are still some standard use cases missing.

With our goal of supporting the opportunistic programmer and our pre-selection
of technologies, a lot of the build system design can be made implicit
("convention over configuration"), which will greatly reduce the amount of
learning necessary for installing a project and building a package.

== User stories ==

=== Jockey ===
The current Jockey project needs a very large and redundant
[[http://bazaar.launchpad.net/%7Ejockey-hackers/jockey/trunk/annotate/548/setup.py|setup.py]],
[[http://bazaar.launchpad.net/%7Ejockey-hackers/jockey/trunk/annotate/548/setup.cfg|setup.cfg]], and
[[http://bazaar.launchpad.net/%7Ejockey-hackers/jockey/trunk/annotate/548/POTFILES.in|POTFILES.in]], and
[[http://bazaar.launchpad.net/%7Ejockey-hackers/jockey/trunk/annotate/548/MANIFEST.in|MANIFEST.in]].

With the improved distutils-extras, `setup.cfg`, `POTFILES.in`, and `MANIFEST.in` can be dropped completely, and `setup.py` can be dropped to the metadata:

{{{
from DistUtilsExtra.auto import setup

setup(
    name='jockey',
    version='0.5.1',
    description='UI for managing third-party and non-free drivers',
    url='https://launchpad.net/jockey',
    license='GPL v2 or later',
    author='Martin Pitt',
    author_email='martin.pitt@ubuntu.com',
)
}}}

== Assumptions ==

 * Uses standard PyGTK and Python libraries and technologies.
 * Automatically generated Debian package produces single binary only. No library/public package building.

== Scope ==

This project is by and large an upstream build system extension. It does not
deal in any way with distribution, code hosting, revision control, Launchpad,
etc. These need to be handled in a higher level in the stack.

== Design ==

=== Project structure ===

 * All directories with `__init__.py` are Python packages which are used and shipped.
 * Most files are identified by extension and/or contents, not by location.
 * Use conventions for locations of files where it is ambiguous whether to install them, and where (such as executable scripts).

=== setup.py ===

For a standard project which uses well-known file types and a standard
directory layout, `setup.py` should only contain metadata about the project
which cannot be inferred from the source code.

Required fields:

 * name
 * version
 * license
 * description
 * author

Optional fields:

 * author_email
 * url (homepage)

=== sdist ===

`DistUtilsExtra.auto` already knows which files in the source tree are "source" and which ones merely build products/noise. This can be turned into a reasonable implicit default `MANIFEST.in`. An existing file supersedes the automatic implicit one.

=== Debian packaging ===

 * fully automatically generated from scratch, working, Policy compliant

== Implementation ==

=== distutils-extra extensions ===

Current python-distutils-extra needs to learn about compiling PyKDE .ui files
to .py files with `pykdeuic4`.

The existing functionality in distutils should be proposed upstream as a PEP and ideally merged into standard distutils gradually.

=== distutils-extra automatic setup.py ===

`DistUtilsExtra.auto` will support the following file types and automatically determine the default values in `setup.py`:
 
 * Python packages (directory with `__init__.py`)

 * D-Bus configuration (`*.conf` with magic string)

 * D-Bus service (`*.service` with magic string)
  * If it has an `User=` line → system service, otherwise session service

 * `.desktop.in`

 * `.policy.in`

 * `.notifyrc.in`

 * `*.po`, build `*.pot`: already provided by distutils-extras

 * Icons in `data/icons/size/category/*.{png,svg}`: already provided by distutils-extras

 * `po/POTFILES.in`

 * `cmdclass` (default to distutils-extra classes)

 * supplementary data files: `data/foo` → `/usr/share/`''project''`/foo`

 * scripts: all in `./bin/*` and `./`''projectname'' which are executable

`DistUtilsExtra.auto`'s goals of "convention over configuration" conflicts with
the upstream paradigm of "explicit is better than implicit", and thus will most
likely not be considered upstream. We will maintain this project in
Debian/Ubuntu for now.

=== Debian packaging ===

Based on the information in `setup.py` and the source code, we can also
generate a working Debian packaging.

The [[http://pypi.python.org/pypi/van.pydeb/|van.pydeb]] project 
([[http://packages.vanguardistas.net/public/pool/main/v/van.pydeb/|packages]],
[[http://svn.debian.org/viewsvn/pkg-zope/van.pydeb/trunk/debian/|svn]])
[already provides a some mechanics, so this should be used by
distutils-extra if appropriate.

`python-distutils-extra` will provide a new command `python-mkdebian` which generates a `debian/` directory with the necessary files from the information contained in the Python `.egg-info`:

==== changelog ====

 * project name, version, author, email from setup.py
 * description: "new release"
 * distro target: needs argument from environment; default to `lsb_release -c`?
 * If already exists, will just add a new changelog entry for a new version.

==== compat ====

 * constant (6 for hardy support)
 * Do not change if already existing.

==== rules ====

 * static, cdbs+python-distutils.mk
 * Do not change if already existing.

==== copyright ====

 * Grep source for "([cC])" statements and add them.
 * Support some standard values of "license" and add GPL stub and author name.
 * Do not change if already existing.

==== control ====

 * no dh_install, upstream build system DTRT
 * Source, Package, Maintainer, Description: from setup.py
 * Section, Priority, Standards-Version, XS-Python-Version: constant
 * Build-Depends: static (no tests), or binary depends (if test cases available)
 * Depends: grep all *.py for import statements, check which package provides them
   * `./setup.py` can also specify them explicitly with the `requires` keyword
   * Debian specific code needs to find out which package ships it; this is a heuristic, but should get it right in 80% of the cases at least
   * The Debian Python Modules Team has a [[http://svn.debian.org/viewsvn/python-modules/tools/|couple of scripts]] to do this
 * If already existing, update build/binary dependencies.

== Test/Demo Plan ==

`python-distutils-extra` contains an automatic test suite which covers all
functionality of the `DistUtilsExtra.auto` automatic build system. It is run on
package build.

For a manual test, check out `lp:jockey` (upstream trunk), and confirm that it
does not have any of `setup.cfg`, `MANIFEST.in`, `POTFILES.in`, and only a very
minimal `setup.py`. Test the following:

 * `./setup.py install --root=/tmp/x` does a complete and working installation
 * `./setup.py sdist` produces a complete source tarball
 * `python-mkdebian` produces a working `debian/` tree; calling `debuild` produces a working `.deb`, no lintian errors, and only trivial lintian warnings (just "binary without manpage" at the time of this writing)

----
CategorySpec