DistributedDevelopment/UnderTheHood/Importer

Importer

Revision 3 as of 2010-02-19 01:18:33

This page explains in some depth how the importer works, and some of the things that you may see it do.

Mainloop

The importer is basically just a while loop that looks at recent uploads to Debian and Ubuntu and triggers imports based on them (with several running in parallel). This check is run frequently, meaning that it should start on your upload a few minutes after you complete it. The process is also reasonably fast, so if the binaries are published and you don't see any change in bzr it is quite likely something went wrong (the build time + publication delay should normally greatly exceed the time taken by the importer).

To find out about uploads the importer uses Launchpad APIs. This means that for Debian we only see uploads when Launchpad imports them. This is done four times a day (same frequency as Debian's dinstall), and so it should only be a few hours on average from a Debian upload to it being available in bzr. Obviously if you are waiting for the upload for some reason this is too long. If this is a frequent occurrence then we can look at alternative solutions, so speak up if you are hit by this.

If there is nothing for the importer to work on (the usual case) then it will do integrity checks instead, making sure it is up to date and all the data is correct. This doesn't delay any imports though.

If the queue fills up (see below for how to find out), perhaps after an autosync run where there are a few hundred uploads to import, then there can be a delay of up to a few hours until the backlog is processed. If you are waiting for something that is stuck in that backlog then request an importer admin increase the priority of that package (you just need to tell them the package name).

Import job

When the mainloop triggers an import it acts on the whole history of the package at once. It looks at everything it should have imported and compares it to what it has. If there are any un-imported uploads it starts the process to import them to bzr. The import itself is based on "bzr import-dsc" from bzr-builddeb.

If there are any problems with the import it will record a failure for that package and put it on hold. This generally means that it won't act on that package until it has been reviewed by a human, so that it won't dig itself in to a whole repeatedly trying to do something that causes it to crash. There are currently a few hundred failures, so that is the usual reason why a package isn't imported, and we are working to reduce that number.

However, not all failures require human intervention. There are often transient problems, usually network issues, and for these the importer will retry the package after a couple of hours, which works around most issues. If you find that a package has failed with what looks like a network problem and it doesn't say that it will be automatically retried then speak to an importer admin and suggest that the package has a "spurious failure" and they can have a look and retry the package such that errors similar to that will be automatically retried in future.

However, there is a cap on automatic retries to prevent and endless loop if the problem appears to be transient but is being caused by the actions of the importer. In these cases human intervention is once again required.

Finding the status

That's great, but how do you find out what is going on?

You can start at the status overview page: http://package-import.ubuntu.com/status/ which lists the currently running jobs, the queue, and the failures.

Clicking on one of the package links for a failure will take you to a page showing the gory details. Feel free to ask for an interpretation if it is a package that you care about.

Importer admins

Who are the admins?

igc
jam
james_w
lifeless
poolie
spiv
vila

If you speak to one and they don't know how to deal with what you are asking them to do then see if the task is on DistributedDevelopment/UnderTheHood/Importer/Operational.

Hacking