AutoDistUpgradeTestingSpec

Differences between revisions 6 and 7
⇤ ← Revision 6 as of 2006-11-09 22:35:18 →
Size: 4378
Editor: 207
Comment:
← Revision 7 as of 2006-11-09 22:38:45 → ⇥
Size: 4392
Editor: 207
Comment:
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
 * '''Packages affected''':  * '''Packages affected''': update-manager

Summary

Automatic non-interactive testing to see if upgrades from the current to the next release work.

Rationale

During the development of the ReleaseUpgrader it turned out that a lot of bugs were found by users (the hard way) that could have been found with automated testing.

Use cases

  • Package A overwrites files in package B without declaring a conflict.
  • Package C has a failing postinst script on the edgy upgrade that only fails if the upgrade happens from the particular version in dapper.
  • If the package D is installed the upgrade can't be calculated because it declares bad dependencies.

Scope

The following class of bugs should be detected:

  • post-inst failing
  • file overwrite problems
  • bogus conffile prompts
  • dependency problems ($foo-desktop not installable/upgradable)
  • held-back packages (e.g. xserver-xorg-driver-$foo)

We should consider adding a feature to simulate a upgrade with a users setup. This would perform a non-interactive dist-upgrade in a chroot with the users settings (package selections+/etc) as the base of the setup. We could then ask users for real-world testing without risking broken systems.

Design

The DistUpgrade code in update-manager is used for doining the actual upgrade. A new non-interactive frontend is written that catches the above mentioned errors. Beside the automatic mode, there should be a way to quickly feed the application with a single package (or a selection of packages) to test upgradability of this particular set (quite useful to confirm bugreports).

This test is done in a chroot with dpkg-diverted invoke-rc.d. First we build the edgy chroot and then it will automatically upgrade by coping the dist-upgrader to the chroot and run it there. After the upgrade the upgrade logs from $chroot/var/log/dist-upgrade/* are copied and stored. The result of the test is mailed or added to a web-page.

We need to test the following cases:

  1. {ubuntu,kubuntu,eduubuntu,xubuntu}-desktop upgrade (no other packages)
  2. server mode
  3. all of main (that we can possibly install, report what we can't install
    • in parallel)
  4. common combinations (based on popcon) - for later
  5. universe

We do this testing for every release architecture.

Before each test we run a simulation with a faked /var/lib/dpkg/status and simulate the upgrade and see how it goes. This is much much quicker than the actual upgrade and we can perform more tests for obscure combinations. We can do this using python-apt or aptitude -s.

We check for packages that were in main in edgy and installed but get removed by the upgrade (assuming that the removed packages are still in main for feisty). For the rare cases that this is not a bug we use a whitelist.

The test results will be mailed to a new testing mailing list. The results could also be send (via http POST) to the ScalableInstallTesting database.

Implementation

This will be implemented as a additional frontend to the ReleaseUpgrader + a tool that drivers this by building the chroot, copying the right files in place and mail the results. It will then be deployed on a machine in the datacenter were it will automatically run through a set of tests and mails the reports.

Code

Coding started in the http://people.ubuntu.com/~mvo/bzr/update-manager/non-interactive/ branch. It will be merged into the main dist-upgrader branch eventually.

Future work

The initial code uses a chroot to do the testing, but this has the disadvantage that we don't catch all error (e.g. because we have to divert some binaries like invoke-rc.d). So runing it inside XEN is probably a good thing for the future.

Additional tests for the future:

  • $foo-desktop + random selection of pacakges from main (with a configurable amount of random pkgs
  • $foo-desktop + random selection of packages from main+universe

  • iwj suggests that we randomly choose a uesr's PopularityContest data instead of just a random package

CategorySpec

AutoDistUpgradeTestingSpec (last edited 2008-08-06 16:31:07 by localhost)