AutoDistUpgradeTestingSpec

Differences between revisions 1 and 7 (spanning 6 versions)
⇤ ← Revision 1 as of 2006-06-08 07:54:09 →
Size: 1544
Editor: p54A6673E
Comment: initial creation
← Revision 7 as of 2006-11-09 22:38:45 → ⇥
Size: 4392
Editor: 207
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
 * '''Launchpad Entry''': https://launchpad.net/distros/ubuntu/+spec/xdeltas  * '''Launchpad Entry''': https://launchpad.net/distros/ubuntu/+spec/auto-dist-upgrade-testing
Line 6: Line 6:
 * '''Packages affected''':  * '''Packages affected''': update-manager
Line 10: Line 10:
Automatic testing if upgrades from the current to the next release work. Automatic non-interactive testing to see if upgrades from the current to the next release work.
Line 14: Line 14:
During the development of the dist-upgrader it turned out that a lot of bugs were found by users (the hard way) that could have been easily found with automated testing (post-inst failing, file overwrite problems). During the development of the ReleaseUpgrader it turned out that a lot of bugs were found by users (the hard way) that could have been found with automated testing.
Line 18: Line 18:
Package A overwrites files in package B without declaring a conflict. Package C has a failing postinst script on the dapper upgrade that only fails if the upgrade happens from the particular version in breezy.  * Package A overwrites files in package B without declaring a conflict.
 * Package C has a failing postinst script on the edgy upgrade that only fails if the upgrade happens from the particular version in dapper.
 * If the package D is installed the upgrade can't be calculated because it declares bad dependencies.
Line 22: Line 24:
Initially it should test upgradability of {ubuntu,kubuntu,xubuntu}-desktop, then most of main, then selected ranges of universe. Failures must be reported automatically via mail. The following class of bugs should be detected:
 * post-inst failing
 * file overwrite problems
 * bogus conffile prompts
 * dependency problems ($foo-desktop not installable/upgradable)
 * held-back packages (e.g. xserver-xorg-driver-$foo)

We should consider adding a feature to simulate a upgrade with a users setup. This would perform a non-interactive dist-upgrade in a chroot with the users settings (package selections+/etc) as the base of the setup. We could then ask users for real-world testing without risking broken systems.
Line 26: Line 35:
Use the dist-upgrader code for the basic functinality of updating the sources.list, doing the dist-ugprade and catching errors via a non-interactive frontend. The DistUpgrade code in update-manager is used for doining the actual upgrade. A new `non-interactive` frontend is written that catches the above mentioned errors. Beside the automatic mode, there should be a way to quickly feed the application with a single package (or a selection of packages) to test upgradability of this particular set (quite useful to confirm bugreports).

This test is done in a chroot with dpkg-diverted invoke-rc.d. First we build the edgy chroot and then it will automatically upgrade by coping the dist-upgrader to the chroot and run it there. After the upgrade the upgrade logs from $chroot/var/log/dist-upgrade/* are copied and stored. The result of the test is mailed or added to a web-page.

We need to test the following cases:
 1. {ubuntu,kubuntu,eduubuntu,xubuntu}-desktop upgrade (no other packages)
 1. server mode
 1. all of main (that we can possibly install, report what we can't install
    in parallel)
 1. common combinations (based on popcon) - for later
 1. universe

We do this testing for every release architecture.

Before each test we run a simulation with a faked {{{/var/lib/dpkg/status}}} and simulate the upgrade and see how it goes. This is much much quicker than the actual upgrade and we can perform more tests for obscure combinations. We can do this using {{{python-apt}}} or {{{aptitude -s}}}.

We check for packages that were in main in edgy and installed but get removed by the upgrade (assuming that the removed packages are still in main for feisty). For the rare cases that this is not a bug we use a whitelist.

The test results will be mailed to a new testing mailing list. The results
could also be send (via http POST) to the ScalableInstallTesting database.
Line 30: Line 58:
This will be implemented as a additional frontend to the ReleaseUpgrader + a tool that drivers this by building the chroot, copying the right files in place and mail the results. It will then be deployed on a machine in the datacenter were it will automatically run through a set of tests and mails the reports.
Line 32: Line 62:
Code was written for the breezy->dapper testing in the http://people.ubuntu.com/~mvo/bzr/update-manager/non-interactive/ branch. This can be used as a basis for the automatic testing. Coding started in the http://people.ubuntu.com/~mvo/bzr/update-manager/non-interactive/ branch. It will be merged into the main dist-upgrader branch eventually.
Line 34: Line 64:
=== Data preservation and migration ===
Line 36: Line 65:
== Outstanding issues == == Future work ==
Line 38: Line 67:
== BoF agenda and discussion == The initial code uses a chroot to do the testing, but this has the disadvantage
that we don't catch all error (e.g. because we have to divert some binaries
like invoke-rc.d). So runing it inside XEN is probably a good thing for the future.

Additional tests for the future:
 * $foo-desktop + random selection of pacakges from main (with a configurable amount of random pkgs
 * $foo-desktop + random selection of packages from main+universe
 * iwj suggests that we randomly choose a uesr's PopularityContest data instead of just a random package

Summary

Automatic non-interactive testing to see if upgrades from the current to the next release work.

Rationale

During the development of the ReleaseUpgrader it turned out that a lot of bugs were found by users (the hard way) that could have been found with automated testing.

Use cases

  • Package A overwrites files in package B without declaring a conflict.
  • Package C has a failing postinst script on the edgy upgrade that only fails if the upgrade happens from the particular version in dapper.
  • If the package D is installed the upgrade can't be calculated because it declares bad dependencies.

Scope

The following class of bugs should be detected:

  • post-inst failing
  • file overwrite problems
  • bogus conffile prompts
  • dependency problems ($foo-desktop not installable/upgradable)
  • held-back packages (e.g. xserver-xorg-driver-$foo)

We should consider adding a feature to simulate a upgrade with a users setup. This would perform a non-interactive dist-upgrade in a chroot with the users settings (package selections+/etc) as the base of the setup. We could then ask users for real-world testing without risking broken systems.

Design

The DistUpgrade code in update-manager is used for doining the actual upgrade. A new non-interactive frontend is written that catches the above mentioned errors. Beside the automatic mode, there should be a way to quickly feed the application with a single package (or a selection of packages) to test upgradability of this particular set (quite useful to confirm bugreports).

This test is done in a chroot with dpkg-diverted invoke-rc.d. First we build the edgy chroot and then it will automatically upgrade by coping the dist-upgrader to the chroot and run it there. After the upgrade the upgrade logs from $chroot/var/log/dist-upgrade/* are copied and stored. The result of the test is mailed or added to a web-page.

We need to test the following cases:

  1. {ubuntu,kubuntu,eduubuntu,xubuntu}-desktop upgrade (no other packages)
  2. server mode
  3. all of main (that we can possibly install, report what we can't install
    • in parallel)
  4. common combinations (based on popcon) - for later
  5. universe

We do this testing for every release architecture.

Before each test we run a simulation with a faked /var/lib/dpkg/status and simulate the upgrade and see how it goes. This is much much quicker than the actual upgrade and we can perform more tests for obscure combinations. We can do this using python-apt or aptitude -s.

We check for packages that were in main in edgy and installed but get removed by the upgrade (assuming that the removed packages are still in main for feisty). For the rare cases that this is not a bug we use a whitelist.

The test results will be mailed to a new testing mailing list. The results could also be send (via http POST) to the ScalableInstallTesting database.

Implementation

This will be implemented as a additional frontend to the ReleaseUpgrader + a tool that drivers this by building the chroot, copying the right files in place and mail the results. It will then be deployed on a machine in the datacenter were it will automatically run through a set of tests and mails the reports.

Code

Coding started in the http://people.ubuntu.com/~mvo/bzr/update-manager/non-interactive/ branch. It will be merged into the main dist-upgrader branch eventually.

Future work

The initial code uses a chroot to do the testing, but this has the disadvantage that we don't catch all error (e.g. because we have to divert some binaries like invoke-rc.d). So runing it inside XEN is probably a good thing for the future.

Additional tests for the future:

  • $foo-desktop + random selection of pacakges from main (with a configurable amount of random pkgs
  • $foo-desktop + random selection of packages from main+universe

  • iwj suggests that we randomly choose a uesr's PopularityContest data instead of just a random package

CategorySpec

AutoDistUpgradeTestingSpec (last edited 2008-08-06 16:31:07 by localhost)