AutoDistUpgradeTestingSpec

Differences between revisions 4 and 5
⇤ ← Revision 4 as of 2006-10-30 12:58:08 →
Size: 3125
Editor: p54A6716D
Comment: Fleshed more ideas out
← Revision 5 as of 2006-11-08 22:58:07 → ⇥
Size: 4671
Editor: 207
Comment: Notes from spec meeting
Deletions are marked like this. Additions are marked like this.
Line 15: Line 15:
Line 26: Line 27:
 * held-back packages  * held-back packages (e.g. xserver-xorg-
Line 31: Line 32:
  * iwj suggests that we randomly choose a uesr's PopularityContest data instead of just a random package.
Line 43: Line 45:
We need to test the following cases:
 1. ubuntu-desktop -> ubuntu-destkop (edubuntu,xubuntu,kubuntu)
 1. server mode
 1. all of main (that we can possibly install, report what we can't install
    in parallel)
 1. common combinations (based on popcon) - for later
 1. universe

We do this testing for every release architecture.

Before each test we run a simulation with a faked {{{/var/lib/dpkg/status}}} and simulate the upgrade and see how it goes. This is much much quicker than the actual upgrade and we can perform more tests for obscure combinations. We can do this using {{{python-apt}}} or {{{aptitude -s}}}.

Checking for packages that were in main before the upgrade and are still in main after the upgrade but are no longer installed after the upgrade. For the rare cases this hapens we use a whitelist.

The current code uses a chroot to do the testing, but this has the disadvantage
that we don't catch all error (e.g. because we have to divert some binaries
like invoke-rc.d). So runing it inside XEN is probably a good thing.

The test results will be mailed to a new testing mailing list. The results
could also be send (via http POST) to the ScalableInstallTesting database.
Line 44: Line 67:

The bits that needs to be implemented:
 1. cleanup of the non-interactive releaseupgrade frontend to make the log better
 1. add the different testcases
 1. deploy it into the datacenter

Summary

Automatic non-interactive testing to see if upgrades from the current to the next release work.

Rationale

During the development of the dist-upgrader it turned out that a lot of bugs were found by users (the hard way) that could have been found with automated testing.

Use cases

Package A overwrites files in package B without declaring a conflict. Package C has a failing postinst script on the dapper upgrade that only fails if the upgrade happens from the particular version in breezy.

Scope

The following class of bugs should be detected:

  • post-inst failing
  • file overwrite problems
  • bogus conffile prompts
  • dependency problems ($foo-desktop not installable/upgradable)
  • held-back packages (e.g. xserver-xorg-

The following tests should be run:

  • {ubuntu,kubuntu,eduubuntu,xubuntu}-desktop upgrade (no other packages)
  • $foo-desktop + random selection of pacakges from main (with a configurable amount of random pkgs)

    • iwj suggests that we randomly choose a uesr's PopularityContest data instead of just a random package.

  • $foo-desktop + random selection of packages from main+universe
  • $foo-desktop + manual selected set of packages
  • $foo-desktop as base-install but with foo-desktop removed to see if the re-selection of missing meta-pkgs works

We should consider adding a feature to simulate a upgrade with a users setup. This would perform a non-interactive dist-upgrade in a chroot with the users settings (package selections+/etc) as the base of the setup. We could then ask users for real-world testing without risking broken systems.

Design

The DistUpgrade code in update-manager is used for doining the actual upgrade. A new non-interactive frontend is written that catches the above mentioned errors. Beside the automatic mode, there should be a way to quickly feed the application with a single package (or a selection of packages) to test upgradability of this particular set (quite useful to confirm bugreports).

A initial version should use $foo-edgy-desktop.tar.gz as the base for the upgrade-testing, then install (if required) additional packages and perform the upgrade. This test is done in a chroot with dpkg-diverted invoke-rc.d. The dist-upgrader is copied to the chroot and run there. After the upgrade the upgrade logs from $chroot/var/log/dist-upgrade/* are copied and stored. The result of the test is mailed or added to a web-page.

We need to test the following cases:

  1. ubuntu-desktop -> ubuntu-destkop (edubuntu,xubuntu,kubuntu)

  2. server mode
  3. all of main (that we can possibly install, report what we can't install
    • in parallel)
  4. common combinations (based on popcon) - for later
  5. universe

We do this testing for every release architecture.

Before each test we run a simulation with a faked /var/lib/dpkg/status and simulate the upgrade and see how it goes. This is much much quicker than the actual upgrade and we can perform more tests for obscure combinations. We can do this using python-apt or aptitude -s.

Checking for packages that were in main before the upgrade and are still in main after the upgrade but are no longer installed after the upgrade. For the rare cases this hapens we use a whitelist.

The current code uses a chroot to do the testing, but this has the disadvantage that we don't catch all error (e.g. because we have to divert some binaries like invoke-rc.d). So runing it inside XEN is probably a good thing.

The test results will be mailed to a new testing mailing list. The results could also be send (via http POST) to the ScalableInstallTesting database.

Implementation

The bits that needs to be implemented:

  1. cleanup of the non-interactive releaseupgrade frontend to make the log better
  2. add the different testcases
  3. deploy it into the datacenter

Code

Some code was written in the http://people.ubuntu.com/~mvo/bzr/update-manager/non-interactive/ branch. This can be used as a basis for the automatic testing. It should be merged into the main-dist-upgrader and a proper cli frontend should be added.

Data preservation and migration

Outstanding issues

BoF agenda and discussion

Proper use of Xen and LVM might make this quite straightforward to administer. -iwj

CategorySpec

AutoDistUpgradeTestingSpec (last edited 2008-08-06 16:31:07 by localhost)