AutoDistUpgradeTestingSpec

Differences between revisions 3 and 4
⇤ ← Revision 3 as of 2006-10-30 10:27:44 →
Size: 1921
Editor: p54A66523
Comment:
← Revision 4 as of 2006-10-30 12:58:08 → ⇥
Size: 3125
Editor: p54A6716D
Comment: Fleshed more ideas out
Deletions are marked like this. Additions are marked like this.
Line 14: Line 14:
During the development of the dist-upgrader it turned out that a lot of bugs were found by users (the hard way) that could have been easily found with automated testing (post-inst failing, file overwrite problems, bogus conffile prompts, dependency problems, held-back packages).
 During the development of the dist-upgrader it turned out that a lot of bugs were found by users (the hard way) that could have been found with automated testing.
Line 22: Line 21:
Initially it should test upgradability of {ubuntu,kubuntu,xubuntu}-desktop, then most of main, then selected ranges of universe. Failures must be reported automatically via mail. The following class of bugs should be detected:
 * post-inst failing
 * file overwrite problems
 * bogus conffile prompts
 * dependency problems ($foo-desktop not installable/upgradable)
 * held-back packages

The following tests should be run:
 * {ubuntu,kubuntu,eduubuntu,xubuntu}-desktop upgrade (no other packages)
 * $foo-desktop + random selection of pacakges from main (with a configurable amount of random pkgs)
 * $foo-desktop + random selection of packages from main+universe
 * $foo-desktop + manual selected set of packages
 * $foo-desktop as base-install but with foo-desktop removed to see if the re-selection of missing meta-pkgs works

We should consider adding a feature to simulate a upgrade with a users setup. This would perform a non-interactive dist-upgrade in a chroot with the users settings (package selections+/etc) as the base of the setup. We could then ask users for real-world testing without risking broken systems.
Line 26: Line 39:
Use the dist-upgrader code for the basic functinality of updating the sources.list, doing the dist-ugprade and catching errors via a non-interactive frontend. Beside the automatic mode, there should be a way to quickly feed the application with a single package (or a selection of packages) to test upgradability of this particular set (quite useful to confirm bugreports). The DistUpgrade code in update-manager is used for doining the actual upgrade. A new `non-interactive` frontend is written that catches the above mentioned errors. Beside the automatic mode, there should be a way to quickly feed the application with a single package (or a selection of packages) to test upgradability of this particular set (quite useful to confirm bugreports).

A initial version should use $foo-edgy-desktop.tar.gz as the base for the upgrade-testing, then install (if required) additional packages and perform the upgrade. This test is done in a chroot with dpkg-diverted invoke-rc.d. The dist-upgrader is copied to the chroot and run there. After the upgrade the upgrade logs from $chroot/var/log/dist-upgrade/* are copied and stored. The result of the test is mailed or added to a web-page.
Line 32: Line 47:
Code was written in the http://people.ubuntu.com/~mvo/bzr/update-manager/non-interactive/ branch. This can be used as a basis for the automatic testing. Some code was written in the http://people.ubuntu.com/~mvo/bzr/update-manager/non-interactive/ branch. This can be used as a basis for the automatic testing. It should be merged into the main-dist-upgrader and a proper cli frontend should be added.

Summary

Automatic non-interactive testing to see if upgrades from the current to the next release work.

Rationale

During the development of the dist-upgrader it turned out that a lot of bugs were found by users (the hard way) that could have been found with automated testing.

Use cases

Package A overwrites files in package B without declaring a conflict. Package C has a failing postinst script on the dapper upgrade that only fails if the upgrade happens from the particular version in breezy.

Scope

The following class of bugs should be detected:

  • post-inst failing
  • file overwrite problems
  • bogus conffile prompts
  • dependency problems ($foo-desktop not installable/upgradable)
  • held-back packages

The following tests should be run:

  • {ubuntu,kubuntu,eduubuntu,xubuntu}-desktop upgrade (no other packages)
  • $foo-desktop + random selection of pacakges from main (with a configurable amount of random pkgs)
  • $foo-desktop + random selection of packages from main+universe
  • $foo-desktop + manual selected set of packages
  • $foo-desktop as base-install but with foo-desktop removed to see if the re-selection of missing meta-pkgs works

We should consider adding a feature to simulate a upgrade with a users setup. This would perform a non-interactive dist-upgrade in a chroot with the users settings (package selections+/etc) as the base of the setup. We could then ask users for real-world testing without risking broken systems.

Design

The DistUpgrade code in update-manager is used for doining the actual upgrade. A new non-interactive frontend is written that catches the above mentioned errors. Beside the automatic mode, there should be a way to quickly feed the application with a single package (or a selection of packages) to test upgradability of this particular set (quite useful to confirm bugreports).

A initial version should use $foo-edgy-desktop.tar.gz as the base for the upgrade-testing, then install (if required) additional packages and perform the upgrade. This test is done in a chroot with dpkg-diverted invoke-rc.d. The dist-upgrader is copied to the chroot and run there. After the upgrade the upgrade logs from $chroot/var/log/dist-upgrade/* are copied and stored. The result of the test is mailed or added to a web-page.

Implementation

Code

Some code was written in the http://people.ubuntu.com/~mvo/bzr/update-manager/non-interactive/ branch. This can be used as a basis for the automatic testing. It should be merged into the main-dist-upgrader and a proper cli frontend should be added.

Data preservation and migration

Outstanding issues

BoF agenda and discussion

Proper use of Xen and LVM might make this quite straightforward to administer. -iwj

CategorySpec

AutoDistUpgradeTestingSpec (last edited 2008-08-06 16:31:07 by localhost)