AutoDistUpgradeTestingSpec

Revision 4 as of 2006-10-30 12:58:08

Clear message

Summary

Automatic non-interactive testing to see if upgrades from the current to the next release work.

Rationale

During the development of the dist-upgrader it turned out that a lot of bugs were found by users (the hard way) that could have been found with automated testing.

Use cases

Package A overwrites files in package B without declaring a conflict. Package C has a failing postinst script on the dapper upgrade that only fails if the upgrade happens from the particular version in breezy.

Scope

The following class of bugs should be detected:

  • post-inst failing
  • file overwrite problems
  • bogus conffile prompts
  • dependency problems ($foo-desktop not installable/upgradable)
  • held-back packages

The following tests should be run:

  • {ubuntu,kubuntu,eduubuntu,xubuntu}-desktop upgrade (no other packages)
  • $foo-desktop + random selection of pacakges from main (with a configurable amount of random pkgs)
  • $foo-desktop + random selection of packages from main+universe
  • $foo-desktop + manual selected set of packages
  • $foo-desktop as base-install but with foo-desktop removed to see if the re-selection of missing meta-pkgs works

We should consider adding a feature to simulate a upgrade with a users setup. This would perform a non-interactive dist-upgrade in a chroot with the users settings (package selections+/etc) as the base of the setup. We could then ask users for real-world testing without risking broken systems.

Design

The DistUpgrade code in update-manager is used for doining the actual upgrade. A new non-interactive frontend is written that catches the above mentioned errors. Beside the automatic mode, there should be a way to quickly feed the application with a single package (or a selection of packages) to test upgradability of this particular set (quite useful to confirm bugreports).

A initial version should use $foo-edgy-desktop.tar.gz as the base for the upgrade-testing, then install (if required) additional packages and perform the upgrade. This test is done in a chroot with dpkg-diverted invoke-rc.d. The dist-upgrader is copied to the chroot and run there. After the upgrade the upgrade logs from $chroot/var/log/dist-upgrade/* are copied and stored. The result of the test is mailed or added to a web-page.

Implementation

Code

Some code was written in the http://people.ubuntu.com/~mvo/bzr/update-manager/non-interactive/ branch. This can be used as a basis for the automatic testing. It should be merged into the main-dist-upgrader and a proper cli frontend should be added.

Data preservation and migration

Outstanding issues

BoF agenda and discussion

Proper use of Xen and LVM might make this quite straightforward to administer. -iwj

CategorySpec