Launchpad Entry: karmic-qa-distribution-upgrade-testing
Created: UDS Karmic
Contributors: SteveBeattie
Packages affected: checkbox, upgrade script
Summary
Set up automated or semi-automated tests of distribution upgrades (i.e. Jaunty -> Karmic). Coordinate with Michael Vogt who already does some automated tests.
Rationale
Currently, most upgrade testing is done ad hoc, yet we recommend users upgrade between releases, rather than doing new installs. Yet we still have users that recommend to each other to do fresh installs. We should expand the amount of upgrade testing we do to catch as many upgrade issues as we can before our users hit them.
Design
Use and extend MichaelVogt's scripts from lp:update-manager/AutoUpgradeTester
- Needs polishing and packaging
- Preferably use kvm to give a more realistic upgrade environment
- schroot/ec2 cloud will likely have kernel version mismatch
Initial step is to upgrade a stock <version-1> desktop install
- Extend to include a wide variety of packages
- Perform live hardware upgrade tests using certification lab hardware
- Run some post-upgrade tests
- Environment sanity checks
- Functional system tests
- Desktop automation suite
Implementation
Upgrade Testing in Certification Environment
- Use the certification hardware to do upgrade testing
- Boot an image, perform upgrade, come back and check on status
- success/failure is pretty easy to detect (read exit status)
- more difficult to detect hang or a system that failed to boot, though this case is likely to be more rare than a failure with an exit status
- might want to do remote logging (syslogd)
- reporting should capture relevant log(s) as attachments to report results
- Run a KVM image upgrades on cert hardware; a good way to test KVM at the same time
Actions
- initially do upgrade testing on a single certification machine to check non-hardware-specific upgrade issues
- Steve and Michael to get method worked out
- coordinate with Ronald, Marc, and/or Dave to get a machine when ready
- Then increase number of machines used as this will help catch real hardware upgrade issues
- Cert team needs to improved test scheduling
- Add test duration to checkbox results
- depends on metric testing support in Checkbox; in the meantime we can potentially add this data as an attachment
Notes
Some tests are non-trivial to automate, like kernel changes between versions (e.g. i386->generic or PAE requirements). We will not try to cover all such corner cases with this first implementation.
There is a new Jaunty -> Karmic test upgrade option using AUFS which in itself requires testing, but is out of scope for this scpec