MultiMachineTestingInfraSpec

Revision 2 as of 2006-06-22 16:38:49

Clear message

Summary

Rationale

Inside Ubuntu, there are plenty of automated test systems. From the unit tests within make check to piuparts, these tests all check for poor implementations. However, all of these systems only test the package itself. There is no facility for higher-level testing.

Since integration testing cannot be automated, it must be done by hand. As Ubuntu gets bigger, with more packages and more supported platforms, this becomes more and more infeasible.

User stories

  • Andrew tests whether Apache works by installing the package, setting up a dynamic website, and using it through his web browser. He wants to write an automated test script that does this all for him.
  • Biella wants to test file sharing between her Ubuntu server and her Windows workstation. She also wants to write an automated test script that does this.
  • Charles wants to test that some new machines work well with Ubuntu. He wants to run the entire test suite on this new hardware.
  • Dora is a system administrator at a university which makes an Ubuntu-derived distribution. She wants to run the entire test suite against a test lab of computers, whenever she rolls out a new version of her distro.
  • Eric is an Ubuntu developer that has made a large library transition. He wants to confirm that he hasn't introduced any regressions into the distro before uploading his new packages.
  • Fiona is a system administrator that wants to help Ubuntu by donating hardware resources. These should be added to the resource pool for the automated test system.
  • Greg is a release manager for an Ubuntu-derived distribution. He looks at a nightly report generated by this system to determine that no regressions have been introduced.
  • Hania is a porter who wants to port Ubuntu from one architecture to the next. She wants to get a nightly report of which features still need work for this new architecture.
  • Ivan sees fixes a bug and wants to prevent it from regressing. He writes a test script and uploads it into the testing infrastructure, as part of uploading his fix.
  • Joanna notices that bugs keep on cropping up in a certain area that she's interested in. She'll write a suite of test scripts that exercise this functionality, some of which may fail, and uploads them into the infrastructure.
  • Kyle wants to query the system for failing tests and tie these results directly into Malone.

Scope

Design

Implementation

Code

Data preservation and migration

Outstanding issues

See also

http://mkdump.sourceforge.net/

CategorySpec