ReleaseValidationProcess

Differences between revisions 4 and 11 (spanning 7 versions)
⇤ ← Revision 4 as of 2007-06-26 16:47:03 →
Size: 4176
Editor: 82-69-40-219
Comment: fix link
← Revision 11 as of 2012-09-03 17:52:40 → ⇥
Size: 3563
Editor: cpe-66-69-240-108
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
To be carried out by: Henrik Nilsen Omma To be carried out by: Ubuntu QA Team Lead
Line 3: Line 3:
Goals: == Goals ==
Line 5: Line 5:
 * This process should apply to betas, [:ReleaseCandidate:ReleaseCandidates], final releases, and milestones  * This process should apply to betas, [[ReleaseCandidate|ReleaseCandidates]], final releases, and milestones
Line 7: Line 7:
T minus 14 days: == T minus 14 days ==
Line 12: Line 12:
  * Review installation test cases with Colin Watson   * Review installation test cases with Evan Dandrea
Line 15: Line 15:
 1. Remind the Canonical distro team to update [https://wiki.canonical.com/StaffHardware StaffHardware]  1. Remind the Canonical distro team to update [[https://wiki.canonical.com/StaffHardware|StaffHardware]]
Line 27: Line 27:
T minus 7 days: == T minus 7 days ==
Line 30: Line 30:
 1. Populate and prioritize test matrix with all test cases and assign testers  1. Populate and prioritize [[https://iso.qa.ubuntu.com/|test tracker]] with all test cases and assign testers
Line 39: Line 39:
T minus 4 days: == T minus 4 days ==
Line 46: Line 46:
 1 Generate tracker bugs in the [https://bugs.launchpad.net/ubuntu-iso-tests/+bugs ISO testing tracker]  1. Set current milestone in the tracker and ensure that the candidate builds are listed
Line 49: Line 49:
 1. Collect test reports and bug references, notifying the release manager of bugs as they are reported  1. Monitor test reports and bug references, notifying the release manager of bugs as they are reported
Line 52: Line 52:
== Draft process for tracking test results == == Tracking test results ==
Line 54: Line 54:
 1 Generate tracker bugs in the [https://bugs.launchpad.net/ubuntu-iso-tests/+bugs ISO testing tracker]
  * For full validation: one per ISO per test case (all cases need to be tested)
  * For minimal validation: one per ISO which needs to be tested (any one of the installation test plans should be sufficient)
  * Tracking bug contains build number, link to test plan(s), instructions for reporting results		  1. Post candidate ISOs and test cases in the tracker
Line 59: Line 56:
  * [http://www.ubuntuforums.org/forumdisplay.php?f=201 ISO testing forum]
  * [http://lists.ubuntu.com/mailman/listinfo/ubuntu-devel ubuntu-devel]
  * XXX: consider making a testing-team-announce list for this purpose as well		   * [[http://www.ubuntuforums.org/forumdisplay.php?f=201|ISO testing forum]]
  * [[http://lists.ubuntu.com/mailman/listinfo/ubuntu-devel|ubuntu-devel]]
  * tracker mass email feature (not yet implemented)
Line 63: Line 60:
  * Testers subscribe to the appropriate tracking bug for any test case they plan to perform (XXX: can we auto-subscribe them for the major releases given, that they are already required to sign up by email?)
 1. Monitor tracking bugs
  * Ensure that separate bugs are filed by testers for genuine problems
  * Update bug status to reflect testing status:
   * ''Unconfirmed'' - newly filed, untested image
   * ''In progress'' - hs received at least one PASS
   * ''Rejected'' - Has failed, build new image
   * ''Fix Released'' - Milestone released		  1. Monitor tracker results
  * Ensure that bugs are filed in Launchpad by testers for genuine problems
Line 72: Line 63:
  * For minimal validation: at least one successful installation per ISO is sufficient
  * For full validation: all test cases must be successful
 1. Notify release manager of test results
  * XXX: How? via the tracker bug status or does a person do this via wiki/IRC		   * For minimal validation (alphas): at least one successful installation per ISO is sufficient
  * For full validation (beta, RC, final): all test cases must be successful
 1. Discuss test results with the release manager, providing details on test coverage
 1. Update https://wiki.ubuntu.com/{ReleaseName}/TechnicalOverview/{Milestone}/Testers (ex: https://wiki.ubuntu.com/QuantalQuetzal/TechnicalOverview/Beta1/Testers) with the list of testers for the release
Line 77: Line 68:

To be carried out by: Ubuntu QA Team Lead

Goals

  • Ensure that all ISOs are suitable for release

  • This process should apply to betas, ReleaseCandidates, final releases, and milestones

T minus 14 days

  1. Prepare the release test plan
    • Review feature goals targeted for this release which should be covered by validation testing
    • Review critical bugs which should be verified
    • Review installation test cases with Evan Dandrea
    • Review upgrade test cases with Michael Vogt
    • Identify set of locales which are sufficiently well supported to be tested

  2. Remind the Canonical distro team to update StaffHardware

  3. Issue call for community testers
    • Commitment to be present and responsive during specific times when candidates will be built
    • Requirement for broadband
    • Reminder to download ISOs in advance, so that rsyncing the candidate is fast
    • List of hardware which needs better representation
    • When volunteering, need to provide:
      • Email address
      • IRC nick
      • Hardware platforms
      • Languages

T minus 7 days

  1. Should now have a complete list of test cases and testers

  2. Populate and prioritize test tracker with all test cases and assign testers

  3. Notify testers via email of their assigned test cases and request confirmation of their ability to carry them out
    • Include instructions for carrying out the tests
      • Installation profile
      • Language
      • Hardware
    • Include instructions for reporting bugs which should be considered for targeting
    • Must respond within 3 days or test cases should be reassigned

T minus 4 days

  1. Review confirmations and reassign test cases as necessary

Upon notification from release manager of the availability of a candidate build:

  1. Establish which test cases must be covered for this build, based on test coverage of previous builds and impact of changes made since
  2. Set current milestone in the tracker and ensure that the candidate builds are listed
  3. Notify testers on IRC of the candidate, remind of their assigned test cases and request acknowledgement
  4. Reassign any test cases for testers who do not acknowledge
  5. Monitor test reports and bug references, notifying the release manager of bugs as they are reported
    • Release manager will identify showstoppers and decide whether to roll a new candidate

Tracking test results

  1. Post candidate ISOs and test cases in the tracker
  2. Announce start of test cycle
  3. Monitor tracker results
    • Ensure that bugs are filed in Launchpad by testers for genuine problems
  4. Proceed with testing until validation criteria are met
    • For minimal validation (alphas): at least one successful installation per ISO is sufficient
    • For full validation (beta, RC, final): all test cases must be successful
  5. Discuss test results with the release manager, providing details on test coverage

  6. Update https://wiki.ubuntu.com/{ReleaseName}/TechnicalOverview/{Milestone}/Testers (ex: https://wiki.ubuntu.com/QuantalQuetzal/TechnicalOverview/Beta1/Testers) with the list of testers for the release

  7. Continue to collect feedback on the build after it is blessed

CategoryProcess

ReleaseValidationProcess (last edited 2012-09-03 17:52:40 by cpe-66-69-240-108)