ReleaseValidationProcess
To be carried out by: Henrik Nilsen Omma
Goals:
- Ensure that all ISOs are suitable for release
This process should apply to betas, [:ReleaseCandidate:ReleaseCandidates], final releases, and milestones
T minus 14 days:
- Prepare the release test plan
- Review feature goals targeted for this release which should be covered by validation testing
- Review critical bugs which should be verified
- Review installation test cases with Evan Dandrea
- Review upgrade test cases with Michael Vogt
- Identify set of locales which are sufficiently well supported to be tested
Remind the Canonical distro team to update [https://wiki.canonical.com/StaffHardware StaffHardware]
- Issue call for community testers
- Commitment to be present and responsive during specific times when candidates will be built
- Requirement for broadband
- Reminder to download ISOs in advance, so that rsyncing the candidate is fast
- List of hardware which needs better representation
- When volunteering, need to provide:
- Email address
- IRC nick
- Hardware platforms
- Languages
T minus 7 days:
- Should now have a complete list of test cases and testers
- Populate and prioritize test matrix with all test cases and assign testers
- Notify testers via email of their assigned test cases and request confirmation of their ability to carry them out
- Include instructions for carrying out the tests
- Installation profile
- Language
- Hardware
- Include instructions for reporting bugs which should be considered for targeting
- Must respond within 3 days or test cases should be reassigned
- Include instructions for carrying out the tests
T minus 4 days:
- Review confirmations and reassign test cases as necessary
Upon notification from release manager of the availability of a candidate build:
- Establish which test cases must be covered for this build, based on test coverage of previous builds and impact of changes made since
1 Generate tracker bugs in the [https://bugs.launchpad.net/ubuntu-iso-tests/+bugs ISO testing tracker]
- Notify testers on IRC of the candidate, remind of their assigned test cases and request acknowledgement
- Reassign any test cases for testers who do not acknowledge
- Collect test reports and bug references, notifying the release manager of bugs as they are reported
- Release manager will identify showstoppers and decide whether to roll a new candidate
Draft process for tracking test results
1 Generate tracker bugs in the [https://bugs.launchpad.net/ubuntu-iso-tests/+bugs ISO testing tracker]
- For full validation: one per ISO per test case (all cases need to be tested)
- For minimal validation: one per ISO which needs to be tested (any one of the installation test plans should be sufficient)
- Tracking bug contains build number, link to test plan(s), instructions for reporting results
- Announce start of test cycle
[http://www.ubuntuforums.org/forumdisplay.php?f=201 ISO testing forum]
[http://lists.ubuntu.com/mailman/listinfo/ubuntu-devel ubuntu-devel]
- XXX: consider making a testing-team-announce list for this purpose as well
- IRC
- Testers subscribe to the appropriate tracking bug for any test case they plan to perform (XXX: can we auto-subscribe them for the major releases given, that they are already required to sign up by email?)
- Monitor tracking bugs
- Ensure that separate bugs are filed by testers for genuine problems
- Update bug status to reflect testing status:
Unconfirmed - newly filed, untested image
In progress - hs received at least one PASS
Rejected - Has failed, build new image
Fix Released - Milestone released
- Proceed with testing until validation criteria are met
- For minimal validation: at least one successful installation per ISO is sufficient
- For full validation: all test cases must be successful
- Notify release manager of test results
- XXX: How? via the tracker bug status or does a person do this via wiki/IRC
- Continue to collect feedback on the build after it is blessed