The purpose of this document is to describe the workflow to manage testing requests parts of the landing process and provide a set of guidelines for testing silos prior to approving them for landing
Managing the Trello board
Lanes
There are 4 lanes on the board:
- Need QA Sign Off
- Ready to test
- Under Testing
- Pass
- Fail
Need QA Sign Off
Description
This lane contains incoming testing requests which are to be worked on. No QA work has been done on it yet. The type of testing request are:
- Silo verification.
- Custom tarballs verification.
- Device tarballs verification.
- New app versions.
- Sanity testing of new builds.
- Regression testing for promotion.
- Desktop iso testing
Cards in this lane that don't have a green label, are untriaged, and are not directly ready for testing, as they lack basic information and haven't been prioritized.
Entering the lane
Cards are created automatically when:
- A silo reaches the the state ‘QA needs to sign off’ in the CI train.
- A request is added to the CI Train for a custom tarball.
- A request is added to the CI Train for a device tarball.
- A new image is available for the list of supported images.
Cards are added manually for:
- Executing regression tests on an image for promotion.
- Any extra test request not in this list eg desktop testing.
Leaving the lane
To leave this lane the card must have been triaged and contain the required information to start testing, with at minimum:
- Changelog
- Importance
- Target Milestone
For updates to the stable release or major updates after feature freeze on the development release, the change must be approved by the product team.
Ready to test
Description
This lane contains cards that have been triaged and sorted according to priorities set by stakeholders. Cards in this lane have all the basic information required to start testing. This includes:
- Project/packages involved.
- Silo owner (e.g. project manager or developer).
- Relevant testing information (e.g. Launchpad bugs, special steps).
- Priority and landing approval status.
The order in the lane indicate their priority. Cards of higher priority are at the top and lower priority at the bottom.
Leaving the lane
To move a card to "Under testing", the tester should pick cards from the top (higher priority) and must take ownership by assigning it to him/herself.
Under Testing
Description:
This lane contains cards that are being actively tested. This is the work in progress.
Entering the lane:
Cards enter this lane it is triaged and a tester takes it from the ‘Needs QA’ lane.
Only put a card in the ‘Under Testing’ column if you are definitely going to work on it - not if you are planning to at some point in the future - the Trello board must reflect the actual status of testing.
Leaving the lane
Cards leave this lane when testing is complete or cannot be achieved and a new update has to be submitted for QA sign off.
- If testing is complete and passed, the card is moved to the ‘Passed’ lane.
- If testing is complete and failed, the card is moved to the ‘Failed’ lane.
- If testing cannot be completed and an update is required ie a silo will be rebuilt, the card is moved to the ‘Failed’ lane.
Passed
Description: This lane contains cards which passed QA verification.
Entering the lane: QA verifications passed.
Leaving the lane: After 1 week, cards in this lane are moved to ‘Archived’.
Failed
Description: This lanes contains cards which failed QA verification.
Entering the lane: QA verifications failed.
Leaving the lane: After 1 month, cards in this lane are moved to ‘Archived’
Moving cards between lanes
Cards move from left to right. Passed or Failed are the final states.
Card properties
Card properties are managed with labels. There are currently 3 different and independent properties:
- Milestones
- Needs approval
- Blocked
Any of these labels can be assigned to a card.
Milestones
‘Milestones’ indicate which target this verification is for. Milestones are tagged with green ‘Milestone’ labels (dark or light green). There are 2 supported milestones:
- Current development release
- Current stable release
Milestones are assigned when the cards are triaged in the ‘Needs QA’ lane.
Needs approval
‘Needs approval’ is an indicator on the board that the card cannot move from the current lane - usually ‘Need QA sign-off’ without first being approved by the management team. Card marked as ‘Needs approval’ are tagged with an orange ‘Needs approval’ label.
There might be different reasons like:
- Update to a stable release without a milestone
- Major update of a component after a freeze
- Any high risk update.
Blocked
Blocked is an indicator on the board that there is a problem at hand. Card marked as blocked are tagged with a red ‘Blocked’ label. While blocking, one needs to enter the reason due to which the card is being blocked. Once a card is blocked, it can’t be moved until blockage is resolved.
Blocked card represent idle time.
Cards can be blocked in ‘Need QA sign-off’ and ‘Under testing’ lanes for any reason that prevent the card from moving forward. Some of the reasons can be:
- Missing information to proceed with testing.
- Needs feedback of the engineering team regarding the fix, the test plan, or the interpretation of the results
While blocking, one needs to enter the reason due to which the card is being blocked. Once a card is blocked, it can’t be moved until blockage is resolved.
While unblocking, one needs to enter the reason due to which the card is being unblocked. Once a card is unblocked, it can continue in the same lane or can be moved to the next lane.
Card ownership
A task owner is the person the who’s currently assigned to it. Anyone who takes a card from the ‘Need QA sign-off’ lane and moves to ‘Under testing’ lane becomes the owner of the card in that lane. One can also become a owner of an unassigned card in the ‘Under testing’ lane.
Unassigned card in the ‘Under testing’ lane are the priority card for owner assignment.
If you can’t finish a silo before you end your day - make sure you leave all of the information that might be necessary for someone else who picks it up to avoid repeating too much of what you did - e.g. if there are any test plans you fully completed, note them.
Triaging Incoming Requests
To triage an incoming testing request verify that at least the following criteria are met:
- Make sure it is approved for landing for a given target
- On RTM, canonical-system-image is subscribed
- On Vivid, some standardized way to indicate that it’s approved (to discuss with the landing team)
- Bug(s), MP(s), and test plan(s) are linked.
- The changelog clearly describes the purpose of the upload.
Update the card:
- Set a label for the corresponding milestone.
- Link or paste the changelog to the card.
- Link the diff files to the card.
Performing Verification
See also Silo Testing Guidelines for additional information about setting up a silo and configuring a device for testing.
How to perform the test
After booting into the affected release of Ubuntu the following steps should be taken:
- Recreate the issue as described in the bug linked to the update or in the CI train request.
- Install the silo
- Silo actually fixes the issue it’s intended to fix. (problem exists pre-silo but not post-silo)
- Behaviour post-silo is better than pre-silo (or no worse, if no visible change) and doesn’t introduce any regression.
- Automated test results are published somewhere: (may need CI changes to add a field for this link)
- Link to the the output of autopilot tests
- Link to the build log with the output of the tests executed during the build.
- If possible review the diff, and verify that it looks sane (can be time-consuming and/or incomprehensible) Having a look at the patch may help to know exactly which part of the package is affected by the fix and what needs to be more specifically tested.
- Check for translation changes
- This change has an associated test (can be difficult to verify)
- Update to an existing system or integration test
- Requires a new system or integration test
- New or updated unit test.
- Wiki test plan(s) passes
Get results of the tests before anything lands.
One of the most important aspects of the testing procedure is to verify that the update does not introduce a regression.
Updating the board
If a silo passes verification, move the card to the Passed lane and set the QA status to Granted on the CI Train spreadsheet.
If a silo fails verification, move the card to the Failed lane, add a comment explaining clearly the reason of the failure and notify the lander.
Updating the test management tool
If there is a valuable test case, add the test case to the test management tool and set its status to Draft or To Review.
If the update changes an existing test case, update the test case in the test management tool and set its status to To Review.