The purpose of this document is to describe the workflow to manage testing requests parts of the landing process and provide a set of guidelines for testing silos prior to approving them for landing

Managing the Trello board

Lanes

There are 4 lanes on the board:

1. Need QA Sign Off
2. Ready to test
3. Under Testing
4. Pass
5. Fail

Need QA Sign Off

Description

This lane contains incoming testing requests which are to be worked on. No QA work has been done on it yet. The type of testing request are:

• Silo verification.
• Custom tarballs verification.
• Device tarballs verification.
• New app versions.
• Sanity testing of new builds.
• Regression testing for promotion.
• Desktop iso testing

Cards in this lane that don't have a green label, are untriaged, and are not directly ready for testing, as they lack basic information and haven't been prioritized.

Entering the lane

Cards are created automatically when:

• A silo reaches the the state ‘QA needs to sign off’ in the CI train.
• A request is added to the CI Train for a custom tarball.
• A request is added to the CI Train for a device tarball.
• A new image is available for the list of supported images.

Cards are added manually for:

• Executing regression tests on an image for promotion.
• Any extra test request not in this list eg desktop testing.

Leaving the lane

To leave this lane the card must have been triaged and contain the required information to start testing, with at minimum:

• Changelog
• Importance
• Target Milestone

For updates to the stable release or major updates after feature freeze on the development release, the change must be approved by the product team.

Ready to test

Description

This lane contains cards that have been triaged and sorted according to priorities set by stakeholders. Cards in this lane have all the basic information required to start testing. This includes:

• Project/packages involved.
• Silo owner (e.g. project manager or developer).
• Relevant testing information (e.g. Launchpad bugs, special steps).
• Priority and landing approval status.

The order in the lane indicate their priority. Cards of higher priority are at the top and lower priority at the bottom.

Leaving the lane

To move a card to "Under testing", the tester must take a cards from the top (higher priority) and take ownership by assigning it to him/herself.

Under Testing

Description:

This lane contains cards that are being actively tested. This is the work in progress.

Entering the lane:

Cards enter this lane it is triaged and a tester takes it from the ‘Needs QA’ lane.

Only put a card in the ‘Under Testing’ column if you are definitely going to work on it - not if you are planning to at some point in the future - the Trello board must reflect the actual status of testing.

Leaving the lane

Cards leave this lane when testing is complete or cannot be achieved and a new update has to be submitted for QA sign off.

• If testing is complete and passed, the card is moved to the ‘Passed’ lane.
• If testing is complete and failed, the card is moved to the ‘Failed’ lane.
• If testing cannot be completed and an update is required ie a silo will be rebuilt, the card is moved to the ‘Failed’ lane.

Passed

Description: This lane contains cards which passed QA verification.

Entering the lane: QA verifications passed.

Leaving the lane: After 1 week, cards in this lane are moved to ‘Archived’.

Failed

Description: This lanes contains cards which failed QA verification.

Entering the lane: QA verifications failed.

Leaving the lane: After 1 month, cards in this lane are moved to ‘Archived’

Moving cards between lanes

Cards move from left to right. Passed or Failed are the final states.

Card properties

Card properties are managed with labels. There are currently 3 different and independent properties:

1. Milestones
2. Needs approval
3. Blocked

Any of these labels can be assigned to a card.

Milestones

‘Milestones’ indicate which target this verification is for. Milestones are tagged with green ‘Milestone’ labels (dark or light green). There are 2 supported milestones:

• Current development release
• Current stable release

Milestones are assigned when the cards are triaged in the ‘Needs QA’ lane.

Needs approval

‘Needs approval’ is an indicator on the board that the card cannot move from the current lane - usually ‘Need QA sign-off’ without first being approved by the management team. Card marked as ‘Needs approval’ are tagged with an orange ‘Needs approval’ label.

There might be different reasons like:

• Update to a stable release without a milestone
• Major update of a component after a freeze
• Any high risk update.

Blocked

Blocked is an indicator on the board that there is a problem at hand. Card marked as blocked are tagged with a red ‘Blocked’ label. While blocking, one needs to enter the reason due to which the card is being blocked. Once a card is blocked, it can’t be moved until blockage is resolved.

Blocked card represent idle time.

Cards can be blocked in ‘Need QA sign-off’ and ‘Under testing’ lanes for any reason that prevent the card from moving forward. Some of the reasons can be:

• Missing information to proceed with testing.
• Needs feedback of the engineering team regarding the fix, the test plan, or the interpretation of the results

While blocking, one needs to enter the reason due to which the card is being blocked. Once a card is blocked, it can’t be moved until blockage is resolved.

While unblocking, one needs to enter the reason due to which the card is being unblocked. Once a card is unblocked, it can continue in the same lane or can be moved to the next lane.

Card ownership

A task owner is the person the who’s currently assigned to it. Anyone who takes a card from the ‘Need QA sign-off’ lane and moves to ‘Under testing’ lane becomes the owner of the card in that lane. One can also become a owner of an unassigned card in the ‘Under testing’ lane.

Unassigned card in the ‘Under testing’ lane are the priority card for owner assignment.

If you can’t finish a silo before you end your day - make sure you leave all of the information that might be necessary for someone else who picks it up to avoid repeating too much of what you did - e.g. if there are any test plans you fully completed, note them.