PPA_Testing

Differences between revisions 38 and 39
⇤ ← Revision 38 as of 2013-07-29 16:20:22 →
Size: 7994
Editor: host-2-99-159-216
Comment:
← Revision 39 as of 2013-07-29 16:30:31 → ⇥
Size: 2609
Editor: host-2-99-159-216
Comment:
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
Stable Release Update (SRU) Verification is the process of testing packages from the -proposed repository, that have been updated to fix bugs in a [[https://wiki.ubuntu.com/Releases|stable release]]. These packages need testing to ensure that the package continues to function as designed and that the bug is fixed. It is also important '''to ensure no regression has been introduced''' by the fix.
Line 7: Line 6:
More information about the process can be found at [[https://wiki.ubuntu.com/StableReleaseUpdates|Stable Release Update]] page. To find out how to enable -proposed look at the [[https://wiki.ubuntu.com/Testing/EnableProposed|Enable Proposed]] page. = PPA Testing =
Sometimes during the test cycle one of the developers may ask you to test something specific for them. This is often phrased as
Line 9: Line 9:

== How to find bugs needing verification ==

There are many ways to find bugs needing SRU verification:

 * by viewing the [[http://people.ubuntu.com/~ubuntu-archive/pending-sru.html|Pending Ubuntu SRUs]],
 * by querying Launchpad for a specific release (here lucid) and the bug tag [[https://launchpad.net/ubuntu/lucid/+bugs?field.status%3Alist=FIXCOMMITTED&field.tag=verification-needed|verification-needed]],
 * or by looking at bugs which the SRU verification team is [[https://bugs.launchpad.net/~sru-verification/+subscribedbugs|subscribed]].

The Pending Ubuntu SRUs are generated by parsing the changelogs of the packages in -proposed repository. By clicking on '''changelog bugs''' entry you will be taken to the Launchpad bug report (if it exists). If no changelog bug is displayed, clicking on the version of the package in the '-proposed' column will take you to the changelog entry for that version of the package in launchpad (the bug number may be displayed in the changelog but not in the report if the bug number doesn't respect a standard format)

Some bugs resolved by -proposed packages require specific hardware and these can be identified by the '''hw-specific''' tag in Launchpad or by the '''(hw)''' next to the bug number at the Pending Ubuntu SRUs page.

== Identifying how to test ==

The first step in identifying how to test, is determining the release or releases of Ubuntu affected by the particular bug. This can be done by looking at the bug report and determining the release affected by the bug.

More than one release may be affected by the bug. In that case, the bug needs to be reproduced and the fix needs to be tested for each affected release.

In the example below the Landspace bug affects the Jaunty, Karmic, Lucid and Maverick releases of Ubuntu and should be verified in each stable release (Jaunty, Karmic and Lucid)
  ||<tablestyle="float:center; font-size: 0.75em; background:#C2B8A1;margin: 0 0 1em 1em;" style="padding:0.5em;">[[https://wiki.ubuntu.com/QATeam/PerformingSRUVerification?action=AttachFile&do=get&target=bug_example_large.png|{{https://wiki.ubuntu.com/QATeam/PerformingSRUVerification?action=AttachFile&do=get&target=bug_example_small.png}}]]||
  || Click to zoom ||

Alternatively, at the [[http://people.ubuntu.com/~ubuntu-archive/pending-sru.html|Pending Ubuntu SRUs]] page there are sections for each release of Ubuntu that is currently supported.

In addition to knowing the release or releases of Ubuntu affected you also need to have detailed steps to recreate the bug. These can be found in the "TEST CASE" section at the end of the bug's description. [[https://bugs.edge.launchpad.net/ubuntu/karmic/+source/flashplugin-nonfree/+bug/429841|Flashplugin-nonfree bug 429841]] has an example of what the test case will look like.

Test cases should be added to the description of the report in a standardised format:
{{{
TEST CASE:
1. description of step 1
2. description of the step 2
3. ...

VERIFICATION DONE
What is the expected result at the end of the test.		 {{{
please install application from ppa:xyz and retest.
Line 47: Line 13:
Note that the verification-failed is either the subject of the bug report or a regression. As the developers spend most of their time talking in a language that I understand as well as I do [[http://en.wikipedia.org/wiki/Klingon#Language | Klingon]] I did ask for an explanation that humans could understand. In the below example, I will use the application '''application''' from the ppa '''ppa:xyz'''. These two pieces of information will be given to you. If you are ever in doubt of either the application or the ppa full names, please do ask the devs. Do '''not''' install ppa's that you do not fully trust, they can be a source of [[http://en.wikipedia.org/wiki/Malware | malware]], if in doubt.... '''ASK''' Some PPA's will ask for acceptance of a GPG key... Again, if in doubt '''ASK'''
Line 49: Line 15:
Writing the test case is a mandatory step of the [[StableReleaseUpdates|SRU procedure]]. But sometimes, there is no such test case. If the test case is missing and the reproduction steps are not obvious, you can provide one if you can reproduce the issue described in the bug report or set the status to 'In progress', describe why you cannot reproduce and set the verification tag to 'verification-failed'. == Adding ==
Line 51: Line 17:
== How to perform the test == Open a [[https://help.ubuntu.com/community/UsingTheTerminal| terminal]] session:
Line 53: Line 19:
After booting into the affected release of Ubuntu the following steps should be taken: {{{
sudo apt-add-repository ppa:xyz
sudo apt-get update
sudo apt-get install application
}}}
Line 55: Line 25:
 1. Ensure that your system is up to date by installing all available updated packages from the -updates and -security repositories
 1. Verify that you do not have the proposed package installed by checking the package version using 'dpkg -l PKGNAME | cat'
 1. Recreate the bug using the steps identified in the "TEST CASE"
 1. Modify your '/etc/apt/sources.list' file to include the -proposed repository as described in [[Testing/EnableProposed|how to enable -proposed]].
  * deb http://archive.ubuntu.com/ubuntu/ oneiric-proposed main restricted universe
 1. Execute 'sudo apt-get update'
 1. Install the proposed package via 'sudo apt-get install PKGNAME' or 'sudo apt-get install PKGNAME=VERSION-NUM'
 1. Verify that you installed the correct package version using 'dpkg -l PKGNAME | cat'
 1. Reboot the system
 1. Try to recreate the bug using the steps identified in the "TEST CASE"
 1. Use the software installed by the package in common ways		 If being asked to accept a GPG key, the steps will be slightly different. At the step
{{{
sudo apt-add-repository ppa:xyz
}}}
you will asked to accept the GPG key, simply press ''Enter'' to accept, provided you trust the PPA.
Line 67: Line 31:
Having a look at the patch may help to know exactly which part of the package is affected by the fix and what needs to be more specifically tested. What the above does:
Line 69: Line 33:
One of the most important aspects of the testing procedure is to verify that the update does not introduce a regression.  1. sudo apt-add-repository ppa:xyz '''''add the ppa called xyz to your system index'''''
 2. sudo apt-get update '''''tell your system it is there to be used'''''
 3. sudo apt-get install application '''''install the application'''''
Line 71: Line 37:
== Updating the bug report == == Removing ==
Line 73: Line 39:
<<Include(StableReleaseUpdates,,from="== Verification ==", to="If you want to help us")>> Now, if the new '''application''' broke your system, then you need to get rid of it and re-install the current version.
Line 75: Line 41:
== Tagging the report == Open a [[https://help.ubuntu.com/community/UsingTheTerminal| terminal]] session:
Line 77: Line 43:
=== SRU specific === {{{
sudo apt-add-repository -r ppa:xyz
sudo apt-get update
sudo apt-get remove application
sudo apt-get install application
}}}
Line 79: Line 50:
<<Include(Bugs/Tags,,from='=== SRU Specific ===',to='=== X ')>> What the above does:
Line 81: Line 52:
=== Regression specific ===

<<Include(Bugs/Tags,,from='=== Regression specific ===',to='=== SRU Specific ===')>>

== Ways to test using virtual machines ==

In the event that your current release of Ubuntu is not the same as the release of Ubuntu affected by the bug there are still many ways for you to perform the verification of the Stable Release Update without installing the affected release on your hardware.

This can be done by using a [[https://help.ubuntu.com/community/VirtualMachines|Virtual Machine]] such as [[https://wiki.ubuntu.com/Testing/VirtualBox|Virtual Box]], [[https://wiki.ubuntu.com/KvmVirtManagerEtc|kvm]], [[https://help.ubuntu.com/community/Installation/QemuEmulator|qemu]] or VMware.

Virtual Machines can not be used to reproduce hardware specific issues.

Depending on the nature of the bug report it may also be possible to use a [[https://wiki.ubuntu.com/DebootstrapChroot|chroot]] to perform the verification.

== Tips and tricks ==

 * An useful tool to quickly check the versions of a package available in the active releases is `rmadison` from the package `devscripts`.
 * You can fetch the changelog of a package directly from the command line with `apt-get changelog PACKAGE_NAME`
 * The full publishing history of a package: https://launchpad.net/ubuntu/+source/PACKAGE_NAME/+publishinghistory
 * If you're wondering where is the uploaded package you should check [[ https://launchpad.net/ubuntu/oneiric/+queue|the state of the upload queue]]

== Links ==

 * [[StableReleaseUpdates|Stable Release Update]] procedure
 * [[http://people.ubuntu.com/~ubuntu-archive/pending-sru.html|Pending Ubuntu SRUs]]
 * List of bugs which the SRU verification team is [[https://bugs.launchpad.net/~sru-verification/+subscribedbugs|subscribed]]
 * How to [[Testing/EnableProposed|Enable Proposed]]
 * How to triage [[Bugs/HowToTriage/|guide]]		  1. sudo apt-add-repository -r ppa:xyz '''''remove the test ppa area'''''
 2. sudo apt-get update '''''tell your system it is no longer to be used'''''
 3. sudo apt-get remove application '''''remove the new version'''''
 4. sudo apt-get install application '''''now that the test ppa is no longer in use, the install will 'grab' the one from the normal area.'''''

PPA Testing

Sometimes during the test cycle one of the developers may ask you to test something specific for them. This is often phrased as 

please install application from ppa:xyz  and retest.

As the developers spend most of their time talking in a language that I understand as well as I do Klingon I did ask for an explanation that humans could understand. In the below example, I will use the application application from the ppa ppa:xyz. These two pieces of information will be given to you. If you are ever in doubt of either the application or the ppa full names, please do ask the devs. Do not install ppa's that you do not fully trust, they can be a source of malware, if in doubt.... ASK Some PPA's will ask for acceptance of a GPG key... Again, if in doubt ASK

Adding

Open a terminal session: 

sudo apt-add-repository ppa:xyz
sudo apt-get update
sudo apt-get install application

If being asked to accept a GPG key, the steps will be slightly different. At the step 

sudo apt-add-repository ppa:xyz

you will asked to accept the GPG key, simply press Enter to accept, provided you trust the PPA.

What the above does:

  1. sudo apt-add-repository ppa:xyz add the ppa called xyz to your system index

  2. sudo apt-get update tell your system it is there to be used

  3. sudo apt-get install application install the application

Removing

Now, if the new application broke your system, then you need to get rid of it and re-install the current version.

Open a terminal session: 

sudo apt-add-repository -r ppa:xyz
sudo apt-get update
sudo apt-get remove application
sudo apt-get install application

What the above does:

  1. sudo apt-add-repository -r ppa:xyz remove the test ppa area

  2. sudo apt-get update tell your system it is no longer to be used

  3. sudo apt-get remove application remove the new version

  4. sudo apt-get install application now that the test ppa is no longer in use, the install will 'grab' the one from the normal area.

QATeam/phillw/PPA_Testing (last edited 2014-07-22 21:57:49 by host-80-41-221-66)