There are a lot of new terms, tools and procedures we want to use for Testing on Ubuntu Personal. This page is here to document those processes. ||<>|| == Silo Testing == === Prechecks === * All merge proposals must have been reviewed and approved. * All automated tests must pass. * New features must come with automated tests. * New strings must be marked for translation. === Installation === Install phablet-tools package in your desktop machine. * Flash the phone with the latest development image (instructions [[https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Image_Testing_Channels|here]] ) * `citrain device-upgrade ` * /!\ This section may not be needed now if you use an up-to-date distro as citrain has built it in /!\ Overlay PPA * In case of Vivid + Overlay PPA, the silo packages need to be pinned to a higher priority. Otherwise, silo packages won't be installed. * Create this file: `sudo vi /etc/apt/preferences.d/silo.pref` * Add these lines (silo number must be 0-padded, e.g. landing-001): {{{ Package: * Pin: release o=LP-PPA-ci-train-ppa-service-landing- Pin-Priority: 1002 }}} * Save the file. * Update the package list: `sudo apt-get update` * Upgrade the packages: `sudo apt-get upgrade` === Trello Board Edits === * The trello board is located [[https://trello.com/b/AE3swczu/qa-testing-requests-for-questions-ping-ubuntu-qa-on-ubuntu-ci-eng|here]] * Grab a ticket from the "Ready for Testing" Lane * Add yourself to the ticket (Hover over the ticket and tap space, or select Members and add yourself from there) * From the testplan for the silo under test add the title for each section to a ticket comment and mark them as passed or failed. Example: {{{ NO ACCOUNT: PASS. EXISTING ACCOUNT: PASS. Security Information: PASS. Security Information: Paypal payment: PASS. Security Information: Add Credit/debit card: FAIL (cancel presents blank page instead Preview). Cancel button flow: PASS. Buy Now button flow: Logged in: PASS. Add credit/debit card flow: Cancel adding card: FAIL (cancel presents blank page instead Preview). Add credit/debit card flow: Add a card: FAIL ("Cancel" in credit card confirmation page -bank website- doesn't add the credit card). The blank page is a regression, it works fine in the current pay-ui. }}} * If there is a failure in a separate comment clearly make a note of the issue. === Test Plan Locations === * [[https://wiki.ubuntu.com/Process/Merges/TestPlan/|List of test plans]]. === Reporting New Cases === * Any relevant silo bugs (both new and fixed) should be added to the Regression Test Suite as "To Review". * Testplan levels Set the level relevant to the level to access the item, High anything on an initial page section, Medium on a secondary, low anything below that. (Example browser app, opens, connects to web, opens a page, open a new tab, save a bookmark = high, open a bookmark, history, private mode = medium, rename a bookmark, clear history = low) === More Info === [[https://wiki.ubuntu.com/LandingTeam/SiloTestingGuidelines|More info can be found here]] == Tarball Testing == === Prechecks === Images are scheduled [[https://docs.google.com/spreadsheets/d/1LiPHq39XvPaTQoE8vmi-dqSVMTGGLfCh7bFY01BcNNk/edit#gid=50|here]] === Installation === For Custom Tarballs Read: https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Testing_Custom_Tarballs <
> For Device Tarballs Read: https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Testing_Device_Tarballs === Test Plan Locations === See Sanity Testing Below for device tarballs <
> For Custom Tarballs look at practitest (cases need adding here) === Reporting New Cases === All new cases must be added to the "To Review" Queue in practitest === More Info === https://wiki.ubuntu.com/QATeam/ChannelsToFlash == Sanity Testing == === Prechecks === All silos are landed and all tarballs are landed. === Installation === * Follow the instructions [[https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Image_Testing_Channels|here]] to get the latest image install * Then run the sanity suite === Test Plan Locations === Sanity test are located in practitest currently === Reporting New Cases === Sanity is a deliberately selected small subset of tests targeting software interactions with hardware. Any additional tests felt required should be discussed with the entire team via the mailing list or standups. === More Info === https://wiki.ubuntu.com/QATeam/ChannelsToFlash == Regression Testing == === Prechecks === Sanity Must of been run to ensure the image is at least testable === Installation === * Follow the instructions [[https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Image_Testing_Channels|here]] to get the latest image install * Then run the regression suite === Test Plan Locations === Regression tests are located in Practitest Currently === Reporting New Cases === New Cases should not be found while testing regression, however you may want make a note here(add a link later) to remind you to look at something during exploratory testing. === More Info === https://wiki.ubuntu.com/QATeam/ChannelsToFlash == Exploratory Testing == /!\ Should be carried out by canonical staff only === Prechecks === The area you wish to test is actually functioning. === Installation === * Follow the instructions [[https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Image_Testing_Channels|here]] to get the latest image install * Work on the area you have selected === Recording Test and Session === Use the web form [[https://docs.google.com/a/canonical.com/forms/d/1rqF4V8VMleQQliEaEqzrUke4aiblSbsczK5BWL33DTU/viewform?edit_requested=true#|here]] to keep track of what you test === Reporting New Cases === If there is automation required here it should be reported as a bug for the development team.<
> If there are new manual cases they should be put into draft for review in practitest. == Click Package Testing == === Prechecks === 1. Goto [[https://docs.google.com/spreadsheets/d/1LiPHq39XvPaTQoE8vmi-dqSVMTGGLfCh7bFY01BcNNk/edit?pli=1#gid=50|here]] and check for a new click package 1. Download the click and note it's location (Default is ~/Downloads) === Installing === 1. Fresh flash the device see [[https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Flashing_the_Current_Release|Flashing Current Release]] 1. Once flash setup the device 1. Now goto system settings → about device → developer mode and enable developer mode 1. While the screen is on do {{{ adb push Download/click_package_name.click /home/phablet }}} 1. Follow this up with {{{ adb shell }}} 1. In that shell do {{{ pkcon install-local --allow-untrusted click_package_name.click sudo reboot }}} 1. Click package is now ready to test === Testing === There are various test plans dotted around but on the whole the spread sheet will point you to the right one, there are also the case in Practitest. == Automated Testing == === Prechecks === === Installation === === Test Plan Locations === === Reporting Of Failures === === More info === == Miscellaneous == === Faking Your Location for Scopes Testing === <> For the Nearby scope, some functionality is supported only enabled cities. You can “fake location” to London which supports both Songkick and Timeout as follows. 1. Set image as writable:{{{ phablet-config writable-image }}} 1. phablet-shell into the device 1. Create this file /custom/custom.prop with below contents:{{{ $ sudo vim.tiny /custom/custom.prop custom.location.fake=true custom.location.lon=-0.0999525 custom.location.lat=51.5062348 }}} 1. Reboot 1. Refresh scope AFTER first display Or you can fake your location to one of these Timeout-supported cities: Accra, Amsterdam, Barcelona, Berlin, Boston, Chicago, Kuala Lumpur, Las Vegas, London, Los Angeles, Madrid, Miami, México DF, New York, Paris, Washington DC. == FAQ == 1. How can the community help? * The community are the main users of the currently supported devices so: * Knowing what is wrong (bug reports, see point 2) * Knowing what is right (ubuntu-phone@lists.launchpad.net send a mail here the devs would love to know what you love) * Knowing images are working (Bug reports when the don't, see point 2) * Knowing if something is missing (Bug reports, please don't be dismayed if they get marked as wishlist lot to do, alternatively talking to the development team/creator of you favourite app about a port to Ubuntu is good too :) ) 1. What info do you need in bug reports? * https://wiki.ubuntu.com/QATeam/BugReportInclusions lists the basics and the start for wifi and phone logs more to come. * Most packages are listed in the https://wiki.ubuntu.com/Avengers Please look for a bug similar to yours before reporting a new one.