* '''Test plan for component''': ubuntu-system-image
 * '''Component Checklist''': https://wiki.ubuntu.com/Process/Merges/Checklists/ubuntu-system-image
 * '''Trunk URL''': lp:ubuntu-system-image/client
 * '''Ubuntu Package URL (LP)''': http://launchpad.net/ubuntu/+source/system-image
 * '''Project wiki page''': https://wiki.ubuntu.com/ImageBasedUpgrades

<<TableOfContents()>>

== Dependents/Clients ==

Dependents:
 * ubuntu-download-manager

Clients:
 * ubuntu-system-settings

== General Recommendations ==

'''Note:''' You'll want to begin each test in a "fresh" state.  Specifically, make sure that before you start a test, the {{{system-image-dbus}}} process is '''not''' running.  I've noticed that this process may already be running after a reboot, and this could affect the success of your test.  You can check this with:

{{{
adb shell ps -waux | grep system-image-dbus
}}}

If you see something like the following, then the process is running and you'll need to kill it (e.g. via {{{SIGTERM}}}) or rebooting.

You may also want to clear the log file of any previous results:

{{{
truncate -s 0 /var/log/system-image/client.log
}}}

If you're testing packages in a landing silo, after flashing your device as instructed in the test plans below, you will want to install the new packages from the silo.  You can do that this way:

 * Install freshly built MPs that are needed for landing, e.g.
   * Make device writable
     * {{{touch /userdata/.writable_image}}}
     * reboot
   * {{{add-apt-repository}}} the relevant PPA.
   * {{{apt-get update}}}
   * {{{apt-get install system-image-cli system-image-common system-image-dbus}}}
 * '''Do not install''' the {{{system-image-dev}}} package as this is normally not installed on devices and its absence/presence has an effect on the operation of system-image.
 * '''Do not''' do an {{{apt-get upgrade}}}
 * Install any new versions of {{{ubuntu-download-manager}}} and {{{ubuntu-system-settings}}} from the same silo.
 * Possibly install any new versions of the Qt networking stack that may also be part of the silo.

Alternatively, just download the .debs from the silo PPA and {{{dpkg -i *.deb}}} manually.

If any of the tests fail, [[https://bugs.launchpad.net/ubuntu-system-image/+filebug|file a bug]] attaching the following information.  It is requested that you file the bug on the upstream package (using that link), although it's okay -- but not necessary -- to ''also'' file a bug on the Ubuntu package.  In either case, please tag the bug with the {{{client}}} tag.

 * If the test fails, capture the log files:
   * {{{adb pull /var/log/system-image/client.log}}}
   * Optionally capture the u-d-m log files (TBD)
 * Attach these logs to the bug.

After you flash your device with {{{--bootstrap}}} you will probably lose your network settings.  Restore them with:

 * {{{adb shell nmcli dev wifi connect <network> password <password>}}}

There are two sets of test plans depending on whether you're QAing system-image 2.5 or system-image 3.

== Test Plan for system-image 3 ==

'''This test plan is not supposed to be complete; use it to guide your manual testing so they don't miss big functional areas that are part in the component; also this should be used as guideline to inspire the exploratory testing which should be adapted smartly based on the real content of a merge proposal.'''

=== Test A -- Good Path Automatic Download ===

This tests standard upgrading from an older version to the current version.  It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.

 * {{{cd /etc/system-image}}}
 * {{{system-image-cli --info | grep "current build number"}}}
 * Subtract 1 from the build number
 * {{{printf "[service]\nbuild_number: NNN\n" > config.d/02_testing.ini}}} where ''NNN'' is the current build number minus 1
 * {{{system-image-cli --info | grep "current build number"}}} to verify the new, lower build number
 * {{{tail -f /var/log/system-image/client.log}}}
 * Open System Settings panel, click on Updates
 * Wait for it to say there is an update available.
 * Wait for updates to finish downloading.
 * Click on ''Restart & install''
 * After the reboot, you '''MUST''' first {{{rm -f /etc/system-image/config.d/02_testing.ini}}}
 * Open System Settings panel, then Updates.  It should report that your ''Software is up to date''

=== Test B -- Good Path Manual Download ===

This tests standard upgrading from an older version to the current version.  It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.


 * Enable manual downloads: {{{system-image-cli --set auto_download=0}}}
 * {{{cd /etc/system-image}}}
 * {{{system-image-cli --info | grep "current build number"}}}
 * Subtract 1 from the build number
 * {{{printf "[service]\nbuild_number: NNN\n" > config.d/02_testing.ini}}} where ''NNN'' is the current build number minus 1
 * Restart {{{system-image-dbus}}} like so: {{{kill -TERM `ps -C system-image-dbus -o pid=`}}} - ignore this if it prints an error; that just means that the process isn't running.
 * {{{system-image-cli --info | grep "current build number"}}} to verify the new, lower build number
 * {{{tail -f /var/log/system-image/client.log}}}
 * Open System Settings panel
 * Click on ''Updates''
 * Click on ''Download''
 * Wait for the ''Download'' button to say ''Install...''
 * Click on ''Install...''
 * Click on ''Restart & install''
 * After the reboot, you '''MUST''' first {{{rm -f /etc/system-image/config.d/02_testing.ini}}} and restart {{{system-image-dbus}}} as above
 * Open System Settings panel, then Updates.  It should report that your ''Software is up to date''
 * {{{system-image-cli --set auto_download=1}}}

== Test Plan for system-image 2.5 ==

'''This test plan is not supposed to be complete; use it to guide your manual testing so they don't miss big functional areas that are part in the component; also this should be used as guideline to inspire the exploratory testing which should be adapted smartly based on the real content of a merge proposal.'''

=== Test A -- Good Path Automatic Download ===

This tests standard upgrading from an older version to the current version.  It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.

 * Flash device to an older version than the current one, e.g. {{{ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --revision -5 --bootstrap}}}
 * '''Make sure system-image-dbus isn't running -- see above'''
 * Open System Settings panel.
 * Wait for it to say there is an update available.
 * Click on ''Updates available''
 * Wait for updates to finish downloading.
 * Click on ''Install & restart''
 * After the reboot, open System Settings panel, then Updates.  It should report that your ''Software is up to date''

=== Test B -- Good Path Automatic Download Current Packages ===

This tests standard upgrading like above, but it uses the current set of packages and ''fakes'' the system upgrading to a newer version.  It actually upgrades from the newest version to the newest version because we're going to lie about what version the device is actually on.  This is a good test to run to ensure that the current latest build is working as expected.

 * Flash device to the current version, e.g. {{{ubuntu-device-flash --channel ubuntu-touch/devel-proposed --bootstrap}}}
 * Falsify which version your device thinks it's on:
   * {{{adb shell}}}
   * {{{cd /etc/system-image}}}
   * {{{cat channel.ini}}}
   * Look at the {{{build_number}}} line to get the current build number.  Subtract 1.  (Next line assumes you're at build 246.)
   * {{{sed -e "s/246/245/" -i.bak channel.ini}}}
   * (If this gets messed up, just restore {{{channel.ini}}} from {{{channel.ini.bak}}})
 * '''Make sure system-image-dbus isn't running -- see above'''
 * Open System Settings panel.
 * Wait for it to say there is an update available.
 * Click on ''Updates available''
 * Wait for updates to finish downloading.
 * Click on ''Install & restart''
 * After the reboot, open System Settings panel, then Updates.  It should report that your ''Software is up to date''

=== Test C -- Good Path Manual Download ===

This tests standard upgrading from an older version to the current version.  It's important to note that this test is dependent on the set of packages in the older version, so it may not be appropriate when significant changes in the upgrade stack (e.g. s-i, u-d-m, or packages they depend on) are in the latest revision.

 * Flash device to an older version than the current one, e.g. {{{ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --revision -1 --bootstrap}}}
 * Enable manual downloads:
   * {{{adb shell system-image-cli --set auto_download=0}}}
 * Open System Settings panel
 * Wait for it to say there is an update available.
 * Click on ''Updates available''
 * Click on ''Install 1 update''
 * Click on ''Install & restart''
 * After the reboot, open System Settings panel, then Updates.  It should report that your ''Software is up to date''
 * '''Restore automatic downloads!'''
   * {{{adb shell system-image-cli --set auto_download=1}}}

=== Test D -- Good Path Manual Download Current Packages ===

This tests standard upgrading like above, but it uses the current set of packages and ''fakes'' the system upgrading to a newer version.  It actually upgrades from the newest version to the newest version because we're going to lie about what version the device is actually on.  This is a good test to run to ensure that the current latest build is working as expected.

 * Flash device to the current version, e.g. {{{ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --bootstrap}}}
 * Enable manual downloads:
   * {{{adb shell system-image-cli --set auto_download=0}}}
 * Falsify which version your device thinks it's on:
   * {{{adb shell}}}
   * {{{cd /etc/system-image}}}
   * {{{cat channel.ini}}}
   * Look at the {{{build_number}}} line to get the current build number.  Subtract 1.  (Next line assumes you're at build 246.)
   * {{{sed -e "s/246/245/" -i.bak channel.ini}}}
   * (If this gets messed up, just restore {{{channel.ini}}} from {{{channel.ini.bak}}})
 * Open System Settings panel.
 * Wait for it to say there is an update available.
 * Click on ''Updates available''
 * Click on ''Install 1 update''
 * Wait for updates to finish downloading.
 * Click on ''Install & restart''
 * After the reboot, open System Settings panel, then Updates.  It should report that your ''Software is up to date''
 * '''Restore automatic downloads!'''
   * {{{adb shell system-image-cli --set auto_download=1}}}

=== Test E -- Auto Download Multiple Checks ===

 * Flash device to an older version than the current one, e.g. {{{ubuntu-device-flash --channel ubuntu-touch/ubuntu-rtm/14.09-proposed --revision -1 --bootstrap}}}
 * Open System Settings panel
 * After one second, open Updates panel
 * Watch for progress
 * If update completes and you are prompted to reboot, then the test succeeded
 * Optionally reboot
 * Install the update, let the system reboot, and then try again.  This time you should be up-to-date.

=== Test F ===

 * Set the device to download manually.
 * Flash to an older revision
 * Open ''System Settings'' and wait for it to say ''Updates available''
 * Click on ''About this phone''
 * Click on ''Check for Update'' and wait for it to say ''Install 1 update''
 * Click on ''Install 1 update'' and while the files are downloading, click ''Back''
 * Click on ''Check for Update'' again
 * Wait for the ''Update System'' overlay to come up, and then install the update, and reboot

=== Test G -- Phased Updates ===

This test verifies the operation of ''phased updates''.  Phased updates means that a percentage is assigned to an end-point target image, and only that percentage of devices should be updated to that image.  Devices with a percentage outside that range will fall back to the previous image.  The server guarantees that only end-point images will have a phased update other than 100% (and the absence of a {{{phased-percentage}}} key in the JSON is equivalent to 100%).  The client ignores phased percentages on any mid-point image in an upgrade path.  An image with a phased percentage of 0% is never selected.

As an example, let's say that channel 'devel' has an image 10 and a new image 11 is released with a phased percentage of 25%.  Each device on channel 'devel' calculates its own percentage (its ''guinea pig'' value) from the following tuple of information:

 * The unique device UUID
 * The update channel (e.g. 'devel' in this example)
 * The highest target image number of all candidate upgrade paths (e.g. 11 in this example)

Let's say the device calculates its percentage as 17%; this device would elect to apply the image 11 update.  Let's say another device calculates its percentage as 33%; this device would elect to ''ignore'' the image 11 update.

When there are multiple upgrade candidate paths available, the client scores each candidate independent of the phased percentage, and sorts them in winning order from lowest score to highest.  If the lowest scored path has a phased percentage lower than the device's percentage, this candidate is discarded and the next lowest scoring candidate is considered.  This continues until a candidate upgrade path with a phased percentage greater than or equal to the device's percentage; this candidate is then selected as the winning upgrade path.  If no candidate matches this criteria, the device is considered up-to-date.

A missing phase percentage means every device should update; it's equivalent to an image percentage of 100%.  An image percentage of 0% means that the image has been effectively ''pull'' from release, and no devices will update to it.

To test this scenario, we need a staging server with a known end-point update on a particular channel with a non-100% phase percentage.  We currently have available:

 * host {{{https://phablet.stgraber.org}}}
 * channel {{{ubuntu-touch/test}}}
 * device {{{mako}}}

You also need a way to override the device's default phase percentage calculation, which can be done with {{{system-image-cli -p}}}.

'''Performing the test'''

 * {{{cd /etc/system-image}}}
 * Grab the staging server's archive master keys:
   * ```mv archive-master.tar.xz{,.aside}```
   * ```mv archive-master.tar.xz.asc{,.aside}```
   * ```wget https://phablet.stgraber.org/gpg/archive-master.tar.xz```
   * ```wget https://phablet.stgraber.org/gpg/archive-master.tar.xz.asc``
 * Edit /etc/system-image/channel.ini (''not'' {{{client.ini}}})
   * Change {{{[service]base}}} to {{{phablet.stgraber.org}}}
 * Do a ''dry-run'' update with your device's natural phased percentage.
   * ```system-image-cli -c "ubuntu-touch/test" -d mako -b 0 --dry-run -v```
   * You will either see an upgrade path to image 1 (with the device's phased percentage included in the output), or the device will claim to be up-to-date
 * Do a dry-run update with a forced phase percentage that is within image 1's phase:
   * ```system-image-cli -c "ubuntu-touch/test" -d mako -b 0 --dry-run -v -p 33```
   * You should see upgrade path to image 1 and a target phase of 33%
 * Do a dry-run update with a forced phase percentage outside the image phase:
   * ```system-image-cli -c "ubuntu-touch/test" -d mako -b 0 --dry-run -v -p 66```
   * The device will claim to be up-to-date.
 * Try some other dry-run updates with various ```-p``` options.  The image's phase is 40%

'''Be sure to restore your archive-master files and channel.ini file if you want to do other tests.'''

== Additional tests candidates for automation ==
The following test cases are being automated. Work items are described in this [[https://blueprints.launchpad.net/ubuntu/+spec/qa-t-system-image-upgrade-testing|blueprint]].

=== Testing the image update path ===
 * Test that updates from image n-1 work
 * Test that updates from image n-2 work

=== Testing the image updater software ===
 * Test that the updater software in image n can do an update to a single newer version in a synthetic channel
 * Test that the updater software in image n can do a multi-step update (applying 5? updates in sequence) in a synthetic channel
 * Test that the updater software in image n correctly applies a full-image update when it’s a smaller download than a series of stacked partial upgrades