NattyGraphicsTestPlan

Differences between revisions 1 and 42 (spanning 41 versions)
⇤ ← Revision 1 as of 2010-11-02 15:40:56 →
Size: 3154
Editor: i59F7B502
Comment: Added Draft for Natty TestPlan
← Revision 42 as of 2011-03-14 09:07:31 → ⇥
Size: 3914
Editor: ACaen-151-1-57-137
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
/!\ '''DRAFT'''
== Summary ==		 ## page was renamed from QATeam/NattyProprietaryXDriversTestPlan
## page was copied from QATeam/NattyTestPlan
||<tablestyle="float:right; font-size: 0.9em; width:40%; background:#F1F1ED; margin: 0 0 1em 1em;" style="padding:0.5em;"><<TableOfContents>>||
Line 4: Line 5:
The purpose of this document is to explain the different testing efforts for Unity in the Natty cycle. = Introduction =
Line 6: Line 7:
== Automated Testing == == Objectives ==
The objective of this document is to define a test plan for ATI/NVidia proprietary driver testing and graphics and performance in Unity.
Line 8: Line 10:
=== Unity itself === == Scope ==
The testing tasks includes:
 * Testing of ATI and NVidia proprietary drivers
 * Test cases that are known for:
   * Crashes
   * Visual corruption
   * Performance impact
Line 10: Line 18:
Although the DX team is the main responsible of testing the Unity interface itself, we need to collaborate with them so our tests work better. For each testing task we will also:
 * Collect test data and review testing results.
 * Reporting of all test results.
 * Risk analysis and communication of the blocking issues.
Line 12: Line 23:
Unity itself is not accessible, so the Unity interface cannot be accessed directly with the tools we have right now. Luckily, [[http://mago.ubuntu.com|Mago]] is tool agnostic and, although is mostly based on [[http://ldtp.freedesktop.org/|LDTP]], we can add other tools to help us testing the interface itself. == Deliverables ==
 * This test plan
 * At the end of each testing milestone QA will provide a test report for that testing cycle.
 * All defects found will be reported in Launchpad.
Line 14: Line 28:
The Unity team will provide with a dbus interface that will allow us knowing the state of the interface at any given time. That will help us checking if the icons are showing correctly. = X Drivers Testing =
Line 16: Line 30:
[[https://wiki.ubuntu.com/Xpresser|Xpresser]], which is based in image comparison, is not suitable to check the state of Unity after running a test, as the design will change a lot during the Natty cycle. There are 11 test cases covering all aspects of the ATI and NVidia proprietary drivers, from installation, basic and standard functionalities (resolutions, rotation, monitor hot plugging,...) and removal of the drivers.
Line 18: Line 32:
=== Applications on Unity === They are available from [[http://testcases.qa.ubuntu.com/Hardware/X/ProprietaryDrivers|testcases.qa.ubuntu.com]]
Line 20: Line 34:
The applications on Unity can be tested using the same a11y tools that we already use for the 2D experience. One of the main caveats is that Unity does not run under Xvfb and, therefore, we cannot run it on a server without having a full desktop up and running. = Performance and reliability =
Line 22: Line 36:
 * Research on how to run the tests using Hudson (or something else) in an automated fashion
 * Run the mago tests on Unity twice per week in a machine with Unity running as mean time solution
 * The tests results will be publicly available		 == Running Autopilot ==
Line 26: Line 38:
Xpresser, although based in image comparison, can be good to launch applications from the launcher, as it is not accessible yet, and the icons don't tend to change that much. We need to be careful, though, as Xpresser is not being actively maintained and there could be some bugs. '''Case ID: ''' upr-001
 1. Open a terminal
 1. Run the command:
   * /usr/lib/unity/autopilot‎.py
 1. Check that the result of the test is PASS
Line 28: Line 44:
=== 2D Experience === == Compiz Scale ==
Line 30: Line 46:
The 2D experience needs to be tested as well and, actually, as the 2D experience runs well in a Xvfb, it will be easier to run using Hudson, without '''Case ID: ''' upr-002
Line 32: Line 48:
==== ACTIONS ====
 * Run a Hudson server on the datacenter
 * Run a cronjob to update the system on a daily basis
 * Run the mago tests on a daily basis in an automated fashion		  1. Ubuntu Unity desktop, complete with decorations
 1. Open up 2 instances of nautilus (both maximised)
 1. Open up a gnome terninal window (unmaximised) - this is our visual test case
 1. On Unity dock - Double click nautilus icon
 1. Scale plugin activated
 1. Focus back on terminal window,
 1. Verify that the decorations are present
Line 37: Line 56:
== Manual Testing == = Target Hardware =
== NVIDIA Cards ==
 * GeForce 8m (8xxxM)
 * GeForce 9m (9xxxM)
 * GeForce 100M (1xxM)
 * GeForce 200M (2xxM)
 * GeForce 300M (3xxM)
 * GeForce 400M (4xxM)
 * GeForce 500M (5xxM)
Line 39: Line 66:
=== ISO Testing ===  * + Mobile cards
 * And more generally all the graphics chips that are sold today by NVidia have the harware capabilities we need to run Unity. We can safely go as far as 5 years and the GPUs that were made by either of these companies will still offer the hardware features we need.
Line 41: Line 69:
As part as our per milestone testing, we need to be testing Unity on desktops and the 2D experience fallback.
Many of us use VMs to test the images and we need to find a VM technology capable of running Unity. 		== ATI Cards ==
 * R6xx (HD 2xxx, HD 3xxx)
 * R7xx (HD 4xxx)
 * R8xx (HD 5xxx) "Evergreen" series
 * (HD 6xxx) "Northern Islands" series
Line 44: Line 75:
Also, we need to modify the testcases (or add new ones) to match the new possibilities.  * + Mobile cards
 * And more generally all the graphics chips that are sold today by AMD have the harware capabilities we need to run Unity. We can safely go as far as 5 years and the GPUs that were made by either of these companies will still offer the hardware features we need.
Line 46: Line 78:
==== ACTIONS ====
 * Communication with the installer and design teams to modify the testcases
 * Include the new testcases into the ISO tracker		 = Acceptance Criteria =
== Alpha ==
The Phase 1 Alpha deliverable will be considered met when all the basic functionality of the software and hardware is complete. The client will have the ability to test the deliverable software and hardware functionality. There may be defects in this milestone that will be addressed in the next Phase. Performance and reliability requirements are out of scope for the Alpha release.
Line 50: Line 82:
=== Desktop Testing Program === The image may have Critical, High and Medium bugs.
Line 52: Line 84:
The Desktop Testing Program started last cycle and it provided a good way for the community to test the desktop applications every milestone. As part of the modifications for this program in the Natty cycle, we will be including Unity as part of the program. == Acceptance Criteria: Phase 2 - Beta ==
The Phase 2 Beta release will be considered feature complete based on the Statement of Work. This image will also include bug fixes. The image may have a limited number of Critical bugs. Performance and reliability requirements are acceptable for the Beta release.
Line 54: Line 87:
For each of the applications, we will be duplicating the testcases, to be run in Unity and also the 2D experience. == Acceptance Criteria: Phase 3 – Final Release ==
This image will be the Final release.

All features will be complete.
There are no Critical bugs. A minimum of High level bugs may be present.

Introduction

Objectives

The objective of this document is to define a test plan for ATI/NVidia proprietary driver testing and graphics and performance in Unity.

Scope

The testing tasks includes:

  • Testing of ATI and NVidia proprietary drivers
  • Test cases that are known for:
    • Crashes
    • Visual corruption
    • Performance impact

For each testing task we will also:

  • Collect test data and review testing results.
  • Reporting of all test results.
  • Risk analysis and communication of the blocking issues.

Deliverables

  • This test plan
  • At the end of each testing milestone QA will provide a test report for that testing cycle.
  • All defects found will be reported in Launchpad.

X Drivers Testing

There are 11 test cases covering all aspects of the ATI and NVidia proprietary drivers, from installation, basic and standard functionalities (resolutions, rotation, monitor hot plugging,...) and removal of the drivers.

They are available from testcases.qa.ubuntu.com

= Performance and reliability =

Running Autopilot

Case ID: upr-001

  1. Open a terminal
  2. Run the command:
    • /usr/lib/unity/autopilot‎.py
  3. Check that the result of the test is PASS

Compiz Scale

Case ID: upr-002

  1. Ubuntu Unity desktop, complete with decorations
  2. Open up 2 instances of nautilus (both maximised)
  3. Open up a gnome terninal window (unmaximised) - this is our visual test case
  4. On Unity dock - Double click nautilus icon
  5. Scale plugin activated
  6. Focus back on terminal window,
  7. Verify that the decorations are present

Target Hardware

NVIDIA Cards

  • GeForce 8m (8xxxM)

  • GeForce 9m (9xxxM)

  • GeForce 100M (1xxM)

  • GeForce 200M (2xxM)

  • GeForce 300M (3xxM)

  • GeForce 400M (4xxM)

  • GeForce 500M (5xxM)

  • + Mobile cards
  • And more generally all the graphics chips that are sold today by NVidia have the harware capabilities we need to run Unity. We can safely go as far as 5 years and the GPUs that were made by either of these companies will still offer the hardware features we need.

ATI Cards

  • R6xx (HD 2xxx, HD 3xxx)
  • R7xx (HD 4xxx)
  • R8xx (HD 5xxx) "Evergreen" series
  • (HD 6xxx) "Northern Islands" series
  • + Mobile cards
  • And more generally all the graphics chips that are sold today by AMD have the harware capabilities we need to run Unity. We can safely go as far as 5 years and the GPUs that were made by either of these companies will still offer the hardware features we need.

Acceptance Criteria

Alpha

The Phase 1 Alpha deliverable will be considered met when all the basic functionality of the software and hardware is complete. The client will have the ability to test the deliverable software and hardware functionality. There may be defects in this milestone that will be addressed in the next Phase. Performance and reliability requirements are out of scope for the Alpha release.

The image may have Critical, High and Medium bugs.

Acceptance Criteria: Phase 2 - Beta

The Phase 2 Beta release will be considered feature complete based on the Statement of Work. This image will also include bug fixes. The image may have a limited number of Critical bugs. Performance and reliability requirements are acceptable for the Beta release.

Acceptance Criteria: Phase 3 – Final Release

This image will be the Final release.

All features will be complete. There are no Critical bugs. A minimum of High level bugs may be present.

QATeam/NattyGraphicsTestPlan (last edited 2011-03-17 17:01:13 by ACaen-151-1-93-135)