NattyGraphicsTestPlan

Differences between revisions 3 and 50 (spanning 47 versions)
⇤ ← Revision 3 as of 2010-11-02 15:42:44 →
Size: 3182
Editor: i59F7B502
Comment:
← Revision 50 as of 2011-03-17 17:01:13 → ⇥
Size: 3439
Editor: ACaen-151-1-93-135
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
/!\ '''DRAFT'''
== Summary ==		 ## page was renamed from QATeam/NattyProprietaryXDriversTestPlan
## page was copied from QATeam/NattyTestPlan
||<tablestyle="float:right; font-size: 0.9em; width:40%; background:#F1F1ED; margin: 0 0 1em 1em;" style="padding:0.5em;"><<TableOfContents>>||
Line 4: Line 5:
The purpose of this document is to explain the different testing efforts for Unity in the Natty cycle. = Introduction =
Line 6: Line 7:
== Automated Testing == == Objectives ==
The objective of this document is to define a test plan for ATI/NVidia proprietary driver testing, graphics and performance in Unity.
Line 8: Line 10:
=== Unity itself === == Scope ==
The testing tasks includes:
 * Testing of ATI and NVidia proprietary drivers
 * Test cases that are known for:
   * Crashes
   * Visual corruption
   * Performance impact
Line 10: Line 18:
Although the DX team is the main responsible of testing the Unity interface itself, we need to collaborate with them so our tests work better. For each testing task we will also:
 * Collect test data and review testing results.
 * Reporting of all test results.
 * Risk analysis and communication of the blocking issues.
Line 12: Line 23:
Unity itself is not accessible, so the Unity interface cannot be accessed directly with the tools we have right now. Luckily, [[http://mago.ubuntu.com|Mago]] is tool agnostic and, although is mostly based on [[http://ldtp.freedesktop.org/|LDTP]], we can add other tools to help us testing the interface itself. == Deliverables ==
 * This test plan
 * At the end of each testing milestone QA will provide a test report for that testing cycle.
 * All defects found will be reported in Launchpad.
Line 14: Line 28:
The Unity team will provide with a dbus interface that will allow us knowing the state of the interface at any given time. That will help us checking if the icons are showing correctly. == Testing procedure ==
Line 16: Line 30:
[[https://wiki.ubuntu.com/Xpresser|Xpresser]], which is based in image comparison, is not suitable to check the state of Unity after running a test, as the design will change a lot during the Natty cycle. The testing procedure and call for testing is available from the [[https://wiki.ubuntu.com/X/Testing/ProprietaryDrivers/Natty/WeeklyProgram|X Testing Wiki]]
Line 18: Line 32:
=== Applications on Unity === = X Drivers Testing =
Line 20: Line 34:
The applications on Unity can be tested using the same a11y tools that we already use for the 2D experience. One of the main caveats is that Unity does not run under Xvfb and, therefore, we cannot run it on a server without having a full desktop up and running. There are 11 test cases covering all aspects of the ATI and NVidia proprietary drivers, from installation, basic and standard functionalities (resolutions, rotation, monitor hot plugging,...) and removal of the drivers.
Line 22: Line 36:
==== ACTIONS ====
 * Research on how to run the tests using Hudson (or something else) in an automated fashion
 * Run the mago tests on Unity at least twice per week in a machine with Unity running as mean time solution
 * The tests results will be publicly available		 They are available from [[http://testcases.qa.ubuntu.com/Hardware/X/ProprietaryDrivers|testcases.qa.ubuntu.com]]
Line 27: Line 38:
Xpresser, although based in image comparison, can be good to launch applications from the launcher, as it is not accessible yet, and the icons don't tend to change that much. We need to be careful, though, as Xpresser is not being actively maintained and there could be some bugs. = Unity Graphics Performance and reliability =
Line 29: Line 40:
=== 2D Experience === These test cases are available from http://testcases.qa.ubuntu.com/Hardware/X/GraphicsVideoPerformance
Line 31: Line 42:
The 2D experience needs to be tested as well and, actually, as the 2D experience runs well in a Xvfb, it will be easier to run using Hudson, without = Target Hardware =
== NVIDIA Cards ==
GPUs such as GeForce series 6 or newer are supported and have the hardware capabilities needed to run Unity.
Line 33: Line 46:
==== ACTIONS ====
 * Run a Hudson server on the datacenter
 * Run a cronjob to update the system on a daily basis
 * Run the mago tests on a daily basis in an automated fashion		 For a complete list, install the package nvidia-current and see /usr/share/doc/nvidia-current/README.txt.gz
Line 38: Line 48:
== Manual Testing == == ATI Cards ==
 * R6xx (HD 2xxx, HD 3xxx)
 * R7xx (HD 4xxx)
 * R8xx (HD 5xxx) "Evergreen" series
 * (HD 6xxx) "Northern Islands" series
Line 40: Line 54:
=== ISO Testing ===  * + Mobile cards
 * And more generally all the graphics chips that are sold today by AMD have the hardware capabilities we need to run Unity. We can safely go as far as 5 years and the GPUs that were made by either of these companies will still offer the hardware features we need.
Line 42: Line 57:
As part as our per milestone testing, we need to be testing Unity on desktops and the 2D experience fallback.
Many of us use VMs to test the images and we need to find a VM technology capable of running Unity. 		= Acceptance Criteria =
== Acceptance Criteria: Phase 1 - Alpha ==
The Phase 1 Alpha deliverable will be considered met when all the basic functionality of the software and hardware is complete. The client will have the ability to test the deliverable software and hardware functionality. There may be defects in this milestone that will be addressed in the next Phase. Performance and reliability requirements are out of scope for the Alpha release.
Line 45: Line 61:
Also, we need to modify the testcases (or add new ones) to match the new possibilities. The image may have Critical, High and Medium bugs.
Line 47: Line 63:
==== ACTIONS ====
 * Communication with the installer and design teams to modify the testcases
 * Include the new testcases into the ISO tracker		 == Acceptance Criteria: Phase 2 - Beta ==
The Phase 2 Beta release will be considered feature complete based on the Statement of Work. This image will also include bug fixes. The image may have a limited number of Critical bugs. Performance and reliability requirements are acceptable for the Beta release.
Line 51: Line 66:
=== Desktop Testing Program === == Acceptance Criteria: Phase 3 – Final Release ==
This image will be the Final release.
Line 53: Line 69:
The Desktop Testing Program started last cycle and it provided a good way for the community to test the desktop applications every milestone. As part of the modifications for this program in the Natty cycle, we will be including Unity as part of the program.

For each of the applications, we will be duplicating the testcases, to be run in Unity and also the 2D experience.		 All features will be complete.
There are no Critical bugs. A minimum of High level bugs may be present.

Introduction

Objectives

The objective of this document is to define a test plan for ATI/NVidia proprietary driver testing, graphics and performance in Unity.

Scope

The testing tasks includes:

  • Testing of ATI and NVidia proprietary drivers
  • Test cases that are known for:
    • Crashes
    • Visual corruption
    • Performance impact

For each testing task we will also:

  • Collect test data and review testing results.
  • Reporting of all test results.
  • Risk analysis and communication of the blocking issues.

Deliverables

  • This test plan
  • At the end of each testing milestone QA will provide a test report for that testing cycle.
  • All defects found will be reported in Launchpad.

Testing procedure

The testing procedure and call for testing is available from the X Testing Wiki

X Drivers Testing

There are 11 test cases covering all aspects of the ATI and NVidia proprietary drivers, from installation, basic and standard functionalities (resolutions, rotation, monitor hot plugging,...) and removal of the drivers.

They are available from testcases.qa.ubuntu.com

Unity Graphics Performance and reliability

These test cases are available from http://testcases.qa.ubuntu.com/Hardware/X/GraphicsVideoPerformance

Target Hardware

NVIDIA Cards

GPUs such as GeForce series 6 or newer are supported and have the hardware capabilities needed to run Unity.

For a complete list, install the package nvidia-current and see /usr/share/doc/nvidia-current/README.txt.gz

ATI Cards

  • R6xx (HD 2xxx, HD 3xxx)
  • R7xx (HD 4xxx)
  • R8xx (HD 5xxx) "Evergreen" series
  • (HD 6xxx) "Northern Islands" series
  • + Mobile cards
  • And more generally all the graphics chips that are sold today by AMD have the hardware capabilities we need to run Unity. We can safely go as far as 5 years and the GPUs that were made by either of these companies will still offer the hardware features we need.

Acceptance Criteria

Acceptance Criteria: Phase 1 - Alpha

The Phase 1 Alpha deliverable will be considered met when all the basic functionality of the software and hardware is complete. The client will have the ability to test the deliverable software and hardware functionality. There may be defects in this milestone that will be addressed in the next Phase. Performance and reliability requirements are out of scope for the Alpha release.

The image may have Critical, High and Medium bugs.

Acceptance Criteria: Phase 2 - Beta

The Phase 2 Beta release will be considered feature complete based on the Statement of Work. This image will also include bug fixes. The image may have a limited number of Critical bugs. Performance and reliability requirements are acceptable for the Beta release.

Acceptance Criteria: Phase 3 – Final Release

This image will be the Final release.

All features will be complete. There are no Critical bugs. A minimum of High level bugs may be present.

QATeam/NattyGraphicsTestPlan (last edited 2011-03-17 17:01:13 by ACaen-151-1-93-135)