NattyGraphicsTestPlan
Differences between revisions 15 and 50 (spanning 35 versions)
|
Size: 10244
Comment:
|
← Revision 50 as of 2011-03-17 17:01:13 ⇥
Size: 3439
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| /!\ '''DRAFT''' |
## page was renamed from QATeam/NattyProprietaryXDriversTestPlan ## page was copied from QATeam/NattyTestPlan |
| Line 6: | Line 6: |
| == Purpose == The purpose of this document is to explain the different testing efforts for Unity in the Natty cycle. It is designed to prescribe the scope, approach, resources, and schedule for all testing activities during this cycle. |
|
| Line 9: | Line 7: |
| This document will be modified and enhanced as required throughout the verification testing engagement. The purpose of this document: * Defines the overall test approach. * Identifies software and system hardware to be tested. * Identifies hardware, software, and tools to be used to support the testing efforts. * Defines the types of tests to be performed. ''Note: Specific effort for regression testing ?'' |
== Objectives == The objective of this document is to define a test plan for ATI/NVidia proprietary driver testing, graphics and performance in Unity. |
| Line 19: | Line 11: |
The testing tasks for the components of Ubuntu includes: * Functional testing of the applications (automated and manual - Desktop Testing initiative) * Integration with the desktop environment * Unity * 2D Desktop (gnome fallback when 3D support is not available) * Server Testing * ISO Testing * Hardware Testing * Additional Testing: i18n, a11y, printing |
The testing tasks includes: * Testing of ATI and NVidia proprietary drivers * Test cases that are known for: * Crashes * Visual corruption * Performance impact |
| Line 31: | Line 19: |
| * Collect, provide and review Test Cases that uncover new bugs or cover existing known issues. | |
| Line 36: | Line 23: |
| Testing will be performed all along during the development cycle of Natty. The main milestones of the test plan are aligned on the [[https://wiki.ubuntu.com/NattyReleaseSchedule|Release Schedule]] Testing is a very dependant activity, As a result, test planning is a continuing activity performed throughout the system development life cycle. | == Deliverables == * This test plan * At the end of each testing milestone QA will provide a test report for that testing cycle. * All defects found will be reported in Launchpad. |
| Line 38: | Line 28: |
| ''Note: HW Testing -> Cert Team: to be included in this doc ?'' | == Testing procedure == |
| Line 40: | Line 30: |
| = Test Items and Features = == Application == === Category of applications to be tested === |
The testing procedure and call for testing is available from the [[https://wiki.ubuntu.com/X/Testing/ProprietaryDrivers/Natty/WeeklyProgram|X Testing Wiki]] |
| Line 44: | Line 32: |
| The categories of applications to be tested are: * Core components (Help, menus, ...) * Audio/Video * Games * Graphics * Internet * Office * Utilities * System Preferences * System Administration |
= X Drivers Testing = |
| Line 55: | Line 34: |
| [[https://wiki.ubuntu.com/QATeam/NattyTestPlan/AppList|Each application]] must be covered by at least 1 automated test case to ensure that it can be launched on the platform. Additionally, because using manual testing will cover very few test cases and requires important man power, we will focus on test cases that uncover new bugs and cover existing known issues. | There are 11 test cases covering all aspects of the ATI and NVidia proprietary drivers, from installation, basic and standard functionalities (resolutions, rotation, monitor hot plugging,...) and removal of the drivers. |
| Line 57: | Line 36: |
| '''ACTION''' * Work with bdmurray and pedro to do some datamining in launchpad and identify known issues for the aforementioned applications'' |
They are available from [[http://testcases.qa.ubuntu.com/Hardware/X/ProprietaryDrivers|testcases.qa.ubuntu.com]] |
| Line 60: | Line 38: |
| === Automated Testing === | = Unity Graphics Performance and reliability = |
| Line 62: | Line 40: |
| We will use Mago (lp:mago) and its helper tools magomatic (lp:magomatic) to automate test cases. | These test cases are available from http://testcases.qa.ubuntu.com/Hardware/X/GraphicsVideoPerformance |
| Line 64: | Line 42: |
| Unity itself is not accessible, so the Unity interface cannot be accessed directly with the tools we have right now. Luckily, [[http://mago.ubuntu.com|Mago]] is tool agnostic and, although is mostly based on [[http://ldtp.freedesktop.org/|LDTP]], we can add other tools to help us testing the interface itself. | = Target Hardware = == NVIDIA Cards == GPUs such as GeForce series 6 or newer are supported and have the hardware capabilities needed to run Unity. |
| Line 66: | Line 46: |
| '''ACTIONS''' * Investigate XPresser (Limited to image matching) * xautomation (Lack of 'record' feature ?) * xnee/cnee (Crashes desktop in Natty) * Tools based on XTEST |
For a complete list, install the package nvidia-current and see /usr/share/doc/nvidia-current/README.txt.gz |
| Line 72: | Line 48: |
| === Manual Testing === | == ATI Cards == * R6xx (HD 2xxx, HD 3xxx) * R7xx (HD 4xxx) * R8xx (HD 5xxx) "Evergreen" series * (HD 6xxx) "Northern Islands" series |
| Line 74: | Line 54: |
| The Desktop Testing Program started last cycle. It provided a good way for the community to test the desktop applications every milestone. As part of the modifications for this program in the Natty cycle, we will be including Unity as part of the program. | * + Mobile cards * And more generally all the graphics chips that are sold today by AMD have the hardware capabilities we need to run Unity. We can safely go as far as 5 years and the GPUs that were made by either of these companies will still offer the hardware features we need. |
| Line 76: | Line 57: |
| For each of the applications, we will be duplicating the testcases, to be run in Unity and also the 2D experience. | = Acceptance Criteria = == Acceptance Criteria: Phase 1 - Alpha == The Phase 1 Alpha deliverable will be considered met when all the basic functionality of the software and hardware is complete. The client will have the ability to test the deliverable software and hardware functionality. There may be defects in this milestone that will be addressed in the next Phase. Performance and reliability requirements are out of scope for the Alpha release. |
| Line 78: | Line 61: |
| The resources available to the community are: * [[http://testcases.qa.ubuntu.com/| Test Cases wiki]] to be extended with the following [[http://people.canonical.com/~j-lallement/natty/testcases.html|additional test cases]] * The [[http://desktop.qa.ubuntu.com/| Desktop Testing Tracker]] |
The image may have Critical, High and Medium bugs. |
| Line 82: | Line 63: |
| == Desktop == === Unity === |
== Acceptance Criteria: Phase 2 - Beta == The Phase 2 Beta release will be considered feature complete based on the Statement of Work. This image will also include bug fixes. The image may have a limited number of Critical bugs. Performance and reliability requirements are acceptable for the Beta release. |
| Line 85: | Line 66: |
| Although the DX team is the main responsible of testing the Unity interface itself, we need to collaborate with them so our tests work better. The QA Team is responsible of the testing of the integration of the application with Unity (ex. Shotwell Menu not integration in the global menu) | == Acceptance Criteria: Phase 3 – Final Release == This image will be the Final release. |
| Line 87: | Line 69: |
| The DX team will provide us "only dbus access, not dbusmenu. The D-Bus Debug Service will provide a service to collect the state of the Unity interface i.g Panel, Menu, Indicator, Files place, ... Based on that, we can send dbus commands to compiz/unity to know the state of the interface without atk support. If we want to test global menu bits, e.g. the global /appmenu/, we can do so with the gnome-panel version which call 'hidden' menus in Unity. The tests should be easily transposable to the Unity version once that menu gets atk support as well. All the applications shipped by default with Natty must be tested in Unity. The applications on Unity can be tested using the same a11y tools that we already use for the 2D experience. One of the main caveats is that Unity does not run under Xvfb and, therefore, we cannot run it on a server without having a full desktop up and running. The subsystems of Unity that needs to be covered are * [[http://testcases.qa.ubuntu.com/Applications/Unity|Basic manual Test Cases]] * [[https://wiki.ubuntu.com/JeanBaptisteLallement/SandBox/NattyTestCases|Launcher]] * Applications * Notifications * Messaging Menu * MeMenu * Sound Menu * Files An example of integration Test Case is: * create a document in gedit, * verify that it appears in the Files Place. * Then rename that document * verify that the document modified appears in places. '''ACTIONS''' * Research on how to run the tests using Hudson (or something else) in an automated fashion * Run the mago tests on Unity at least twice per week in a machine with Unity running as mean time solution * The tests results will be publicly available '''Additional tool to extend Mago''' [[https://wiki.ubuntu.com/Xpresser|Xpresser]], which is based in image comparison, is not suitable to check the state of Unity after running a test, as the design will change a lot during the Natty cycle. Xpresser, although based in image comparison, can be good to launch applications from the launcher, as it is not accessible yet, and the icons don't tend to change that much. We need to be careful, though, as Xpresser is not being actively maintained and there could be some bugs. === 2D Experience === All the applications must be tested in the 2D (legacy) environment. The 2D experience needs to be tested as well and, actually, as the 2D experience runs well in a Xvfb, it will be easier to run using Hudson. The subsystems of the 2D experience that needs to be covered are * Notifications * Messaging Menu * MeMenu * ApplicationIndicators * Application Menu * Sound Menu '''ACTIONS''' * Run a Hudson server on the datacenter * Run a cronjob to update the system on a daily basis * Run the mago tests on a daily basis in an automated fashion == Support Features == The following additional component are part of the Testing effort: * Accessibility (see with charlie_tca which items and features need to be reviewed) * Translation (see with dpm) * Printing (tkampeter was expected to provide test cases) == Server == Although we now have expanded hardware support on Natty (AMD64, AMD64-mac, i386, PowerPC, PowerPC-ps3, ARMEL-dove, possibly others?) we will test only i386 and AMD64. This expanded support is mostly a consolidation of the different ports -- see [[https://blueprints.launchpad.net/ubuntu/+spec/ubuntutheproject-foundations-n-cdimage-ports-consolidation|Natty cdimage ports consolidation]]. === Server itself === Expand tool-chain testing with more packages (currently we daily-build libvirt, postgresql-8.4, mysql-dfsg-5.1, openldap, php5, python2.6, and coreutils). === Applications === === Cloud === We will test both Eucalyptus and Openstack; amount of coverage of Openstack will depend on availability. * add euca2ools tests * verify euca2ools compatibility with EC2, and other Cloud technologies * automate as much as possible the UEC tests currently run (depends on updates on euca-* utilities) * test image compatibility (Lucid, Maverick, Natty) with EUC-Natty == Hardware Testing == === HW Team === * Any input from the HW Team ? === Laptop Testing Initiative === * [[https://blueprints.launchpad.net/ubuntu/+spec/ubuntutheproject-qa-n-laptop-testing-tracker|Laptop Testing BluePrint]] (From the Ubuntu-IT Team) We are going to set up a new tracker to track the results (http://laptop.qa.ubuntu.com). The Italian Testing team will be the main responsible to maintain the milestones and to provide reports. == ISO Testing == As part as our per milestone testing, we need to be testing Unity on desktops and the 2D experience fallback. Many of us use VMs to test the images and we need to find a VM technology capable of running Unity. Also, we need to modify the testcases (or add new ones) to match the new possibilities. === Arch / Flavour ? === * AMD64 * i386 ''' Other Reference ''' * [[https://blueprints.launchpad.net/ubuntu/+spec/other-qa-n-testing-different-architectures|Testing on different architectures]] === Schedule === '''ACTIONS''' * Communication with the installer and design teams to modify the testcases * Include the new testcases into the ISO tracker == Feature not to be tested == = Community Involvement = == Engage the LoCo Teams == Additionally to the Desktop Testing program and the Laptop Testing initiative, we will rely on the following efforts: * [[https://blueprints.launchpad.net/ubuntu/+spec/ubuntutheproject-qa-n-loco-testing|LoCo Testing Blueprint]] * [[https://wiki.ubuntu.com/Testing/LoCoTeam|LoCo Testing Wiki]] = General Schedule = '''ACTION''' * Create general schedule based on [[https://wiki.ubuntu.com/NattyReleaseSchedule|Natty Release Schedule]] |
All features will be complete. There are no Critical bugs. A minimum of High level bugs may be present. |
Contents |
Introduction
Objectives
The objective of this document is to define a test plan for ATI/NVidia proprietary driver testing, graphics and performance in Unity.
Scope
The testing tasks includes:
- Testing of ATI and NVidia proprietary drivers
- Test cases that are known for:
- Crashes
- Visual corruption
- Performance impact
For each testing task we will also:
- Collect test data and review testing results.
- Reporting of all test results.
- Risk analysis and communication of the blocking issues.
Deliverables
- This test plan
- At the end of each testing milestone QA will provide a test report for that testing cycle.
- All defects found will be reported in Launchpad.
Testing procedure
The testing procedure and call for testing is available from the X Testing Wiki
X Drivers Testing
There are 11 test cases covering all aspects of the ATI and NVidia proprietary drivers, from installation, basic and standard functionalities (resolutions, rotation, monitor hot plugging,...) and removal of the drivers.
They are available from testcases.qa.ubuntu.com
Unity Graphics Performance and reliability
These test cases are available from http://testcases.qa.ubuntu.com/Hardware/X/GraphicsVideoPerformance
Target Hardware
NVIDIA Cards
GPUs such as GeForce series 6 or newer are supported and have the hardware capabilities needed to run Unity.
For a complete list, install the package nvidia-current and see /usr/share/doc/nvidia-current/README.txt.gz
ATI Cards
- R6xx (HD 2xxx, HD 3xxx)
- R7xx (HD 4xxx)
- R8xx (HD 5xxx) "Evergreen" series
- (HD 6xxx) "Northern Islands" series
- + Mobile cards
- And more generally all the graphics chips that are sold today by AMD have the hardware capabilities we need to run Unity. We can safely go as far as 5 years and the GPUs that were made by either of these companies will still offer the hardware features we need.
Acceptance Criteria
Acceptance Criteria: Phase 1 - Alpha
The Phase 1 Alpha deliverable will be considered met when all the basic functionality of the software and hardware is complete. The client will have the ability to test the deliverable software and hardware functionality. There may be defects in this milestone that will be addressed in the next Phase. Performance and reliability requirements are out of scope for the Alpha release.
The image may have Critical, High and Medium bugs.
Acceptance Criteria: Phase 2 - Beta
The Phase 2 Beta release will be considered feature complete based on the Statement of Work. This image will also include bug fixes. The image may have a limited number of Critical bugs. Performance and reliability requirements are acceptable for the Beta release.
Acceptance Criteria: Phase 3 – Final Release
This image will be the Final release.
All features will be complete. There are no Critical bugs. A minimum of High level bugs may be present.
QATeam/NattyGraphicsTestPlan (last edited 2011-03-17 17:01:13 by ACaen-151-1-93-135)