Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

• Launchpad Entry: desktop-automated-tests

• Packages affected:

Summary

Leverage the ATK framework to implement a set of automated test routines for the desktop.

Release Note

This release contains the beginnings of an automated desktop test suite for GNOME/GTK software that support accessibility. The test suite can be used by users as well as developers to verify that the installed system works.

Rationale

At the moment, pretty much all quality assurance related testing in Ubuntu is done manually. There are wiki pages with checklists of steps that testers follow to verify that at least basic functionality works. This works, but requires a lot of people to do the work, and it needs to be done many times. Obviously, an automated solution would be better.

Use Cases

• Aaron wants to upload a new version of the libgnomepanzerwagen package (a library that is used by many GUI programs), and wants to verify before the upload that it doesn't break anything. He installs the new version on his development machine, and runs the automated desktop tests for the entire distribution to see that everything still works.
• Bertha works on Ubuntu QA and wants to check that the systems installed by daily ISO builds work. She installs each of them in a virtual machine, and runs the desktop test suite to verify the results.
• Charlie supplies her customers with working systems based on Ubuntu, but also using some additional software she supplies herself. A new version of Ubuntu has been released, and she wants to make sure her customers can upgrade safely. She runs the upgrade on a test system, configured identically to those of her customers, and uses the test suite before and after to verify that nothing breaks. In addition, she has added some tests of her own, to verify that her own software also survives the upgrade.
• Diogenes wants to help Ubuntu development and reports a bug he has found. To show how the bug can be reproduced, he starts a recording tool, does whatever is necessary to reproduce the bug, and the attaches the recording to the bug report. Developers can then use this to verify that the bug exists, and that their fix actually fixes the bug.

Assumptions

The test suite is run as a non-privileged user on an installed system. The live CD is one such environment, but any installed Ubuntu (GNOME) system should suffice. The system may run on real hardware or in a VM.

Design

Use the accessibility layer of the GTK/GNOME library stack (the ATK layer) to implement this. ATK allows programs to operate other programs more or less as if the user was operating them. There are several tools using this interface: LDTP, dogtail, Accerciser.

Test scripts for LDTP, dogtail or Accerciser can be written by hand, possibly with assistance of recording tools to speed up the process. These scripts will be run individually or chained together from an automated procedure that can start with initiating a fresh test environment (such as copying and booting a VM instance).

The test cases should concentrate on integration and functional aspects as well as more detailed testing of tools that are developed within the Ubuntu project, such as the installer, Jockey, update manager and display config tool. Detailed application specific tests of upstream applications should come from upstream developers, as part of the code.

Implementation

• Extract widget maps from key applications automatically after each new upload and make these available for download. Also flag changed maps and post diffs in some useful form.
• Write LDTP test scripts for Ubiquity
• Integrate existing LDTP test for Nautilus and Evolution

• Implement further desktop tests in Testing/Cases.

• Select tests from FixesToVerify will be written during hardy in cases where they are fairly simple to automate.

• Write a simple script to launch all the desktop files installed in /usr/share/applications/ and check for crashers and run this on a continual basis.

Outstanding Issues

• Programs are not usually designed to be tested this way. That results in recording tools working badly. Many widgets do not have names, for example. If this becomes an issue, bugs will be filed with upstream, possibly with patches.
• Sharing tests with upstream developers and other distributions would be a good thing, both ways. Novell, for example, has test cases for Evolution and perhaps other programs as well and uses LDTP. It will be necessary to work together with others to develop a culture for encouraging tests to be written, and for applications to be designed to be tested. A couple of scripts are already available on the LDTP page for Nautilus and Evolution, however they need to be updated.
• Image comparison probably won't work because there are expected desktop theme changes going forward.

Notes

• Using ATK currently means that only the GTK based Ubuntu flavors can be tested. KDE4 is said to get an accessibility layer as well, but it is not yet known to us whether that will be compatible with ATK or not.
• Recording tools used for this project could also be used by users to record scripts to show bugs.