Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

• Launchpad Entry: mobile-automated-tests

• Packages affected:

Summary

This spec outlines the investigation of automated test frameworks for the mobile platform. The automation of test cases for Ubuntu Mobile Edition would allow for increased testing coverage with limited resources. A successful automation effort must follow a well defined process, with solid guidelines and goals. Community involvement is also key to this effort's success and the process should be carefully designed so that community members can easily contribute.

Use Cases

• Check Moblin Compliance (1.0, 2.0, etc.)
  • Mobile must be compliant and the test is an excellent basis
• Validate basic functionality of application stack
  • Application stack will shift, but core apps will always be present
• Use hwtest-verify to collect information about the mobile platform.

For a detailed testcase see:Mobile_case

Tools Available

• Dogtail
• LDTP
• VNC
• DejaGNU
• Autotest/LTP

Analysis

• Dogtail - Had issues with hildon and potentially touchscreen
• LDTP - Best option, verbal commitment from dev to help with implementation.
• VNC - Tools on horizon for automating in VM
• DejaGNU - Gnash is currently using to automate.
• Autotest/LTP

With the creation of a suite of automation based tests, great care must be taken when determining the depth and breath of the cases. If the cases exercise functionality that is subject to rapid change, the debugging and repair of broken scripts could end up becoming a maintenance headache.

Sleep/Hibernate testing will be difficult to implement due to the fact that only device based testing can recreate the various sleep states. If this area of testing is pursued, a method for automating wake actions will need to be determined.

Implementation Solutions

• Community Involvement
  • The creation and maintenance of automated scripts should be promoted in the community heavily. This will reduce the load on QA, and shift our responsibility from owner to reviewer. Community involvement should be explored as early as possible.
• Creation of Example Scripts
  • Detailed example scripts should be created and published. This will further reduce the effort required to maintain the automation suite by ensuring each script follows a well defined format.
• Best practice Guidelines
  • In addition to the sample scripts, a guideline document should also be drafted and published. This document should contain information on existing script usage, new script formatting, QA process outlines.
• Integration into Launchpad
  • We should explore the integration of testing into Launchpad. Automated scripts is a good place to begin because they are code and we can take advantage of the various resources available to us when checking code into Launchpad. These include integration with projects, source control, and linking to bugs.
• Ship Ubuntu version with tools for recording
  • Once developed, this automated test suite should be included in the main distro or as a simple single package that can be easily added by the community for testing.
• Recording - Exerciser - analyse scripts output by user
  • Exploration of gaining access directly to users script output should also be undertaken. With a facility similar to apport, QA would be able to easily gather information about failures within the test suite.

Next Steps

• Determine the cases/depth of cases for what needs to be automated
• Targets for testing (apps)
• Examine LDTP compatability