##(see the SpecSpec for an explanation)

 * '''Launchpad Entry''': [[https://blueprints.edge.launchpad.net/checkbox/+spec/qa-ldtp-checkbox-integration|qa-ldtp-checkbox-integration]]
 * '''Created''': AraPulido
 * '''Contributors''': 
 * '''Packages affected''': checkbox, ubuntu-desktop-testing

== Summary ==

This should provide an overview of the issue/functionality/change proposed here. Focus here on what will actually be DONE, summarising that so that other people don't have to read the whole spec. See also CategorySpec for examples.

== Release Note ==

When these changes are implemented tests written for LDTP and, specifically, for Ubuntu Desktop Testing, Checkbox will be used to run desktop tests and reporting will be consistent.

== Rationale ==

We want to have a consistent test harness for the majority of our tests. Checkbox, formerly hwtest, it is a tool to run tests. It is flexible enough to be used to run all kind of tests, including desktop tests.

== Design ==

=== Permissions ===
We need Checkbox to be able to run as a normal user, rather than running always as root. This need changes from both Checkbox and Ubuntu Desktop Testing

 * Checkbox can be run as non-root:
{{{
./bin/checkbox-cli --config="checkbox/plugins/blacklist=permission_prompt" 
}}}
 * Suites and scripts are written on installation under /usr/share/checkbox, with root-only permissions. It would be better to have a .checkbox folder in the user's home folder to store this kind of information.
 * Results are stored by default in /var/lib/checkbox, and this is only writable by root.
 * Ubuntu Desktop Testing package installs scripts in /usr/share/ubuntu-desktop-tests only readable by root.

=== Test results ===

Right now test results in Checkbox only accept zero for pass, non-zero for a fail, in a unix-like manner. The problem is that if we want more complex reporting (i.e. screenshots when something fails) we need more flexibility here.
 
When running LDTP tests a XML file is generated. This should be parsed by maybe a Checkbox plugin in order to transform that answer to a Checkbox consistent format, while keeping this XML for human-readable reporting.

An example of this XML file:
{{{
<ldtp>
<group name="group1">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenAisleRiotSolitaire">
<COMMENT>elapsed_time: 6.22046804428</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:15:17 AM on 04-Dec-2008" elapsed="0:0:11"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group2">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenBlackjack">
<COMMENT>elapsed_time: 5.48234200478</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:15:28 AM on 04-Dec-2008" elapsed="0:0:11"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group3">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenChess">
<COMMENT>elapsed_time: 5.38424086571</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:15:39 AM on 04-Dec-2008" elapsed="0:0:10"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group4">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenFiveormore">
<COMMENT>elapsed_time: 5.5373609066</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:15:50 AM on 04-Dec-2008" elapsed="0:0:11"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group5">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenFour-in-a-row">
<COMMENT>elapsed_time: 5.54238200188</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:16:01 AM on 04-Dec-2008" elapsed="0:0:11"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group6">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenGnometris">
<COMMENT>elapsed_time: 5.35859894753</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:16:12 AM on 04-Dec-2008" elapsed="0:0:10"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group7">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenKlotski">
<COMMENT>elapsed_time: 5.61202001572</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:16:23 AM on 04-Dec-2008" elapsed="0:0:11"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group8">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenMahjongg">
<COMMENT>elapsed_time: 5.51350283623</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:16:34 AM on 04-Dec-2008" elapsed="0:0:10"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group9">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenMines">
<COMMENT>elapsed_time: 5.41266489029</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:16:45 AM on 04-Dec-2008" elapsed="0:0:11"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group10">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenNibbles">
<COMMENT>elapsed_time: 5.88356804848</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:16:56 AM on 04-Dec-2008" elapsed="0:0:11"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group11">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenSamegNOME">
<COMMENT>elapsed_time: 5.41124200821</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:17:07 AM on 04-Dec-2008" elapsed="0:0:10"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group12">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenSudoku">
<COMMENT>elapsed_time: 5.65414404869</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:17:18 AM on 04-Dec-2008" elapsed="0:0:11"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group13">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenTali">
<COMMENT>elapsed_time: 5.56410288811</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:17:29 AM on 04-Dec-2008" elapsed="0:0:11"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
<group name="group14">
<script name="/usr/share/ubuntu-desktop-tests/gnome-panel/open_app.py">
<test name="OpenTetravex">
<COMMENT>elapsed_time: 5.51043796539</COMMENT>
<pass>1</pass>
</test>
</script>
<timeinfo start="11:17:40 AM on 04-Dec-2008" elapsed="0:0:11"></timeinfo>
<groupsstatus total="1" pass="1" fail="0"></groupsstatus>
</group>
</ldtp>
}}}

For Intrepid milestones these results have been parsed and transformed into a HTML file. An example can be found here http://people.ubuntu.com/~ara/ldtp/test_results/intrepid/alpha-6/openAll/Accessories.html

=== Test definition ===
The current test definition in Checkbox should be enough to define the tests. When the desktop-testing plugin is written, that would be the type of test.

It is necessary, though, to implement a way to differentiate a test failure from a script error. This is going to be tracked in Launchpad Bug #306113 (https://bugs.launchpad.net/checkbox/+bug/306113)

== Implementation ==

=== Code Changes ===

==== New desktop-testing plugin ====
==== New error test status ====

== Unresolved issues ==

Sometimes the LDTP server breaks its connection. We should be able in further implementations to start a new LDTP server automatically to run the remaining tests of the suite.

== BoF agenda and discussion ==


----
CategorySpec