TestingWeatherReport

Revision 6 as of 2009-05-28 14:59:49

Clear message

Summary

A web application similar to the Ubuntu release weather report oriented towards Ubuntu pre-release testers that will display various bits of relevant information on the state of the development branch (archive integrity, ongoing library transitions, high-impact bugs, critical or high importance bugs in core components, daily image health and build status, new packages, updates in components related to release goals, calls for testing by developers, etc.)

Release Note

Not required.

Rationale

Use Cases

Jono is an inexperienced Ubuntu tester, and wants to know whether it's a good idea to jump into the development branch today. Among the things he's likely to care about are whether today's daily CD is healthy, whether there are any grave bugs affecting his configuration that will make it meaningless to start testing, and whether there's any activity regarding the design goals he's interested in.

Ara has been seeing calls for urgent testing of certain components during the development cycle in various places such as Planet Ubuntu, ubuntu-devel, the Ubuntu Forums, personal blogs, and would like a one-stop shop where she can follow them.

Efrain is a would-be tester who wants to get started in contributing to free software through testing, and seeing that a particular prominent free software distribution has a pretty "dashboard" that displays its development branch status, feels slightly more eager to get started with that project.

Assumptions

The metrics listed in "Design" can all be pulled from their respective sources.

Design

Data

Possible non-exhaustive list of data to include:

  • Held back packages
  • Critical bugs in core packages (kernel, gcc, glibc, etc.) that may have high impact on a broad range of testers
  • Ongoing library transitions
  • Archive integrity
  • Daily image build status
  • Daily image health check (oversized or not)
  • ISO package diff (how up-to-date are the current ISO images with the archive?)
  • Upgrade situation ( possibly with automatic upgrade test results)
  • Current milestone, release notes / technical overview, next milestone name, how many days are left
  • Freeze status (freezes we are on and upcoming ones)
  • New packages in the archive
  • Bugs targeted for the next milestone
  • Calls for testing of specific packages (possibly not yet pushed to the development branch) by developers
  • Significant updates in components related to release goals
  • Archive mirror status
  • Build daemon status

Design Goals

  • Come up with a list of most relevant metrics and other data that will help testers
  • Display the data in a manner in which the most relevant elements can be evaluated at a single glance, on a single page, and the slightly less relevant can still be easy to reach
  • Present it all on an attractive web page that will hopefully present a small bit of extra incentive for people to do testing for Ubuntu.

Implementation

UI Changes

Code Changes

Migration

Test/Demo Plan

Unresolved issues

BoF agenda and discussion

Goal

have a webpage to track metrics about the current state of testing pulling in existing testing information

  • centralize data
  • present it in an attractive way
    • helps to attract more testers
  • provide a service to the testing community to show them if it's safe to begin/continue testing

Currently the testing information is very scattered and confusing for new users and testers

  • difficult to know what exists
  • or where to find anything
  • spread across multiple sites
  • difficult for users and testers to know e.g. when packages are being held back and shouldn't be upgraded

Design Rules

  • find most relevant data
    • metrics about builds, etc.
    • build status
    • notices of significant ongoing work
      • e.g. major subsystem updates
    • automated upgrade test results
    • name of next milestone and number of days before reaching it
    • freeze status
      • upcoming freezes and what they mean
    • new packages in archive (RSS feed)
    • bugs targeted for next milestone
    • calls by developers for testing specific packages
    • could use OpenID to pull info from Launchpad
      • subscribed bugs, blueprints, etc.
  • find where to get data
    • launchpad
    • screen-scraping
    • etc.
  • display data in a way that the most relevant data will be together in one page
    • but provide links back to where the data came from whenever possible

Intended Audience

  • ubuntu+1 testers who are currently testing in a "disorganized" way
    • ad-hoc testing rather than scripted tests
  • will also support wider QA audience

Ubuntu Release Weather Report:

http://qa.ubuntu.com/reports/ogasawara/weatherreport.html

Issues

  • will requre a great deal of development
    • may be doable with django
  • 'weather report' might not be the best name for this as it's somewhat different than the testing weather report
    • perhaps 'dashboard' instead

Design

  • should probably be a webpage
    • interactive django page?
  • first goal is to figure out what gets us the best bang for the buck and get some data going
  • needs to be certain to be attractive
  • makes sense to make it in Python

First Steps

  • get a place to host this and get it started
    • a place on the QA server
    • get something in place and begin iterating
  • initial data to display
    • CD size
    • CD build status
    • important development messages (rollbacks, etc.)

CategorySpec