AutomatedProblemReportsTagging

Revision 1 as of 2006-07-29 06:17:26

Clear message

Summary

This spec describes a method for categorizing and sorting AutomatedProblemReports to help developers identify similar or identical bugs.

Rationale

Collecting AutomatedProblemReports can result in DrinkingFromTheFirehose due to the massive influx of problem reports. We must provide a way for developers to wade through the inundation without getting lost in sudden influxes of thousands of copies of the same problem.

Use cases

There are many use cases:

  • Rhythmbox spontaneously crashes. On six thousand machines.
  • Totem and Nautilus both crash, due to a bug in the gstreamer XviD plug-in being triggered during playback in Totem or thumbnailing in Nautilus.

In each of these cases the AutomatedProblemReports daemon would report a problem back to Ubuntu. On the server, these reports would be analyzed, categorized, and tagged using the resulting information.

Scope

The scope of this spec includes all problems reported by AutomatedProblemReports.

Design

We will need a crash handler and reporter, which is from AutomatedProblemReports.

The server handling the AutomatedProblemReports should tag problems exhibiting known characteristics to fall into certain categories.

There are several heuristics we can use, including:

  • Crash occurs at the same point
    • The same function

      • i.e. we found a stack smash because it called __stack_chk_fail(), the calling function was some_vuln_function()

    • The same module (library or program)

      • i.e. some_vuln_function() is from some_vuln_lib.so

  • list all crashes with SOME_NUMBER of common back trace from the crash point

    • i.e. we found a double-free(), determined some_stupid_function() called free(), and the last 3 up to there were good_function(), fine_function(), some_stupid_function().

  • Crashes fault on the same problem

    • i.e. SIGILL crash.

    • SIGSEGV detected attempting to execute unmapped memory

AutomatedProblemReports would be tagged with information such as this. Developers would later be able to use these tags as search criteria to find similar bugs. For example, pitti may select a crash and "search for similar crashes." He may then enter (through check boxes) various similarities to locate. Let's take the example of a SIGSEGV attempting to execute the stack; the below options may be available to him:

  • Similar fault

    • SIGSEGV

      • Attempt to execute
        • Attempt to execute non-executable area
  • Same module

    • Previous executing function was very_broken_function(), so we assume we are in module very_broken_lib.so

      • The version was 0.1.19 (i.e. very_broken_lib.so.0.1.19)

  • Same program

    • Running /usr/bin/program_that_crashed

  • Similar backtrace

    • Fault occurred in very_broken_function()

    • Previous N calls (you enter a value for N)
  • Related (as manually tagged by the developers)
    • Yes
    • No
    • Unknown

The range of options is limitless; but some example searches include:

  • SIGSEGV in /usr/bin/program_that_crashed in module very_broken_lib.so (any version) apparently from very_broken_function()

    • Will not worry about matching any part of the tail of the backtrace, or that the attempt was to execute.

  • SIGSEGV on attempt to execute related to very_broken_lib.so.

  • Any fault in very_broken_lib.so.

This would prove to be a powerful tool for taking characteristics of an automatically detected problem and matching it with other problems.

The other problems listed may or may not be related. Developers would manually tag them as being the same bug; group them in the same group; and have quick and easy future reference to all the different reports.

Implementation

Code

We will need AutomaticProblemReports working first.

The server handling the AutomatedProblemReports must tag problems exhibiting known characteristics, allowing developers to sort and examine them.

Data preservation and migration

No issues exist.

Unresolved issues

BoF agenda and discussion

Comments

CategorySpec