• Launchpad Entry: qa-upstream-kernel-bugs

• Created:

• Contributors:

• Packages affected:

Summary

Improve the collaboration with kernel upstream around bug reports. We will make our bug reports more relevant to upstream by facilitating testing on stock upstream kernels and make them more useful by adding kerneloops information.

Rationale

We are in a position to connect end users with the upstream kernel developers. We should focus on facilitating the communication between the two parties. Currently, when a bug is reported against the Ubuntu kernel it is not necessarily tested and reported against the upstream kernel. Testing the upstream kernel currently requires knowledge on how to build and install the upstream kernel from source. This is not something every Ubuntu kernel bug reporter is comfortable doing. As a result, bugs do not get tested and escalated upstream. With the majority of Ubuntu kernel bug fixes originating from the upstream kernel, we've become a bottleneck to getting bugs resolved. A bug in the upstream kernel is a bug in the Ubuntu kernel.

Querying the upstream kernel bugzilla, there are currently 1306 bugs in an open state ("New", "Assigned", "Reopened", "Verified", "Deferred", "Needinfo"). Compare that to the 4150 bugs in an open state ("New", "Incomplete", "Confirmed", "Triaged", "In Progress", "Fix Committed") against just the "linux" kernel package in Launchpad and it's easy to see why we need to improve in this area.

Use Cases

• Ubuntu user John experiences a kernel oops every time he boots his system. John should also test the upstream kernel to verify if he is able to trigger this oops when booting the upstream kernel. If the issue exists with the upstream kernel, John should notify the upstream kernel developers of the issue.
• Ubuntu user Sue has a new usb wifi dongle that is not working in Ubuntu. Sue tests the upstream kernel and discovers the device works perfectly. Sue should work with the Ubuntu kernel devs to narrow down the fix that should be pulled into the Ubuntu kernel.

Assumptions

• The kernel team will start providing an upstream vanilla kernel package for users to install and test with - DONE (jaunty-kernel-team-process)

Implementation

1. Work with the Ubuntu kernel team to provide an upstream vanilla kernel package that our users can easily install and test.
   • Build a package for the latest upstream kernel -rc candidate or final release
   • Build a package for the most recent upstream stable kernel
2. Work on re-educating our bug reporters to test the upstream kernel and report their bugs upstream:
   1. Bug is triggered using the Ubuntu kernel

   2. Next step is to test the stable kernel package which the Ubuntu kernel was based on. Use cat /proc/version_signature to determine which stable kernel package the Ubuntu kernel was based on.

      • If the bug does not exist with the stable kernel package this indicates the issue is with an Ubuntu specific patch(es)
      • If the bug exists with the stable kernel package, proceed to the next test
   3. Test the latest upstream kernel -rc candidate or final release package (which ever is the most recent)
      • If the bug does not exist in the upstream kernel, narrow down which patch to backport to the Ubuntu kernel
      • If the bug exists in the upstream kernel, report the bug upstream (create docs to instruct user the proper was to report the bug upstream - may vary between reporting the bug to a mailing list or filing a bug report at kernel.bugzilla.org)
3. Enable kerneloops by default and enable apport to detect and report bugs from oopses on the next boot

   • Add extra version info to identify the kernel as an Ubuntu kernel so that oops reports are more useful to upstream - DONE

   • Ensure that the hardware certification infrastructure takes proper advantage of the oops reports - this includes booting the system with a working kernel / in a working mode to let apport generate the bug report
4. Work with upstream kernel bugzilla maintainers to update to bugzilla 3.0 and install the Launchpad plugin

   • https://lists.ubuntu.com/archives/kernel-team/2008-December/003837.html

BoF agenda and discussion

How can bug reports be forwarded better to the upstream developers and kernel bug tracker?

• automation of reporting a kernel bug report upstream - similar to apport
  • or at least a tool for the QA team to forward them more easily
  • focus on not saturating upstream with poor reports
• ubuntu qa team can not be a middle man for bug reporters
  • education of Ubuntu users
  • test upstream vanilla kernel and file upstream bug if necessary
  • Ubuntu QA Team to work on direct communication of bugs with upstream mailing lists
• important to reproduce/confirm bugs if possible

flag bugs that are on "supported" hardware - a system that ships with Ubuntu

• these bugs should be identified in Launchpad and higher priority

check if kernel has debugging enabled by default:

~/ubuntu-jaunty/debian/config$ grep -rn "CONFIG_DEBUG_KERNEL" *
amd64/config:539:CONFIG_DEBUG_KERNEL=y
armel/config:182:CONFIG_DEBUG_KERNEL=y
i386/config:569:CONFIG_DEBUG_KERNEL=y

---

CategorySpec