Filing Kernel Bug reports

Running the following command from a terminal window will automatically file a bug against the Ubuntu kernel package in Launchpad. This will also automatically gather and attach important system information to your bug report.

$ ubuntu-bug linux

Minimal information

Running the above command is the preferred method for reporting Ubuntu kernel bugs. If you manually reported a kernel bug you still have the option of running the following which will gather and attach the requested system information to your bug report.

$ apport-collect -p linux-image-`uname -r` <bug#>

As a last resort, you may manually file your kernel bug, but at a minimum it should contain the output of the following commands:

$ uname -a > uname-a.log
$ cat /proc/version_signature > version.log
$ dmesg > dmesg.log
$ sudo lspci -vvnn > lspci-vvnn.log

These four files should be attached separately to the bug report (not pasted into comments or tar and zipped, as it makes things harder to read, and formatting is completely broken). Please note the comment about dmesg output below.

It is also important to identify your hardware in the title of the bug to allow easy searching.

Additional information

Dmesg output may be annotated to show before and after the problem occurred (e.g. if a device is attached and does not work, annotate where in dmesg the device was attached so that messages related to the attachment can be isolated). Also, dmesg output should be done as early as possible after bootup to avoid extraneous output.

The submitter should provide as much information as possible in regards to whether the bug exists in previous versions of Ubuntu. Where possible, test the latest development version as well. Whether or not the bug affects other Linux distributions, or stock mainline kernels is also of importance.

If this is a sound related bug, also run the following to gather and attach important sound debugging information to your bug:

$ apport-collect -p alsa-base <bug#>

Reporting Bugs Upstream

Depending on the phase of the Ubuntu release cycle, the Ubuntu kernel team will rebase the Ubuntu kernel with the upstream mainline kernel. Unfortunately, it is sometimes the case that bugs can still exist in the upstream mainline kernel. The upstream kernel has its own bugtracking system at http://bugzilla.kernel.org . If you know your bug exists upstream, it would be helpful if you would also report your bug with the upstream kernel bug tracker. It is often the case that once a bug is escalated upstream there is a quick resolution through the help and support of the mainline kernel community. Bug reports in Launchpad can also be set up to monitor bugs reported in other bug tracking systems. The following steps should help you report your bug upstream and monitor it in Launchpad:

1. Go to http://bugzilla.kernel.org

2. Verify your bug does not already exist in the upstream bugtracking system.
   1. If it does exist, please add any useful information that may be lacking in the current report.
      • Link your Launchpad report to the upstream kernel bug report (see step 5).
   2. If your bug does not exist, proceed to the next step.

3. Enter your bug report at http://bugzilla.kernel.org/enter_bug.cgi . Follow the instructions on that page.

4. When creating your bug report be sure to _attach_ any relevant information. This will usually include:
   1. the version of the kernel you are running (uname -r)
   2. dmesg output.
   3. lspci -vvnn output.
   4. If the bug is about interrupts, then /proc/interrupts is also helpful.
   5. If a BIOS issue is suspected, dmidecode output is also helpful.
   6. If the bug is regarding memory oddities, /proc/mtrr info is also helpful.
   7. Most importantly, if this is a regression, the best information you can provide the kernel developers is the information from doing a git bisect. This will hopefully narrow down the exact patch that is causing the regression. The following link will help you through this process.

      • http://www.kernel.org/doc/local/git-quick.html#bisect

5. After you've reported your bug to the kernel bugzilla, remember to set up your Launchpad bug report to monitor the upstream bugzilla report. Refer to Watching Another Project for a more detailed step by step process.

   1. On your Launchpad bug report page, look in the "Actions" area on the left hand side of the page.
   2. Click the "Also affects project" link.
   3. Enter the URL of your bugzilla report.
   4. Click the "Add to Bug Report" button.
   5. Your Launchpad report is now monitoring your upstream bugzilla report.

Problems in capturing information

Bootloader

If the bug occurs during bootup, you can disable the splash screen in one of the following ways:

• Permanent: Edit /boot/grub/menu.lst and remove splash and quiet kernel parameters to bootloader from the entry for the buggy kernel

• Temporary: Press 'Escape' key at the 3 second pause by Grub bootloader. Then press 'e' (edit) on the buggy kernel entry, followed by 'e' again on the kernel line. Then remove the splash and quiet keywords and press 'b' to boot.

Capturing OOPs

If the bug report involves a crash, it is hoped that a kernel backtrace (aka OOPS, kernel panic) is available. If the machine does not completely lockup from the crash, the backtrace should be available in the dmesg output. If the crash completely locks the system:

• Try rebooting to a known working kernel and see if any backtrace was logged to /var/log/kern.log.0. Please attach this file if anything was captured.

• When no clues found in /var/log/kern.log.0, supply a digital photo of the screen to capture the crash.

• When all fails, try to see if KernelTeam/Netconsole can help out.

In X window mode

Sometimes crashes occur in X, and so terminal access is not available (to capture the kernel backtrace). When this occurs, the user should try to recreate the crash at the console (Ctrl+Alt+F1). If this is not possible, then annotate the bug as such.

DIY Debugging Hints

For a complete list of debugging procedures refer to https://wiki.ubuntu.com/DebuggingProcedures

Bug Triage

Triaging kernel bugs is a day-to-day effort, and can be very time consuming. Luckily, we have a lot of community members willing and able to help with this effort. In order to make sure everyone working on kernel bugs follows the same policy, this document will describe how to handle the kernel bug workflow.

Simple rules

The HOWTO below really explains most of the details of working with kernel bugs, but this list of rules should help clarify some common mistakes:

Triaged bugs should have Importance set

Bugs marked Triaged should have their Importance set to something other than Undecided. If a bug is Triaged, then there should be enough information to know how important it is.

Incomplete means subscribe yourself to the bug

If you are working on triaging a bug report, and it requires more information, you should mark the bug Incomplete and subscribe yourself to the bug. You will then be notified of any additional information that may get added to the bug. Note, the Incomplete state can also mean that the person triaging the bug is still investigating the data and background in order to assess if the bug is Confirmed, so does not necessarily mean more information is needed from the bug submitter.

In Progress means you are working on the bug

If a bug is marked In Progress, then it should be assigned to yourself. Marking a bug In Progress means you are actively working on it. If you need to keep the bug marked In Progress for an extended period (greater than a week or two), then you should update it periodically with progress, or just put it back to Confirmed and assign it back to the appropriate team.

Cannot have Importance without Status

A bug should not be given an importance until its status has been set, usually to Triaged.

Bug triage HOWTO

Handling incoming bugs is a fairly common and well defined task. The outline below will help you gather the correct information, and process bugs so that they will be visible to the kernel team, and to others helping with these bug reports.

New and Unassigned

This is how bugs come in to the system. No new bug should come in with a pre-existing Status or Assignee.

After reviewing the bug, you should assess whether there is enough information to actually fix or reproduce the bug. In most cases, there is always more needed information. You should at this point, set the Status to Incomplete, and assign the bug to yourself. Note that the Incomplete phase is usually one of the longest, requiring requests from yourself, and replies from the bug submitter.

Note, you should never have any bugs assigned to yourself in the New, Confirmed or Triaged state.

Most common information needed:

• dmesg output

• lspci -vv and lspci -vvn output

Please ensure that the bug submitter attaches these outputs as opposed to pasting them into comments.

If the bug report is a crash, then have the user describe what they were doing during this crash, and if they are able to reproduce it. Hopefully the crash doesn't completely lock the machine, so that dmesg can be obtained. If this is not possible, then ask the user if they can get the crash output with a digital photo of the screen.

Once all the required info is obtained, it's time to decide where the bugs go.

• If it turns out to be a bug in something other than the kernel, it should be re-assigned to that source package.

• If the bug is not a legitimate bug (e.g. it was user caused), then the bug should be marked Rejected.

• If the bug is a legitimate bug in the kernel, then it should be marked as Confirmed or Triaged.

Setting the severity takes experience. Generally speaking, if the bug is a crash, it should be marked as High. Most bugs will start out as Medium though.

Confirmed and Triaged

Bugs marked as Confirmed indicate someone other than the original bug reporter can confirm the issue being reported. Bugs set to Triaged indicate there is enough debug information supplied to the bug report for a developer to start working on the issue. You must be a member of the Ubuntu Bug Control team in order to set bugs to Triaged.

It used to be part of the kernel team bug policy to also assign Confirmed and Triaged bugs directly to the ubuntu-kernel-team. After discussions during UDS Jaunty, this is no longer the case. Please do not arbitrarily assign the ubuntu-kernel-team to bugs. It's intended that assignment of bugs to the ubuntu-kernel-team will be used in a much more restricted sense going forward - for example release critical bugs.

In Progress

Bugs marked as In Progress are expected to be worked on by the assignee in a reasonable amount of time. If the person handling this bug cannot fix it in a reasonable amount of time, it should be reassigned back to the kernel team, and updated with any information that may help fix it.

Fix Committed

Bugs marked Fix Committed are considered fixed by a patch committed to the git repository. Fix is not yet applied to the built kernel. It is expected that the next kernel upload will contain this fix. There is no determinate time for when this will happen though.

Fix Released

The hopeful final end of any legitimate bug is to be Fix Released, meaning that the kernel in the archive (in the updates pocket) contains the fix needed to close this bug report.

Tags

Tagging a bug is an easy way for us to group bugs across packages, find certain types of bugs or break a group of bugs into smaller parts. Tags which are commonly used for kernel bugs are:

Include: Nothing found for "== Kernel Specific ==$"!

Please refer to Bug Tags for more information.

Caveats

Sometimes kernel bugs are opened to track security vulnerabilities. These bugs usually contain the word "CVE" either in the title or bug description and will most likely have the ubuntu-security team subscribed to the bug. We should try to avoid spamming these bugs with comments to test the latest kernel to verify if the issue still exists. Whether manually posting to a bug or using python-launchpad-bugs to script comments, please take extra care to not cause more unnecessary traffic for the security team to deal with. Thanks.

CategoryKernel CategoryBugSquad