Filing Kernel Bug reports

Running the following command from a terminal window will automatically file a bug against the Ubuntu kernel package in Launchpad:

ubuntu-bug -p linux

This will also automatically gather and attach important system information to your bug report.

Minimal information

Running the above command is the preferred method for reporting Ubuntu kernel bugs. Alternatively, you may manually file your bug at https://bugs.launchpad.net/ubuntu/+source/linux/+filebug . All bug reports at a minimum should contain the output of the following commands:

• uname -a > uname-a.log

• cat /proc/version_signature > version.log

• dmesg > dmesg.log

• sudo lspci -vvnn > lspci-vvnn.log

These four files should be attached to the bug report (not pasted into comments, as it makes things harder to read, and formatting is completely broken). Please note the comment about dmesg output below.

It is also important to identify your hardware in the title of the bug to allow easy searching.

Additional information

Dmesg output may be annotated to show before and after the problem occurred (e.g. if a device is attached and does not work, annotate where in dmesg the device was attached so that messages related to the attachment can be isolated). Also, dmesg output should be done as early as possible after bootup to avoid extraneous output.

The submitter should provide as much information as possible in regards to whether the bug exists in previous versions of Ubuntu. Where possible, test the latest development version as well. Whether or not the bug affects other Linux distributions, or stock kernels is also of importance.

Reporting Bugs Upstream

Depending on the phase of the Ubuntu release cycle, the Ubuntu kernel team will rebase the Ubuntu kernel with the upstream mainline kernel. Unfortunately, it is sometimes the case that bugs can still exist in the upstream mainline kernel. The upstream kernel has its own bugtracking system at http://bugzilla.kernel.org . If you know your bug exists upstream, it would be helpful if you would also report your bug with the upstream kernel bug tracker. It is often the case that once a bug is escalated upstream there is a quick resolution through the help and support of the mainline kernel community. Bug reports in Launchpad can also be set up to monitor bugs reported in other bug tracking systems. The following steps should help you report your bug upstream and monitor it in Launchpad:

1. Go to http://bugzilla.kernel.org

2. Verify your bug does not already exist in the upstream bugtracking system.
   1. If it does exist, please add any useful information that may be lacking in the current report.
      • Link your Launchpad report to the upstream kernel bug report (see step 5).
   2. If your bug does not exist, proceed to the next step.

3. Enter your bug report at http://bugzilla.kernel.org/enter_bug.cgi . Follow the instructions on that page.

4. When creating your bug report be sure to _attach_ any relevant information. This will usually include:
   1. the version of the kernel you are running (uname -r)
   2. dmesg output.
   3. lspci -vvnn output.
   4. If the bug is about interrupts, then /proc/interrupts is also helpful.
   5. If a BIOS issue is suspected, dmidecode output is also helpful.
   6. If the bug is regarding memory oddities, /proc/mtrr info is also helpful.
   7. Most importantly, if this is a regression, the best information you can provide the kernel developers is the information from doing a git bisect. This will hopefully narrow down the exact patch that is causing the regression. The following link will help you through this process.

      • http://www.kernel.org/doc/local/git-quick.html#bisect

5. After you've reported your bug to the kernel bugzilla, remember to set up your Launchpad bug report to monitor the upstream bugzilla report. Refer to Watching Another Project for a more detailed step by step process.

   1. On your Launchpad bug report page, look in the "Actions" area on the left hand side of the page.
   2. Click the "Also affects project" link.
   3. Enter the URL of your bugzilla report.
   4. Click the "Add to Bug Report" button.
   5. Your Launchpad report is now monitoring your upstream bugzilla report.

Problems in capturing information

Bootloader

If the bug occurs during bootup, you can disable the splash screen in one of the following ways:

• Permanent: Edit /boot/grub/menu.lst and remove splash and quiet kernel parameters to bootloader from the entry for the buggy kernel

• Temporary: Press 'Escape' key at the 3 second pause by Grub bootloader. Then press 'e' (edit) on the buggy kernel entry, followed by 'e' again on the kernel line. Then remove the splash and quiet keywords and press 'b' to boot.

Capturing OOPs

If the bug report involves a crash, it is hoped that a kernel backtrace (aka OOPS, kernel panic) is available. If the machine does not completely lockup from the crash, the backtrace should be available in the dmesg output. If the crash completely locks the system, try rebooting to a known working kernel and see if any backtrace was logged to /var/log/kern.log.0 . Please attach this file if anything was captured. If all else fails, supply a digital photo of the screen to capture the crash.

In X window mode

Sometimes crashes occur in X, and so terminal access is not available (to capture the kernel backtrace). When this occurs, the user should try to recreate the crash at the console (Ctrl+Alt+F1). If this is not possible, then annotate the bug as such.

DIY Debugging Hints

https://wiki.ubuntu.com/DebuggingKernelSuspend

https://wiki.ubuntu.com/DebuggingIRQProblems

https://wiki.ubuntu.com/DebuggingHardwareDetection

https://wiki.ubuntu.com/DebuggingACPI

https://help.ubuntu.com/community/DebuggingSystemCrash

https://wiki.ubuntu.com/DebuggingSoundProblems

Bug Triage

Triaging kernel bugs is a day-to-day effort, and can be very time consuming. Luckily, we have a lot of community members willing and able to help with this effort. In order to make sure everyone working on kernel bugs follows the same policy, this document will describe how to handle the kernel bug workflow.

Simple rules

The HOWTO below really explains most of the details of working with kernel bugs, but this list of rules should help clarify some common mistakes:

Confirmed bugs are assigned to teams

When a bug is marked Confirmed it should be assigned to a team. In most cases this is the ubuntu-kernel-team. In other cases, noted below, bugs can be assigned to other teams. This rule implies that you should not have a bug in Confirmed state assigned to yourself. Once you take a bug to work on, it should be marked In Progress.

It also should be noted that Confirmed state should only be set by a person triaging the bug report. The bug submitter or other person experiencing the bug should not arbitrarily set Confirmed state.

Other teams are to be ignored right now, as they are all manned by the ubuntu-kernel-team anyway.

Triaged bugs should have Importance set

Bugs marked Triaged should have their Importance set to something other than Undecided. If a bug is Triaged, then there should be enough information to know how important it is.

Incomplete means subscribe yourself to the bug

If you are working on triaging a bug report, and it requires more information, you should mark the bug Incomplete and subscribe yourself to the bug. You will then be notified of any additional information that may get added to the bug. Note, the Incomplete state can also mean that the person triaging the bug is still investigating the data and background in order to assess if the bug is Confirmed, so does not necessarily mean more information is needed from the bug submitter.

In Progress means you are working on the bug

If a bug is marked In Progress, then it should be assigned to yourself. Marking a bug In Progress means you are actively working on it. If you need to keep the bug marked In Progress for an extended period (greater than a week or two), then you should update it periodically with progress, or just put it back to Confirmed and assign it back to the appropriate team.

Cannot have Importance without Status

A bug should not be given an importance until its status has been set, usually to Triaged.

Bug triage HOWTO

Handling incoming bugs is a fairly common and well defined task. The outline below will help you gather the correct information, and process bugs so that they will be visible to the kernel team, and to others helping with these bug reports.

New and Unassigned

This is how bugs come in to the system. No new bug should come in with a pre-existing Status or Assignee.

After reviewing the bug, you should assess whether there is enough information to actually fix or reproduce the bug. In most cases, there is always more needed information. You should at this point, set the Status to Incomplete, and assign the bug to yourself. Note that the Incomplete phase is usually one of the longest, requiring requests from yourself, and replies from the bug submitter.

Note, you should never have any bugs assigned to yourself in the New, Confirmed or Triaged state. Bugs in New should be assigned to no one, and bugs that are Confirmed or Triaged should always be assigned to the kernel team.

Most common information needed:

• dmesg output

• lspci -vv and lspci -vvn output

Please ensure that the bug submitter attaches these outputs as opposed to pasting them into comments.

If the bug report is a crash, then have the user describe what they were doing during this crash, and if they are able to reproduce it. Hopefully the crash doesn't completely lock the machine, so that dmesg can be obtained. If this is not possible, then ask the user if they can get the crash output with a digital photo of the screen.

Once all the required info is obtained, it's time to decide where the bugs go.

• If it turns out to be a bug in something other than the kernel, it should be re-assigned to that source package.

• If the bug is not a legitimate bug (e.g. it was user caused), then the bug should be marked Rejected.

• If the bug is a legitimate bug in the kernel, then it should be marked as Confirmed or Triaged and assigned to the kernel team.

Setting the severity takes experience. Generally speaking, if the bug is a crash, it should be marked as High. Most bugs will start out as Medium though.

Confirmed

Confirmed bugs should be assigned to the kernel team. When someone (either kernel team or community) wishes to work on one of these bugs, they should assign it to themselves, and mark its status as In Progress.

In Progress

Bugs marked as In Progress are expected to be worked on by the assignee in a reasonable amount of time. If the person handling this bug cannot fix it in a reasonable amount of time, it should be reassigned back to the kernel team, and updated with any information that may help fix it.

Fix Committed

Bugs marked Fix Committed are considered fixed by a patch committed to the git repository. Fix is not yet applied to the built kernel. It is expected that the next kernel upload will contain this fix. There is no determinate time for when this will happen though.

Fix Released

The hopeful final end of any legitimate bug is to be Fix Released, meaning that the kernel in the archive contains the fix needed to close this bug report.

Tags

Tagging a bug is an easy way for us to group bugs across packages, find certain types of bugs or break a group of bugs into smaller parts. Tags which are commonly used for kernel bugs are:

Include: Nothing found for "== Kernel Specific ==$"!

Please refer to Bug Tags for more information.

Caveats

Sometimes kernel bugs are opened to track security vulnerabilities. These bugs usually contain the word "CVE" either in the title or bug description and will most likely have the ubuntu-security team subscribed to the bug. We should try to avoid spamming these bugs with comments to test the latest kernel to verify if the issue still exists. Whether manually posting to a bug or using python-launchpad-bugs to script comments, please take extra care to not cause more unnecessary traffic for the security team to deal with. Thanks.

CategoryKernel CategoryBugSquad