Workflow

Security updates for the Linux kernel are managed in a collaboration between the Security team and the Kernel SRU team. The Kernel SRU cycle runs on a 4/2 cadence, where security fixes are included in both the kernel SRU team's 4-week standard stable update cycle and also in update released at the 2-week midpoint of the following 4-week cycle. In the latter it is expected to contain only CVE fixes rated critical or high.

The security team's responsibilities in this are to:

References

This is the Kernel Stable Release Update Workflow that all the involved teams collaborate on. Launchpad is overloaded to build a release "state machine", managed by a bot.

There is a internal JIRA dashboard for tracking security team's work with kernels:

Code

There are several git trees with scripts in them. UCT carries the main CVE tracking and several of the triage and processing scripts. UQT carries validation scripts. kteam contains scripts written by the kernel team, some of which are used for interfacing with UCT and LP. kernel-versions contains relevant information for each kernel per cycle.

You will also need the python3-launchpadlib package installed.

It is also useful to have both the upstream linux kernel and stable kernel trees as well as the primary ubuntu kernel trees checked out somewhere, too.

An example of how to set them up: 

  $ mkdir -p ~/git/kernel-trees/ && cd ~/git/kernel-trees/
  $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
  $ git clone --reference linux https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
  $ export RELEASES=bionic focal impish jammy
  $ for release in $RELEASES ; do
      git clone --reference linux --reference linux-stable https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/$release ubuntu-$release
    done

Additional, optional external git trees useful for triage:

Security Team Duties

Triage

Per CVE

Handling CVE triage is basically the same here as with standard CVE triage, mainly all kernel CVEs are comming from the Kernel.org CNA and end up being added to UCT during the daily CVE triage routine. 

Patches_linux:
 break-fix: 1234567812345678123456781234567812345678 abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd
 break-fix: 1234567812345678123456781234567812345678 efefefefefefefefefefefefefefefefefefefef

Patches_linux:
 break-fix: - efefefefefefefefefefefefefefefefefefefef

Patches_linux:
 break-fix: 1234567812345678123456781234567812345678 -

Merging status updates from the kernel team's cve tracker

The kernel team uses automated tools (frequently known as autotriager) that take the identified git commits (the break-fix entries) and updates the status of CVEs in their branch of the ubuntu cve tracker . To handle this, one needs to add their tree as a remote to your UCT git tree:

In this fork, the kernel team maintain a few different branchs with different purposes, such as:

There should be a cron job set up to notice and send email to the team if there are unmerged commits to the kernel team's tree (Subject: Missing kernel CVE merge commits). This cron job is currently turned off but you can run this manually if your tree is set up like the above and the git remotes have been fetched by doing:

If there are outstanding commits to be merged, review them to ensure that the changes are sensible, and then merge them to the security team's tree with something like:

You can use the bash shell function uct_kernel_merge_commit from $UCT/scripts/dot.uct-functions.sh to do the merge; it accepts an optional argument as a different branch to merge from. Sometimes there will be merge conflicts between the kernel team's branch and the primary branch, so it can be helpful to resolve this on a separate local branch, and then merge *that* branch into the primary UCT repo.

If, when you try to push a merge to the shared UCT tree, someone beat you and your tree is out of date, you can do git pull --rebase=merges to rebase on top of the missing commits while preserving the merged branch.

Retriage

Kernel CVEs need to be occasionally re-triaged to identify CVEs either without a breaks-fix line or with an unpopulated fix entry.

To re-triage:

Security team kernel signoffs

Once the kernel team has prepared a kernel and pushed it to the appropriate proposed pocket, the kernel team's launchpad bot will mark the tracking bug for that kernel as needing the security team's signoff. To get a list of kernel tracking bugs needing signoff, do:

This should be in a cron job.

To perform signoffs: 

lookup_upstream_commit () 
{ 
    commit="$1";
    if git merge-base --is-ancestor "${commit}" HEAD; then
        git describe --contains "${commit}";
        return;
    fi;
    local_commit=$(git log --grep "${commit}" -1 --pretty=format:"%H");
    if [ -n "${local_commit}" ]; then
        git describe --contains "${local_commit}";
    else
        echo "Unable to find ${commit} in $(git branch)";
    fi
}

USN publication

Once the kernels have been verified and tested, they will be published to the appropriate security pocket, and thus the security team needs to publish USNs for them.

Each kernel is actually composed from two or three source packages: the main kernel package (e.g. linux) which is what is tracked in UCT, the corresponding meta package (e.g. linux-meta) which generates the meta packages for each kernel that depend on the updated binary packages, to ensure that an update will pull in the binary kernel packages (and so that users can have multiple binary kernels installed), and for kernels that are signed, the corresponding signed kernel source (e.g. linux-signed).

The script $UCT/scripts/kernel-abi-check will report if new kernels have been published to security pockets, and also does a consistency check to ensure that the primary source is in sync with its meta and signed source package ABIs. This script is run out of cron and sends email to the team if it has anything to report. The --check-esm argument can be passed to look for new kernels published in the ~ubuntu-esm/esm-infra-security ppa (this should be added to the cron job run on the shared server). Additionally the argument --check-cycle returns which cycle that the kernel is placed, helping organize the USNs in set of CVEs per cycle.

To generate USNs:

Tracking

Adding tracking for a new derived kernel

For each new added kernel to support, we need to add tracking to the CVE tracker. You can check that the kernel isn’t already in UCT with grep <kernel-name> $UCT/boilerplates/linux 

    '<kernel-name>': {
        '<release>': {
            '<last-vers>': '<curr-vers>',
        },
    },

    "<kernel-name>": {
    [...]

Dropping tracking support

When a kernel is out of support, we need to mark UCT as such. A kernel going EOL usually happen because it was superseded (e.g. linux-hwe-6.5 is superseded by linux-hwe-6.8) The confirmation for when a kernel is out support can be acquired from the kernel-versions repository, info/kernel-series.yaml file specifically. You will find information such as: 

    linux-hwe-6.5:
...
      supported: false
...

So, when it is time to drop support for a kernel, we add ignored (superseded by <>)' information in boilerplates/linux file in place of needs-triage, like: 

  Patches_linux-hwe-6.5:
  upstream_linux-hwe-6.5: ignored (superseded by linux-hwe-6.8)
  ...
  focal_linux-hwe-6.5: DNE
  jammy_linux-hwe-6.5: ignored (superseded by linux-hwe-6.8)
  noble_linux-hwe-6.5: DNE
  oracular_linux-hwe-6.5: DNE
  ...

In general we try to give a reason for being out of support that is different from "end of life" to try to clarify that it is not the release going EOL but the kernel package instead. The reasons we have used so far (but are not limited to) are:

Besides updating the boilerplates/linux for the incoming CVEs, it is also good to update the active CVEs with the corresponding information, adding the previous state (e.g. ignored (end of kernel support, was needs-triage)). This could be incorporated to scripts/check-syntax at some point. But before that, there may be cases where we have a pending with an already released version, so instead of marking as ignored it will be released.

Deprecated bits

Reviewing the state of the CVEs between UCT, the kernel team's UCT tree, and the USN database should happen at least daily. In practice, the USN comparison usually happen much more rarely due to its current fragility.

(NOTE: we no longer require a bug report for each CVE being addressed.)

SecurityTeam/UpdatePublication/Kernel (last edited 2025-02-24 15:35:08 by rodrigo-zaiden)