Kernel

Revision 64 as of 2021-06-09 22:03:41

Clear message

Workflow

Security updates for the Linux kernel are managed in a collaboration between the Security team and the Kernel SRU team. Security fixes that are identified as being medium or lower priority are included in the kernel SRU team's update cycle; issues that are high or critical priority will often require an exceptional "emergency" kernel outside of the normal cycle.

The security team's responsibilities in this are to:

Additionally, as the kernel team adds or drops support for kernels, the security team needs to keep the set of kernels being tracked and publishing USNs for in sync in the ubuntu cve tracker.

References

This is the Kernel Stable Release Update Workflow that all the involved teams collaborate on. Launchpad is overloaded to build a release "state machine", managed by a bot.

Code

There are several git trees with scripts in them. UCT carries the main CVE tracking and several of the triage and processing scripts. UQT carries validation scripts. kteam contains scripts written by the kernel team, some of which are used for interfacing with UCT and LP.

You will also need the python3-launchpadlib package installed.

It is also useful to have both the upstream linux kernel and stable kernel trees as well as the primary ubuntu kernel trees checked out somewhere, too.

An example of how to set them up: 

  $ mkdir -p ~/git/kernel-trees/ && cd ~/git/kernel-trees/
  $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
  $ git clone --reference linux https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
  $ export RELEASES=bionic focal groovy hirsute
  $ for release in $RELEASES ; do
      git clone --reference linux --reference linux-stable https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/$release ubuntu-release
    done

Additonal, optional external git trees useful for triage:

Security Team Duties

Triage

Per CVE

Handling CVE triage is basically the same here as with standard CVE triage, except that since kernel CVEs tend to take a long time to get from the oss-security mailing list into Mitre, it's best to review the mailing list for new CVEs.

  • Triage normally.
  • Attempt to identify the upstream SHAs that fix the CVE.
  • Attempt to identify the upstream SHA that introduced the vulnerability. If it isn't relatively easy, just skip it. Having this isn't required, but makes backporting easier if it is known, and improves the value of future data-mining about when/where CVEs appear in the kernel. If skipped, or predates the Linus git tree, just use "-" for the SHA.
  • In the "Patches_linux:" tag, add one "break-fix" line per SHA fix. Please use full SHAs for the commit references, not 8 character shorthands, as the kernel team's triage scripts want the full commit hash.
    • For example, if 12345678 introduced the vulnerability, with abcdabcd and efefefef needed to fix it, this would be: 

Patches_linux:
 break-fix: 1234567812345678123456781234567812345678 abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd
 break-fix: 1234567812345678123456781234567812345678 efefefefefefefefefefefefefefefefefefefef

  • If we have a "SAUCE" or otherwise not upstream patch to fix an issue, or other quirks are needed, the $UCT/active/10autotriage.linux file can be used to create synthetic references for break-fix: entries. Ask apw for details on how to use this.

Merging status updates from the kernel team's cve tracker

The kernel team uses automated tools that take the identified git commits (the break-fix entries) and updates the status of CVEs in their branch of the ubuntu cve tracker. To handle this, one needs to add their tree as a remote to your UCT git tree:

  • git add remote kernel-team https://git.launchpad.net/~canonical-kernel-team/ubuntu-cve-tracker

  • git fetch --all -v -p -t

There is a cron job set up to notice and send email to the team if there are unmerged commits to the kernel team's tree (Subject: Missing kernel CVE merge commits). You can run this manually if your tree is set up like the above and the git remotes have been fetched by doing:

  • $UCT/scripts/report-missing-kmerge

If there are outstanding commits to be merged, review them to ensure that the changes are sensible, and then merge them to the security team's tree with something like:

  • git merge --no-ff --signoff -m 'merge cve updates from kernel team' kernel-team/master

(If, when you try to push a merge to the shared UCT tree, someone beat you and your tree is out of date, you can do git pull --rebase=preserve to rebase on top of the missing commits while preserving the merged branch.)

Security team kernel signoffs

Once the kernel team has prepared a kernel and pushed it to the appropriate proposed pocket, the kernel team's launchpad bot will mark the tracking bug for that kernel as needing the security team's signoff. To get a list of kernel tracking bugs needing signoff, do:

  • $UQT/security-tools/kernel-sru-check

This should be in a cron job.

To perform signoffs:

  • Update UCT tree: cd $UCT && git pull --ff-only

  • validate CVEs for USN publication: $UCT/scripts/prepare-kernel-usn.py -p Proposed -n RELEASE SRC VERSION

    • For esm kernels, add --include-eol --ppa canonical-kernel-esm/proposed

      • e.g. ./scripts/prepare-kernel-usn.py -n --include-eol --ppa canonical-kernel-esm/proposed  precise linux-lts-trusty 3.13.0-124.173~precise1

    • If there are missing CVE descriptions, they need to be written and added to the tracker
    • verify that the relevant commits have been made in the specific kernel's git tree
    • If an editor pops up, that means all CVEs have descriptions. Simply exit the editor.

  • If the specific kernel update contains security related fixes (or regression fixes introduced by a prior kernel published to security), mark the Security-signoff workflow item "Fix Released", otherwise mark it as "Invalid" to indicate that it should only go to the updates pocket.

USN publication

Once the kernels have been verified and tested, they will be published to the appropriate security pocket, nd thus the security team needs to publish USNs for them.

Each kernel is actually composed from two or three source packages: the main kernel package (e.g. linux) which is what is tracked in UCT, the corresponding meta package (e.g. linux-meta) which generates the meta packages for each kernel that depend on the updated binary packages, to ensure that an update will pull in the binary kernel packages (and so that users can have multiple binary kernels installed), and for kernels that are signed, the corresponding signed kernel source (e.g. linux-signed).

The script $UCT/scripts/kernel-abi-check will report if new kernels have been published to security pockets, and also does a sanity check to ensure that the primary source is in sync with its meta and signed source package ABIs. This script is run out of cron and sends email to the team if it has anything to report.

To generate USNs:

  • Update pickle: cd $UCT && git fetch --all -v -p -t && git pull --ff-only && ./scripts/fetch-db database.pickle.bz2

  • You can run the $UCT/scripts/kernel-abi-script locally to report on kernels needing USNs (same as what's run out of cron on people.c.c)

  • check the USN still generates correctly using $UCT/scripts/prepare-kernel-usn.py -n REL SRC VERSION

    • Check multiple derived kernels for the same Release $UCT/scripts/prepare-kernel-usn.py -n REL SRC VERSION SRC2 VERSION2 SRC3 VERSION3

    • Edge kernels (i.e. 'linux-hwe-edge' and linux-azure-edge') that get published in the security pocket need to be ignored. Edit the kernel_glitches dict in $UCT/scripts/cve_lib.py to match the recently published version and move on to the next kernel USN, if any.

    • Also, when the edge meta kernels aren't different from the stable version (this happens before the jump to a new kernel version in edge), kernel ABI warnings will occur. Edit the kernel_mabi_glitches dict in $UCT/scripts/cve_lib.py to silence these warnings.

      • Other kernel ABI warnings mean that something has gone wrong in the kernel publishing process; if the error is not transient (occurs more thn once), then raise issue with the kernel team and/or an archive admin. (The ABI warnings are generated when the kernel meta source package ABI does not match the ABI of the corresponding kernel it should point to.)

    • For embargoed kernels, the triage bot will likely be out of sync, and unless the embargoed kernel is on top of a kernel at the end of its SRU cycle, the CVEs the tracker thinks will be fixed will not. Thus these CVEs need to be ignored. You can use the --embargoed option to $UCT/scripts/prepare-kernel-usn.py to do this.

  • publish USN for real using new USN number: $UCT/scripts/prepare-kernel-usn.py -f REL SRC VERSION SRC2 VERSION2 SRC3 VERSION3 (fetches a new USN, include other arguments from above as needed)

    • Example $UCT/scripts/prepare-kernel-usn.py -f xenial linux 4.4.0-70.91 linux-raspi2 4.4.0-1050.57 linux-snapdragon 4.4.0-1053.57  linux-aws 4.4.0-1011.20 linux-gke 4.4.0-1008.8

    • lts-backport/HWE kernels should be a XXXX-2 USN of the kernel they are derived from, drop the -f and use --usn XXXX-2 when invoking prepare-kernel-usn.py.

    • Bits of the USN to be sure to edit:

      • Change the --title (aka Subject) to just refer to either "Linux kernel vulnerabilities" or a specific subtype "Linux (HWE) vulnerabilities" (examples 3696-1 3696-2 (HWE))

      • Change the --summary to drop the meta and signed packages and re-organize so either linux, linux-hwe, or linux-lts-RELEASE is first, with derived kernels afterward.

      • Drop the meta and signed sources from the source descriptions section.

      • Drop the meta and signed source packages from the source packages section.

      • Prune the binary meta packages so that the meta packages correspond with the actual binary packages
  • ESM kernels are published separately from regular Ubuntu USNs:
    • for lts-backport/HWE kernels, use a -2 USN
    • for primary ESM kernel, fetch a new USN

    • again, use --include-eol --ppa ubuntu-esm/esm as addiitonal arguments to prepare-kernel-usn.py

    • because the esm ppa is not public, the process where it tries to ensure kernels have been published to the archive will fail.

    • you will need to be a member of https://launchpad.net/~canonical-kernel-security-team/+members

  • set the USN env variable: export USN="1000-1".

  • You now need to do steps 6 to 11 of Announce Publication. You can pass multiple USN numbers to $UCT/scripts/publish-usn-to-website

Adding tracking for a new derived kernel

For each new added kernel to support, we need to add tracking to the CVE tracker

  • Make sure the $UCT tree is up to date: cd $UCT && git pull --ff-only

  • Use $UCT/scripts/add-derived-kernel to add the basic tracking; it will pull CVE statuses from the kernel release it's derived from. Examples:

    • add xenial linux-aws kernel derived from primary xenial kernel: ./scripts/add-derived-kernel -r xenial -d xenial aws

    • add linux-lts-vivid for the trusty release: ./scripts/add-derived-kernel -r trusty -d vivid lts-vivid

  • Update scripts as documented from the output at the end of running scripts/add-derived-kernel

Deprecated bits

Reviewing the state of the CVEs between UCT, the kernel team's UCT tree, and the USN database should happen at least daily. In practice, the USN comparison usually happen much more rarely due to its current fragility.

  • Update git tree: cd $UCT && git fetch --all -p -t && git pull --ff-only

  • UCT merge with kernelteam: ./scripts/process_cves merge

  • [Ignore this process for the time being] sync UCT to USNs (for any CVEs that have changed state, been revoked, etc)

    • fetch the full USN database: ./scripts/fetch-db database-all.pickle.bz2

    • run report: ./scripts/report-mismatched-cve-fixes.py

    • pull out hair, fix things (Important prerequisites: adequate sleep, money for swear jar)
    • declare a social lock on database-all.pickle
    • refetch and unpack database-all.pickle

    • perform any moves/insertions: ./scripts/report-mismatched-cve-fixes.py -u --ignore-...

    • keep a backup of the database: ssh people.canonical.com "cp ~ubuntu-security/usn/database-all.pickle ~ubuntu-security/usn/database-all.pickle.$(date +%Y-%m-%d)"

    • upload updated database: scp database-all.pickle people.canonical.com:~ubuntu-security/usn/

    • declare unlock

    • Publish the USN changes: ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"

    • Use the "Updated:" report to refresh affected USNs (w3m -dump http://www.ubuntu.com/usn/update/usn-$USN)

    • Fetch updated non-all database: ./scripts/fetch-db database.pickle.bz2

    • Mark pending entries as released: ./scripts/sync-from-usns.py -u

    • Commit the tree, rejoice

(NOTE: we no longer require a bug report for each CVE being addressed.)