KernelMaintenance
|
Size: 16149
Comment: Add comments to security section.
|
Size: 21852
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 2: | Line 2: |
| [[Include(KernelTeamHeader)]] ||<tablestyle="float:right; font-size: 0.9em; width:40%; background:#F1F1ED; margin: 0 0 1em 1em;" style="padding:0.5em;">'''Contents'''[[BR]][[TableOfContents]]|| |
<<Include(KernelTeamHeader)>> ||<tablestyle="float:right; font-size: 0.9em; width:40%; background:#F1F1ED; margin: 0 0 1em 1em;" style="padding:0.5em;"><<TableOfContents>>|| |
| Line 16: | Line 16: |
| This requires some knowledge of the Debian build system (including the ''debian/rules'', etc). The ''gutsy'' build system was rewritten from the ground up, to mainly exclude ''kernel-package'', since it was getting too heavy and restrictive for the kernels builds. We still use ''kernel-wedge'' for udeb creation. | This requires some knowledge of the Debian build system (including the ''debian/rules'' file, etc). During gutsy, the build system was rewritten from the ground up, mainly to remove the use of ''kernel-package'' as a build tool, since it was getting too heavy and restrictive for our needs. We still use ''kernel-wedge'' for udeb creation. |
| Line 33: | Line 33: |
| When the kernels are built, the common and flavour specific config files are concatenated together. | When the kernels are built, the common and flavour specific config files are concatenated together in the build directory. |
| Line 52: | Line 52: |
| 2.6.20-6-generic }}} The ABI in the above example is '''6'''. The ABI is represented in the ''debian/changelog'' as: {{{ linux-source-2.6.20 (2.6.22-1.3) feisty; urgency=low }}} The minor version after the ABI is an ever increasing counter. It basically denotes how many uploads have been made for this kernel version. |
2.6.26-1-generic }}} The ABI in the above example is '''1'''. The ABI is represented in the ''debian/changelog'' as: {{{ linux (2.6.26-1.1) intrepid; urgency=low }}} The minor version after the ABI is an ever increasing revision. It basically denotes how many uploads have been made for this kernel version. |
| Line 73: | Line 73: |
| Module lists are tracked similarly to the ABI, and are kept in ''debian/abi/<prev-ver>/<arch>/<flavour>.modules''. The file is a basename sorted list, sans .ko extension, of all the modules in a particular flavour. This is checked regardless of ABI, since it's meant to track release-to-release to avoid modules going missing by mistake. The check will honor ''debian/abi/<prev-ver>/<arch>/ignore'' to ignore an entire architecture, or ''debian/abi/<pre-ver>/<arch>/<flavour>.ignore for a specific kernel flavour. |
Module lists are tracked similarly to the ABI, and are kept in `debian/abi/<prev-ver>/<arch>/<flavour>.modules`. The file is a basename sorted list, sans .ko extension, of all the modules in a particular flavour. This is checked regardless of ABI, since it's meant to track release-to-release to avoid modules going missing by mistake. The ABI check will honor `debian/abi/<prev-ver>/<arch>/ignore` to ignore an entire architecture, or `debian/abi/<pre-ver>/<arch>/<flavour>.ignore` for a specific kernel flavour. This is useful for architectures that failed to build the last ABI. The module check will honor a list of modules in `debian/abi/<prev-ver>/modules.ignore` that lists modules known to have gone missing since the last upload. === GIT Hooks === On kernel.ubuntu.com there is a git repo called buildscripts. It also contains some git hooks that we use to ensure policy compliance for commits to our tree. If you are doing a lot of work with the Ubuntu Linux kernel tree, you should put these scripts into the `.git/hooks/` directory of your working tree. |
| Line 79: | Line 85: |
| This section describes all the steps involved in a common development cycles. That is, the cycles from one upload (release) of the Ubuntu kernel to the next. | This section describes all the steps involved in a common development cycle. That is, the cycles from one upload (release) of the Ubuntu kernel to the next. |
| Line 86: | Line 92: |
| debian/bin/misc/getabis 2.6.22 6.11 }}} This will download all the previous deb's, and extract the ABI files, and module lists. You may find it easier to do this from a DC machine where the archive is closer too (downloading all the kernel deb's is a bandwidth eater). |
bash debian/scripts/misc/getabis 2.6.26 1.1 }}} In the above example, `2.6.26` is the kernel version, and `1.1` is the last uploaded revision. This will download all the previous deb's, and extract the ABI files, and module lists. You may find it easier to do this from a DC machine where the archive is closer (downloading all the kernel deb's is a bandwidth eater). |
| Line 101: | Line 109: |
| === Add new code === Now you can rebase your tree, pull from other branches or apply patches that are going into this update. === Updating configs(optional) === As new drivers are added and merges are done with upstream, it becomes necessary to update our configs in ''debian/config/''. This is easily done for all architectures with one simple command: {{{ debian/rules updateconfigs }}} This runs '''silentoldconfig''' for all flavours, so if any possible config options are unanswered, it will ask you. Alternatively, if you know of a new config option, you can alleviate answering the same question for it on all flavours by appending it to the ''debian/config/<arch>/config'' files, and running the above command. Once the configs are updated, you simply commit the new changes: {{{ git-add debian/config/ git-commit -s -F debian/commit-templates/update-configs }}} === Build === Now build the kernel for all archs and flavours |
|
| Line 103: | Line 136: |
| If the build fails because of the ABI checker, this is a sign that the ABI needs to be bumped. |
|
| Line 111: | Line 146: |
| For example, if the current changelog looks like: {{{ linux (2.6.26-1.1) intrepid; urgency=low }}} And an ABI bump is required, you would change the above line to look like: {{{ linux (2.6.26-2.2) intrepid; urgency=low }}} If an ABI bump is required for the kernel, it is also required for L-U-M and L-R-M git repositories. '''NOTE''': LUM is no longer used in intrepid and beyond. A number of parties have requested notice of ABI changes since it affects third party modules. Send an email with the kernel release version and ABI number to kernel-team@lists.ubuntu.com and ubuntu-installer@lists.ubuntu.com. |
|
| Line 131: | Line 182: |
| During development cycle, when we are following upstream git very closely, we will use '''git-rebase''' to stay synced to upstream. Once upstream version has been released, we no longer rebase. This may cause times where you need to force a GIT pull (because the '''master''' of our tree is now rebased). |
|
| Line 133: | Line 186: |
| In prior releases, third party drivers were kept as part of the main kernel tree in the ubuntu/ directory. This has changed in gutsy. Third party drivers are now separate, and in the linux-ubuntu-modules package. === Updating configs === As new drivers are added and merges are done with upstream, it becomes necessary to update our configs in ''debian/config/''. This is easily done for all architectures with one simple command: {{{ debian/rules updateconfigs }}} This runs '''silentoldconfig''' for all flavours, so if any possible config options unanswered, it will ask you. Alternatively, if you know of a new config option, you can alleviate answering the same question for it on all flavours by prepending it to the ''debian/config/<arch>/config'' files, and running the above command. Once the configs are updated, you simply commit the new changes: {{{ git-update-index debian/config/*/* git-commit -s -F debian/commit-templates/update-configs }}} |
During gutsy, feisty and hardy, third-party drivers were kept in a package called linux-ubuntu-modules. They are now kept in a subdirectory `ubuntu/` in the main kernel tree. Modules added by Ubuntu have to meet some basic criteria: * Modules added to the main tree must be open source. * Must be stable, and pass code review. * Must support hardware and/or features relevant to a good portion of users. It is preferable that these drivers are maintained upstream, and be scheduled for inclusion. In some cases, we keep drivers that do not meet these last criteria. However, this is mainly due to being grandfathered in and our policy of not losing support for hardware. |
| Line 157: | Line 201: |
| * It may also be a good idea to run ''git-ls-files --others'' to check for extraneous files. | |
| Line 160: | Line 205: |
| <open edit debian/changelog in editor to verify release and date> git-update-index debian/changelog git-commit -s -F debian/commit-templates/updateconfigs |
<open debian/changelog in an editor to verify release and date> git-add debian/changelog git-commit -s -m "UBUNTU: Ubuntu-2.6.20-6.11" |
| Line 169: | Line 214: |
| * Run dpkg-buildpackage. '''NOTE''': Requires latest dpkg-dev so that it ignores GIT meta-data. | * If the tree has a .orig.tar.gz in the repo, download that to the same directory as the git tree. * Run dpkg-buildpackage: |
| Line 171: | Line 217: |
| dpkg-buildpackage -S -rfakeroot | dpkg-buildpackage -S -rfakeroot -I.git -I.gitignore -i'\.git.*' |
| Line 173: | Line 219: |
| or to build _with_ source use: {{{ dpkg-buildpackage -S -rfakeroot -I.git -I.gitignore -i'\.git.*' -sa }}} |
|
| Line 174: | Line 226: |
It should be noted that for development releases, a new tarball is generated at each upload. However at release time, a .orig.tar.gz is generated, sans ''debian/'' directory (but with ''debian/firmware/'', to more easily track changes after release. |
* Submit your patch for upstream inclusion when appropriate, e.g., quirk updates. It should be noted that for development releases, a new tarball is generated at each upload. However at release time, a .orig.tar.gz is used, which is based on the upstream tarball of the released kernel we are using. |
| Line 179: | Line 232: |
| The kernel team should push changes to our internal repo (see KernelGitGuide for public repo). To make this easy, you can setup a script called ''push-gutsy'' with this: | The kernel team should push changes to our repo (see KernelGitGuide for public repo). To make this easy, you can setup a script called ''push-intrepid'' with this: |
| Line 185: | Line 238: |
| remotes="rookery.ubuntu.com:/srv/kernel-team/private/ubuntu-gutsy.git" | remotes="zinc.ubuntu.com:/srv/kernel.ubuntu.com/git/ubuntu/ubuntu-intrepid.git" |
| Line 199: | Line 252: |
| push-gutsy Ubuntu-2.6.22-1.3 }}} |
push-intrepid Ubuntu-2.6.26-1.1 }}} '''NOTE''': Each release tree is someone's responsibility. If it's not yours, it's probably best to email patches to kernel-team@lists.ubuntu.com rather than committing yourself. |
| Line 211: | Line 266: |
| URL: ssh://master.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.20.y.git | URL: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.20.y.git |
| Line 232: | Line 287: |
| === Grab the Pre-requisite Tools === {{{ For Hardy and older: $ sudo apt-get install linux-kernel-devel devscripts fakeroot libncurses5-dev ccache For Intrepid and newer: $ sudo apt-get install build-essential devscripts fakeroot libncurses5-dev ccache $ sudo apt-get install kernel-wedge makedumpfile xmlto docbook-utils gs transfig sharutils $ sudo apt-get build-dep linux }}} A good way to ensure that you have all of the tools installed that are required to build a package is to use 'dbuild -b'. If you are missing something, it will fail very quickly. === Pre-upload test builds === A set of scripts to help farm out builds across all supported architectures is described in https://wiki.ubuntu.com/KernelTeam/KernelBuildScripts === Quick builds === |
|
| Line 235: | Line 310: |
| $ cd ubuntu-gutsy | $ cd ubuntu-intrepid |
| Line 264: | Line 339: |
| So you can change ''generic'' with any flavour you want to build. | So you can change ''generic'' with any flavour you want to build. You may also want to do: {{{ fakeroot debian/rules binary-headers }}} to get the architecture independent headers package. LUM builds are done using: {{{ fakeroot debian/rules binary-modules-<flavour> }}} where flavour = generic, ume, rt, xen, etc. |
| Line 275: | Line 361: |
| * '''skipabi''' - Used for test only, causes the build to skip the ABI consistency check. * '''skipmodule''' - Same as above, but for the module check. |
* '''skipabi=true''' - Used for test only, causes the build to skip the ABI consistency check. * '''skipmodule=true''' - Same as above, but for the module check. |
| Line 278: | Line 364: |
| * '''PRINTSHAS''' - Include GIT SHAs in '''printchanges''' and '''insertchanges''' targets. E.g. When trying to avoid "previous or current ABI file missing" and not wanting to do an ABI-bump: {{{ CONCURRENCY_LEVEL=8 fakeroot debian/rules binary-generic binary-headers skipabi=true skipmodule=true }}} |
|
| Line 283: | Line 374: |
| However, for an ABI changing upload, packages need to be rebuilt against the linux-headers. Note that these packages have a extra version who's major matches the ABI of the kernel it is being built against. | However, for an ABI changing upload, packages need to be rebuilt against the linux-headers. Note that these packages have an extra version whose major matches the ABI of the kernel it is being built against. |
| Line 291: | Line 382: |
| * Can someone please put up a bit of info on how to get this package from git (and re-build the package), the way KernelGitGuide does? The same for linux-ubuntu-modules. --BogdanButnaru * I'm not certain, but I believe this can't be done with git. instead, {{{apt-get source linux-restricted-modules-VERSION}}} then look for the ''abi-version'' in debian/rules and make sure it corresponds to the abi of your new kernel. then {{{fakeroot dpkg-buildpackage }}} which should build the package properly. * Edit: linux-ubuntu-modules are (as far as I can tell) built exactly the same as the kernel, but using {{{ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-gutsy-lum.git ubuntu-gutsy-lum}}} for getting the sources, and {{{fakeroot debian/rules binary-modules-generic}}} for building the package. I'm not sure about linux-restricted-modules; it's not in git, is it? --BogdanButnaru |
|
| Line 293: | Line 392: |
| Same as with '''linux-restricted-modules'''. | This package is based on the linux-source build system, so shares the same intricacies. Setting the ABI is the same as in linux-source (via ''debian/changelog''). A quick summary: to checkout, use {{{ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-hardy-lum.git ubuntu-hardy-lum }}} and to build, use {{{fakeroot debian/rules binary-modules-FLAVOUR}}} Please note that the alsa tree is now in linux-ubuntu-modules, so if you're having sound issues you might want to try building this package in addition to the main kernel package. --MattPrice |
| Line 305: | Line 409: |
| The ABI bump method is the same as for linux-meta. | This package is based heavily on ''linux-ubuntu-modules''. |
| Line 317: | Line 421: |
| Technically known as ''<rel>-proposed'' (e.g. ''edgy-proposed'') is where we can push changes that need testing, and are unsuited for a wide audience of users who expect a rock solid distribution. | Technically known as `<rel>-proposed` (e.g. `edgy-proposed`) is where we can push changes that need testing, and are unsuited for a wide audience of users who expect a rock solid distribution. |
| Line 321: | Line 425: |
| The GIT tree is named ''ubuntu-<rel>-updates.git''. It's ABI remains a constant '''-50''', and the changelog for this release never rolls, meaning the it only has one changelog set for every upload. This changelog set is updated every upload to reflect new patches, and to remove ones that are either broken, or have been merged to the stable kernel tree. | The GIT tree is named `ubuntu-<rel>-updates.git`. It's ABI remains a constant '''-50''', and the changelog for this release never rolls, meaning the it only has one changelog set for every upload. This changelog set is updated every upload to reflect new patches, and to remove ones that are either broken, or have been merged to the stable kernel tree. |
| Line 327: | Line 431: |
| This is self explanatory, or so it would seem. While the ''ubuntu-<rel>.git'' tree is meant primarily for security updates, occasionally stable changes are merged into it from proposed. | This is self explanatory, or so it would seem. While the `ubuntu-<rel>.git` tree is meant primarily for security updates, occasionally stable changes are merged into it from proposed. |
| Line 330: | Line 434: |
| CategoryKernel | CategoryKernel CategoryDocumentation |
This document is geared towards gutsy and later releases. For feisty, please see KernelMaintenanceFeisty.
This is meant as a reference for the kernel team.
Getting the source
Ubuntu kernel source is available via git. More information can be obtained from KernelGitGuide.
Build system breakdown
This requires some knowledge of the Debian build system (including the debian/rules file, etc). During gutsy, the build system was rewritten from the ground up, mainly to remove the use of kernel-package as a build tool, since it was getting too heavy and restrictive for our needs. We still use kernel-wedge for udeb creation.
Flavours
Flavours are how we define different types of kernels on the same architecture. Each flavour can be targeted to a specific CPU, or provide some functionality for a specific purpose.
For example, the amd64 (x86_64) architecture has these flavours:
- generic - The default install kernel.
- server - Optimized for server installations.
When building on the amd64 architecture, these are the kernels that are built. In reality, they only differ in slight variations of kernel config options.
Configs
Each flavour is built using a pre-defined configuration file (used by the kernel build system). Each architecture contains it's config files in debian/config/<arch>/. The files are split for each flavour in debian/config/<arch>/config.<flavour>, and common options are shared in debian/config/<arch>/config.
When the kernels are built, the common and flavour specific config files are concatenated together in the build directory.
debian/control
The debian/control file is generated from debian/control.stub by the kernel-wedge command.
The debian/control.stub file is generated from debian/control.stub.in by a debian/rules target. This is populated by files defining the flavours in debian/control.d/vars.*.
The vars files define package descriptions for each flavour.
debian/changelog
The changelog file is where we track all changes made to the repository. This is a standard debian/changelog. However, changes are not added to it manually. Instead, it is done automatically at release time using git-log. See below for process.
ABI
The ABI is probably one of the most important features of the build system. In the kernel package naming, the ABI is the numerical component immediately following the kernel version. E.g.
2.6.26-1-generic
The ABI in the above example is 1. The ABI is represented in the debian/changelog as:
linux (2.6.26-1.1) intrepid; urgency=low
The minor version after the ABI is an ever increasing revision. It basically denotes how many uploads have been made for this kernel version.
The kernel ABI is derived from the Modules.symver file produced during the kernel build. Each exported function is given a hash defining the function's arguments and return value, making it easy to detect changes which can break modules built against the kernel (such as our own linux-restricted-modules package).
The ABI files for the previous build are located in debian/abi/<prev-rev>/. The layout is similar to the debian/config/ directory in that each architecture is a subdirectory, and each flavour is a file with the ABI for that flavour in it.
When the kernel build process finishes, it checks each flavour against the prev ABI. If the previous and current ABI's are the same version (e.g. 6), and the ABI has changed, the build will fail. This signifies that an ABI bump is needed (see below).
As of now, a single ABI change in any flavour results in an ABI bump for the entire build. There is no per-flavour or per-architecture ABI number. This is for sanity reasons.
Module lists
Module lists are tracked similarly to the ABI, and are kept in debian/abi/<prev-ver>/<arch>/<flavour>.modules. The file is a basename sorted list, sans .ko extension, of all the modules in a particular flavour. This is checked regardless of ABI, since it's meant to track release-to-release to avoid modules going missing by mistake.
The ABI check will honor debian/abi/<prev-ver>/<arch>/ignore to ignore an entire architecture, or debian/abi/<pre-ver>/<arch>/<flavour>.ignore for a specific kernel flavour. This is useful for architectures that failed to build the last ABI.
The module check will honor a list of modules in debian/abi/<prev-ver>/modules.ignore that lists modules known to have gone missing since the last upload.
GIT Hooks
On kernel.ubuntu.com there is a git repo called buildscripts. It also contains some git hooks that we use to ensure policy compliance for commits to our tree. If you are doing a lot of work with the Ubuntu Linux kernel tree, you should put these scripts into the .git/hooks/ directory of your working tree.
Development cycle
This section describes all the steps involved in a common development cycle. That is, the cycles from one upload (release) of the Ubuntu kernel to the next.
Updating ABI and modules lists
To start a new release, you need to download the ABI and module files from the previous release. To do this, use:
bash debian/scripts/misc/getabis 2.6.26 1.1
In the above example, 2.6.26 is the kernel version, and 1.1 is the last uploaded revision.
This will download all the previous deb's, and extract the ABI files, and module lists. You may find it easier to do this from a DC machine where the archive is closer (downloading all the kernel deb's is a bandwidth eater).
Starting a new release
Immediately after an upload you should run:
debian/rules startnewrelease
This will generate a new debian/changelog entry for the next revision number, and modify appropriate other files under the debian/ directory.
Add new code
Now you can rebase your tree, pull from other branches or apply patches that are going into this update.
Updating configs(optional)
As new drivers are added and merges are done with upstream, it becomes necessary to update our configs in debian/config/. This is easily done for all architectures with one simple command:
debian/rules updateconfigs
This runs silentoldconfig for all flavours, so if any possible config options are unanswered, it will ask you. Alternatively, if you know of a new config option, you can alleviate answering the same question for it on all flavours by appending it to the debian/config/<arch>/config files, and running the above command.
Once the configs are updated, you simply commit the new changes:
git-add debian/config/ git-commit -s -F debian/commit-templates/update-configs
Build
Now build the kernel for all archs and flavours
Bumping the ABI
If the build fails because of the ABI checker, this is a sign that the ABI needs to be bumped.
If it becomes necessary to increase the ABI, you simply need to edit debian/changelog, increase the ABI in the version, then run this command:
debian/rules debian/control.stub
This will regenerate all of the control files with the appropriate ABI.
For example, if the current changelog looks like: