Contents

  1. Setting up and using sbuild
    1. Creating the schroots
    2. Deleting a schroot
    3. Using the schroot
    4. Maintaining the schroots
    5. Expiring active schroot sessions
  2. Other hints, tips, and tricks
    1. Mount your home dir
    2. Parallel builds
    3. Temporarily adding PPAs
    4. Local packages
    5. bzr-debuild
    6. CCache
    7. EatMyData
    8. ARM
    9. Pre-populated chroots
    10. Dirty builds
    11. Juju

This is an extract from SecurityTeam/BuildEnvironment that tries to make it as quick and easy as possible to reduce your pbuilder / PPA / distro build failure rate and improve your build and turnaround time, as well as allow you to debug the builds in an environment close to the Ubuntu builders.

This has been tested on all stable releases starting with Xenial, although this guide is geared towards the current development release, Bionic.

This guide assumes that your host architecture is amd64, although you can create schroots for different architectures. If you do so, remember to substitute amd64 with your arch in the following instructions. Step 10 has more details on creating an schroot that is not amd64.

If you are not using the stock Ubuntu kernel, you need to make sure your kernel supports overlayfs (the mainline PPA kernels do not).

The original wiki page deals with many more details (especially around security) that are not required on a daily basis, but make it quite cumbersome to actually follow the guide. If anything here doesn't work, or is unclear, or you want more detail, check the other page - maybe there's a solution to your issue. You could also try looking at the Debian wiki page, which uses a slightly different setup procedure than this page, but the procedures for ccache, autopkgtests, etc. are correct.

Setting up and using sbuild

Creating the schroots

  1. Install sbuild and schroot:

    $ sudo apt install sbuild debhelper ubuntu-dev-tools piuparts

  2. Optional: Install a local package proxy, so that downloaded packages get cached between sbuild/schroot runs: 

    $ sudo apt install apt-cacher-ng
$ echo 'Acquire::http::Proxy "http://127.0.0.1:3142";' | sudo tee /etc/apt/apt.conf.d/01acng

  3. Make sure you are in the 'sbuild' group:

    $ sudo adduser $USER sbuild
  4. Setup for mounting $HOME and extracting the ddebs:

    • Create the directory for the mount point: 

      $ mkdir -p $HOME/ubuntu/scratch

    • Append to /etc/schroot/sbuild/fstab:

      /home/<username>/ubuntu/scratch  /scratch          none  rw,bind  0  0
      If you use an encrypted home directory, your $HOME is mounted differently,

      so you will also have to add:

      /home/<username>                 /home/<username>  none  rw,bind  0  0

  5. Create ~/.sbuildrc:

    # Name to use as override in .changes files for the Maintainer: field
# (mandatory, no default!).
$maintainer_name='Your Name <user@ubuntu.com>';

# Default distribution to build.
$distribution = "bionic";
# Build arch-all by default.
$build_arch_all = 1;

# When to purge the build directory afterwards; possible values are "never",
# "successful", and "always".  "always" is the default. It can be helpful
# to preserve failing builds for debugging purposes.  Switch these comments
# if you want to preserve even successful builds, and then use
# "schroot -e --all-sessions" to clean them up manually.
$purge_build_directory = 'successful';
$purge_session = 'successful';
$purge_build_deps = 'successful';
# $purge_build_directory = 'never';
# $purge_session = 'never';
# $purge_build_deps = 'never';

# Directory for writing build logs to
$log_dir=$ENV{HOME}."/ubuntu/logs";

# don't remove this, Perl needs it:
1;

  6. Make the following directory (change if specified something different in ~/.sbuildrc):

    $ mkdir -p $HOME/ubuntu/{build,logs}

  7. Create ~/.mk-sbuild.rc:

    SCHROOT_CONF_SUFFIX="source-root-users=root,sbuild,admin
source-root-groups=root,sbuild,admin
preserve-environment=true"
# If you work on security uploads, you want to skip both proposed and updates.
# Otherwise (e.g. working on SRUs or on the devel series), leave them enabled
# to mirror the Launchpad builder configuration.
# SKIP_UPDATES="1"
# SKIP_PROPOSED="1"
# if you have e.g. apt-cacher-ng around
# DEBOOTSTRAP_PROXY=http://127.0.0.1:3142/

  8. Enter the sbuild group if you were added during this session:

    $ sg sbuild

  9. Finally, create the schroots:

    $ mk-sbuild bionic

    This creates the schroot for your current architecture but if you would like to specify another architecture, use the following command instead:

    $ mk-sbuild bionic --arch=<ARCH>

    So for i386, you would use:

    $ mk-sbuild bionic --arch=i386

    You cannot specify an architecture that is not supported by your processor (for example, you cannot create an amd64 schroot on an i386 system). Use uname -p if you are unsure what your system supports.

    See man mk-sbuild for more details. Note that this will automatically use the host's apt-cacher-ng for the generated schroot if you installed it above.

  10. Optional: It can often be useful to perform builds in /dev/shm, which is a tmpfs directory in RAM. While there are limitations on what you can compile and the number of concurrent builds that can be performed (based on how much memory your build machine has), building in /dev/shm can lead to much faster build times and reduce disk I/O. A way to create a 'shared' source schroot like so:

    $ sudo sh -c "sed -e 's#]#-shm]#' \
  -e 's#^\(directory=.*\)#\1\nunion-overlay-directory=/dev/shm/schroot/overlay#' \
  /etc/schroot/chroot.d/* > /etc/schroot/chroot.d/shm-overlays.conf"
$ schroot --list

    Please note that the directory you specify for 'union-overlay-directory' must exist before using the shm chroot (can add an entry to /etc/rc.local):

    $ mkdir -p /dev/shm/schroot/overlay/
    It might also be useful to remount /dev/shm with more memory than the default, which is 50% of RAM. Can adjust like so: 
    $ sudo mount -o remount,size=75% /dev/shm
    On the other hand, it is possible for a tmpfs to fill up and cause problems in some fairly common cases, since builds can easily approach the order of the size of RAM. It is perfectly reasonable to skip this step and allow sbuild to use a disk-backed overlay, even though builds will be a bit slower as a result.

Deleting a schroot

  1. Remove the stanza for the chroot:

    $ sudo rm /etc/schroot/chroot.d/sbuild-bionic-amd64

  2. Remove the chroot from the disk:

    $ sudo rm -rf /var/lib/schroot/chroots/bionic-amd64

  3. Optional Run the shm snippet above.

Using the schroot

Maintaining the schroots

See also Barry's chup script for an easy way to keep your chroots up-to-date.

Expiring active schroot sessions

Other hints, tips, and tricks

Mount your home dir

Parallel builds

Temporarily adding PPAs

Local packages

You sometimes need newer packages (libraries, whatnot) to build against. Sometimes you might have a new dependency that isn't yet in the archive, but you are working on the new packaging. BarryWarsaw did some nice things to help with that. Here's an article on his blog which describes how to build Debian packages with local dependencies (and an older article with some now obsolete recommendations). You can also check out his repo of useful tools and scripts.

  1. Clone repotools to your scratch:

    $ git clone https://git.launchpad.net/~barry/+git/repotools ~/ubuntu/repo

  2. Install sponge:

    $ sudo apt install moreutils

  3. Modify clean.sh and scan.sh to set the repository path:

    ...
where=/home/<username>/ubuntu/repo
...

  4. Add the following snippet to your ~/.sbuildrc:

    $external_commands = {
    "pre-build-commands" => [
        ['/home/'.$ENV{USER}.'/ubuntu/repo/scan.sh'],
    ],
    "chroot-setup-commands" => [
        ['/repo/prep.sh'],
    ],
};

  5. And add the following lines to /etc/schroot/sbuild/fstab:

    # Expose local apt repository to the chroot
/home/<username>/ubuntu/repo     /repo             none  ro,bind  0  0

  6. Create a directory to hold your local debs:

    $ mkdir ~/ubuntu/debs

  7. Copy your local debs into ~/ubuntu/debs

  8. Start an http server that will act as a local repository: 

    $ cd ~/ubuntu/debs
$ python3 -m http.server 8000 --bind 127.0.0.1

  9. Do the build:

    $ sbuild -d unstable \
  --extra-repository="deb [trusted=yes] http://127.0.0.1:8000 ./" \
  build-area/foo_9.0.1-1.dsc

See the blog post for more details and recommendations (e.g. how to run autopkgtest against local debs).

bzr-debuild

CCache

EatMyData

ARM

ARM should basically Just Work, e.g. by creating an armhf chroot:

Now, everything works just as if you were building i386 packages on amd64. Just use "armhf" as the host architecture. See CrossBuilding for more info.

Pre-populated chroots

A lot of time is spent on apt and dpkg installing build dependencies. You can just create a custom chroot, e.g. for bionic:

Start a root session that makes persistent changes:

Install all build dependencies for project foo:

Then, when building with that chroot, you can skip apt updates, and it will go straight into building:

Dirty builds

If you want to save even more time, you can use "dirty" builds. Prepare a pre-populated chroot as above, and either build packages:

Or just configure and build yourself:

Juju

CategoryPackaging

SimpleSbuild (last edited 2024-08-14 13:47:17 by schopin)