Ubuntu Wiki

This is an extract from SecurityTeam/BuildEnvironment that tries to make it as quick and easy as possible to reduce your pbuilder / PPA / distro build failure rate and improve your build and turnaround time, as well as allow you to debug the builds in an environment close to the Ubuntu builders.

This has been tested on Zesty, Yakkety, Xenial, and Trusty, although this guide will highlight instructions for Zesty.

This guide assumes that your host architecture is amd64, although you can create schroots for different architectures. If you do so, remember to substitute amd64 with your arch in the following instructions. Step 10 has more details on creating an schroot that is not amd64.

If you are not using the stock Ubuntu kernel, you need to make sure your kernel supports overlayfs (the mainline PPA kernels do not).

The original wiki page deals with many more details (especially around security) that are not required on a daily basis, but make it quite cumbersome to actually follow the guide. If anything here doesn't work, or is unclear, or you want more detail, check the other page - maybe there's a solution to your issue.

This does not work if /tmp is mounted under btrfs due to incompatibilities with aufs. A workaround is to mount /tmp as a tmpfs, but even this might not work 100% of the time. It's best to just use ext4.

Setting up and using sbuild

Creating the schroots

1. Install sbuild and schroot:

   $ sudo apt install sbuild debhelper ubuntu-dev-tools

2. Optional: Install a local package proxy, so that downloaded packages get cached between sbuild/schroot runs:

   $ sudo apt install apt-cacher-ng
   $ echo 'Acquire::http::Proxy "http://127.0.0.1:3142";' | sudo tee /etc/apt/apt.conf.d/01acng

3. Make sure you are in the 'sbuild' group:

   $ sudo adduser $USER sbuild

4. Setup for mounting $HOME and extracting the ddebs:

   • Create the directory for the mount point:

     $ mkdir -p $HOME/ubuntu/scratch

   • Append to /etc/schroot/sbuild/fstab:

     /home/<username>/ubuntu/scratch  /scratch          none  rw,bind  0  0

     If you use an encrypted home directory, your $HOME is mounted differently,

     so you will also have to add:

     /home/<username>                 /home/<username>  none  rw,bind  0  0

5. Create ~/.sbuildrc:

   # Mail address where logs are sent to (mandatory, no default!)
   $mailto = $ENV{USER};
   
   # Name to use as override in .changes files for the Maintainer: field
   # (mandatory, no default!).
   $maintainer_name='Your Name <user@ubuntu.com>';
   
   # Default distribution to build.
   $distribution = "zesty";
   # Build arch-all by default.
   $build_arch_all = 1;
   
   # When to purge the build directory afterwards; possible values are "never",
   # "successful", and "always".  "always" is the default. It can be helpful
   # to preserve failing builds for debugging purposes.  Switch these comments
   # if you want to preserve even successful builds, and then use
   # "schroot -e --all-sessions" to clean them up manually.
   $purge_build_directory = 'successful';
   $purge_session = 'successful';
   $purge_build_deps = 'successful';
   # $purge_build_directory = 'never';
   # $purge_session = 'never';
   # $purge_build_deps = 'never';
   
   # Directory for writing build logs to
   $log_dir=$ENV{HOME}."/ubuntu/logs";
   
   # don't remove this, Perl needs it:
   1;

6. Make the following directory (change if specified something different in ~/.sbuildrc):

   $ mkdir -p $HOME/ubuntu/{build,logs}

7. Create ~/.mk-sbuild.rc:

   SCHROOT_CONF_SUFFIX="source-root-users=root,sbuild,admin
   source-root-groups=root,sbuild,admin
   preserve-environment=true"
   # you will want to undo the below for stable releases, read `man mk-sbuild` for details
   # during the development cycle, these pockets are not used, but will contain important
   # updates after each release of Ubuntu
   SKIP_UPDATES="1"
   SKIP_PROPOSED="1"
   # if you have e.g. apt-cacher-ng around
   # DEBOOTSTRAP_PROXY=http://127.0.0.1:3142/

8. Enter the sbuild group if you were added during this session:

   $ sg sbuild

9. Generate a GPG keypair for sbuild to use:

   $ sbuild-update --keygen

10. Finally, create the schroots:

    $ mk-sbuild zesty

    This creates the schroot for your current architecture but if you would like to specify another architecture, use the following command instead:

    $ mk-sbuild zesty --arch=<ARCH>

    So for i386, you would use:

    $ mk-sbuild zesty --arch=i386

    You cannot specify an architecture that is not supported by your processor (for example, you cannot create an amd64 schroot on an i386 system). Use uname -p if you are unsure what your system supports.

    See man mk-sbuild for more details. Note that this will automatically use the host's apt-cacher-ng for the generated schroot if you installed it above.

11. Optional: It can often be useful to perform builds in /dev/shm, which is a tmpfs directory in RAM. While there are limitations on what you can compile and the number of concurrent builds that can be performed (based on how much memory your build machine has), building in /dev/shm can lead to much faster build times and reduce disk I/O. A way to create a 'shared' source schroot like so:

    $ sudo sh -c "sed -e 's#]#-shm]#' \
      -e 's#^\(directory=.*\)#\1\nunion-overlay-directory=/dev/shm/schroot/overlay#' \
      /etc/schroot/chroot.d/* > /etc/schroot/chroot.d/shm-overlays.conf"
    $ schroot --list

    Please note that the directory you specify for 'union-overlay-directory' must exist before using the shm chroot (can add an entry to /etc/rc.local):
    

    $ mkdir -p /dev/shm/schroot/overlay/ 

    It might also be useful to remount /dev/shm with more memory than the default, which is 50% of RAM. Can adjust like so:

    $ sudo mount -o remount,size=75% /dev/shm

    On the other hand, it is possible for a tmpfs to fill up and cause problems in some fairly common cases, since builds can easily approach the order of the size of RAM. It is perfectly reasonable to skip this step and allow sbuild to use a disk-backed overlay, even though builds will be a bit slower as a result.

Deleting a schroot

1. Remove the stanza for the chroot:

   $ sudo rm /etc/schroot/chroot.d/sbuild-zesty-amd64

2. Remove the chroot from the disk:

   $ sudo rm -rf /var/lib/schroot/chroots/zesty-amd64

3. Optional Run the shm snippet above.

Using the schroot

• Using a schroot is similar to the chroot command but you specify the chroot and user you want to use rather than the directory.

  $ schroot -c zesty-amd64 -u root

  Or without root with:

  $ schroot -c zesty-amd64

  Or building via sbuild directly:

  $ apt source foo
  $ cd ./foo-*
  # do work...
  $ dch -i
  $ update-maintainer
  $ sbuild -d zesty-amd64
  # or from a source package
  $ bzr bd -S
  $ sbuild -d zesty-amd64 ../foo_*.dsc

Maintaining the schroots

• Can see a listing of all your chroots with:

  $ schroot -l
  chroot:zesty-amd64
  chroot:zesty-amd64-shm
  source:zesty-amd64
  source:zesty-amd64-shm
  ...

  The 'source:' chroot is the pristine chroot and you shouldn't go into it unless you want to change something for all future schroots into the chroot. Eg, if you wanted to always have 'vim' installed in your zesty-amd64 chroot, use:

  $ schroot -c source:zesty-amd64 -u root
  $ apt install vim
  $ exit

  Or, to upgrade the schroot (you shouldn't do that with PPAs enabled unless you have a "special" chroot for a certain PPA setup):

  $ schroot -c source:zesty-amd64 -u root
  $ apt update
  $ apt upgrade
  $ exit

  Or, more simply:

  sbuild-update -udc zesty-amd64

See also Barry's chup script for an easy way to keep your chroots up-to-date.

Expiring active schroot sessions

• Sometimes needed if schroots are left hanging around due to a crash or reboot. You can expire all active schroot sessions with:

  $ schroot -e --all-sessions

Other hints, tips, and tricks

Mount your home dir

• To mount your home directory in the chroot, so that you can use files from there, add this entry to /etc/schroot/sbuild/fstab:

  /home/<username>                 /home/<username>  none  rw,bind  0  0

  Note: this is a read-write bind mount, so changes will be written to your real home directory.

Parallel builds

• To make use of all your machine's power, you can use the usual -jN option for sbuild:

  $ sbuild -d zesty-amd64 -j4

Temporarily adding PPAs

• To avoid "polluting" the chroot sbuild provides the options:
  1. --extra-repository=spec

  2. --extra-repository-key=file.asc
     
     

  For a given ppa one needs to get the key signing that ppa. To do so one can go on a ppa's launchpad page and open the section "Technical details about this PPA" In there one finds the "Signing key" id after the /. This key has to be fetched and stored locally to be added to sbuild later.

  As an example, to add https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test for a build

  Find ID BA9EF27F on https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test
  gpg --keyserver keyserver.ubuntu.com --recv BA9EF27F
  gpg --export --armour 'BA9EF27F' > /tmp/myrepo.asc
  gpg --delete-key "BA9EF27F"
  sbuild --extra-repository "<your repo spec>" --extra-repository-key=/tmp/myrepo.asc <your usual options>

  If you just want to add a ppa and don't want to deal with the keys you can use apt feature options to add [trusted=yes] when adding the repo spec, example:

   --extra-repository="deb [trusted=yes] http://ppa.launchpad.net/paelzer/deb-dpdk-16.07/ubuntu zesty main"

  You can also access the source chroot directly in /var/lib/schroot/chroots/zesty-amd64/.

  Another approach is to use sbuild's --chroot-setup-command to run a script which adds the PPA to sources.list and then updates to pick up any packages which are newer in the PPA. It's a little tricky though because you actually have to copy the contents of the PPA's .list file to the top of sources.list. Take a look at this example although you'll almost definitely need to adapt it to your needs.

Local packages

You sometimes need newer packages (libraries, whatnot) to build against. Sometimes you might have a new dependency that isn't yet in the archive, but you are working on the new packaging. BarryWarsaw did some nice things to help with that. Here's an article on his blog which describes how to build Debian packages with local dependencies (and an older article with some now obsolete recommendations). You can also check out his repo of useful tools and scripts.

1. Clone repotools to your scratch:

   $ git clone https://git.launchpad.net/~barry/+git/repotools ~/ubuntu/repo

2. Install sponge:

   $ sudo apt install moreutils

3. Modify clean.sh and scan.sh to set the repository path:

   ...
   where=/home/<username>/ubuntu/repo
   ...

4. Add the following snippet to your ~/.sbuildrc:

   $external_commands = {
       "pre-build-commands" => [
           ['/home/'.$ENV{USER}.'/ubuntu/repo/scan.sh'],
       ],
       "chroot-setup-commands" => [
           ['/repo/prep.sh'],
       ],
   };

5. And add the following lines to /etc/schroot/sbuild/fstab:

   # Expose local apt repository to the chroot
   /home/<username>/ubuntu/repo     /repo             none  ro,bind  0  0

6. Create a directory to hold your local debs:

   $ mkdir ~/ubuntu/debs

7. Copy your local debs into ~/ubuntu/debs

8. Start an http server that will act as a local repository:

   $ cd ~/ubuntu/debs
   $ python3 -m http.server 8000 --bind 127.0.0.1

9. Do the build:

   $ sbuild -d unstable \
     --extra-repository="deb [trusted=yes] http://127.0.0.1:8000 ./" \
     build-area/foo_9.0.1-1.dsc

See the blog post for more details and recommendations (e.g. how to run autopkgtest against local debs).

bzr-debuild

• Install bzr-debuild (you already have it installed, right?) and create ~/.bazaar/builddeb.conf with those contents:

  [BUILDDEB]
  builder=sbuild -d zesty --arch=amd64 -c zesty-amd64-shm -j9
  quick-builder=debuild -j9

  This way calling `$ bzr bd` in a bzr tree will build in your shiny in-memory sbuild chroot. You can still call `$ bzr bd --quick` for the usual build with debuild.

  Read bzr-builddeb docs for more info.

CCache

• Read Debian's "Using ccache with sbuild" wiki entry.

EatMyData

• Recent versions of mk-sbuild support a "--eatmydata" flag, which enables eatmydata (discarding blocking fsync calls) in that chroot.

ARM

ARM should basically Just Work, e.g. by creating an armhf chroot:

• $ mk-sbuild --target armhf zesty

Now, everything works just as if you were building i386 packages on amd64. Just use "armhf" as the host architecture. See CrossBuilding for more info.

Pre-populated chroots

A lot of time is spent on apt and dpkg installing build dependencies. You can just create a custom chroot, e.g. for zesty:

• $ mk-sbuild --target armhf --name foo-amd64-armhf zesty

Start a root session that makes persistent changes:

• $ schroot --chroot source:foo-amd64-armhf --user root 

Install all build dependencies for project foo:

• (foo-amd64-armhf)$ apt build-dep --host-architecture=armhf foo

Then, when building with that chroot, you can skip apt updates, and it will go straight into building:

• $ sbuild --chroot foo-amd64-armhf --no-apt-update --no-apt-distupgrade 

Dirty builds

If you want to save even more time, you can use "dirty" builds. Prepare a pre-populated chroot as above, and either build packages:

•  $ schroot --chroot foo-amd64-armhf
   (foo-amd64-armhf)$ dpkg-buildpackage -aarmhf -nc 

Or just configure and build yourself:

•  $ schroot --chroot foo-amd64-armhf
   (foo-amd64-armhf)$ export `dpkg-architecture -aarmhf`
   (foo-amd64-armhf)$ cmake
   (foo-amd64-armhf)$ make 

Juju

• TBD: maybe it'd be possible to deploy this locally with Juju and LXD/LXC?

CategoryPackaging