Ubuntu Wiki

This is an extract from SecurityTeam/BuildEnvironment that tries to make it as quick and easy as possible to reduce your pbuilder / PPA / distro build failure rate and improve your build & turnaround time, as well as allow you to debug the builds in an environment close to the Ubuntu builders.

This has been tested on Saucy and Trusty. If you are not using the stock Ubuntu kernel, you need to make sure your kernel supports overlayfs (the mainline ppa kernels do not).

The original wiki page deals with many more details (especially around security) that are not required on a daily basis, but make it quite cumbersome to actually follow the guide. If anything here doesn't work, or is unclear, or you want more detail, check the other page - maybe there's help on your issue.

Setting up and using sbuild

Creating the schroots

1. Install sbuild and schroot:

   $ sudo apt-get install sbuild debhelper ubuntu-dev-tools

2. Optional: Install a local package proxy, so that downloaded packages get cached between sbuild/schroot runs:

   $ sudo apt-get install apt-cacher-ng

3. Make sure you are in the 'sbuild' group:

   $ sudo adduser $USER sbuild

4. Setup for mounting $HOME and extracting the ddebs:

   • Create the directory for the mount point:

     $ mkdir -p $HOME/ubuntu/scratch

   • Append to /etc/schroot/sbuild/fstab:

     /home/<username>/ubuntu/scratch  /scratch          none  rw,bind  0  0

     If you use an encrypted home directory, your $HOME is mounted differently,

     so you will also have to add:

     /home/<username>                 /home/<username>  none  rw,bind  0  0

5. Create ~/.sbuildrc:

   # Mail address where logs are sent to (mandatory, no default!)
   $mailto = $ENV{USER};
   
   # Name to use as override in .changes files for the Maintainer: field
   # (mandatory, no default!).
   $maintainer_name='Your Name <user@ubuntu.com>';
   
   # Default distribution to build.
   $distribution = "trusty";
   # Build arch-all by default.
   $build_arch_all = 1;
   
   # When to purge the build directory afterwards; possible values are "never",
   # "successful", and "always".  "always" is the default. It can be helpful
   # to preserve failing builds for debugging purposes.  Switch these comments
   # if you want to preserve even successful builds, and then use
   # "schroot -e --all-sessions" to clean them up manually.
   $purge_build_directory = 'successful';
   $purge_session = 'successful';
   $purge_build_deps = 'successful';
   # $purge_build_directory = 'never';
   # $purge_session = 'never';
   # $purge_build_deps = 'never';
   
   # Directory for writing build logs to
   $log_dir=$ENV{HOME}."/ubuntu/logs";
   
   # don't remove this, Perl needs it:
   1;

6. Make the following directory (change if specified something different in ~/.sbuildrc):

   $ mkdir -p $HOME/ubuntu/logs

7. Create ~/.mk-sbuild.rc:

   SCHROOT_CONF_SUFFIX="source-root-users=root,sbuild,admin
   source-root-groups=root,sbuild,admin
   preserve-environment=true"
   # you might want to undo the below for stable releases, read `man mk-sbuild` for details
   SKIP_UPDATES="1"
   SKIP_PROPOSED="1"
   # if you have e.g. apt-cacher-ng around
   # DEBOOTSTRAP_PROXY=http://127.0.0.1:3142/

8. Generate a GPG keypair for sbuild to use:

   $ sbuild-update --keygen

9. Finally, create the schroots:

   $ sg sbuild # only needed if you were added to the sbuild group during this session
   $ mk-sbuild saucy

   See man mk-sbuild for details. Note that this will automatically use the host's apt-cacher-ng for the generated schroot if you installed it above.

10. Optional: It can often be useful to perform builds in /dev/shm, which is a tmpfs directory in RAM. While there are limitations on what you can compile and the number of concurrent builds that can be performed (based on how much memory your build machine has), building in /dev/shm can lead to much faster build times and reduce disk I/O. A way to create a 'shared' source schroot like so:

    $ sudo sh -c "sed -e 's#]#-shm]#' \
      -e 's#^\(directory=.*\)#\1\nunion-overlay-directory=/dev/shm/schroot/overlay#' \
      /etc/schroot/chroot.d/* > /etc/schroot/chroot.d/shm-overlays.conf"
    $ schroot --list

    Please note that the directory you specify for 'union-overlay-directory' must exist before using the shm chroot (can add an entry to /etc/rc.local):
    

    $ mkdir -p /dev/shm/schroot/overlay/ 

    It might also be useful to remount /dev/shm with more memory than the default, which is 50% of RAM. Can adjust like so:

    $ sudo mount -o remount,size=75% /dev/shm

    On the other hand, it is possible for a tmpfs to fill up and cause problems in some fairly common cases, since builds can easily approach the order of the size of RAM. It is perfectly reasonable to skip this step and allow sbuild to use a disk-backed overlay, even though builds will be a bit slower as a result.

Deleting a schroot

1. Remove the stanza for the chroot:

   $ sudo rm /etc/schroot/chroot.d/sbuild-saucy-amd64

2. Remove the chroot from the disk:

   $ sudo rm -rf /var/lib/schroot/chroots/saucy-amd64

3. Optional Run the shm snippet above.

Using the schroot

• Using a schroot is similar to the chroot command but you specify the chroot and user you want to use rather than the directory.

  $ schroot -c saucy-amd64 -u root

  Or without root with:

  $ schroot -c saucy-amd64

  Or building via sbuild directly:

  $ apt-get source foo
  $ cd ./foo-*
  # do work...
  $ dch -i
  $ update-maintainer
  $ sbuild -d saucy-amd64
  # or from a source package
  $ bzr bd -S
  $ sbuild -d saucy-amd64 ../foo_*.dsc

Maintaining the schroots

• Can see a listing of all your chroots with:

  $ schroot -l
  chroot:saucy-amd64
  chroot:saucy-amd64-shm
  source:saucy-amd64
  source:saucy-amd64-shm
  ...

  The 'source:' chroot is the pristine chroot and you shouldn't go into it unless you want to change something for all future schroots into the chroot. Eg, if you wanted to always have 'vim' installed in your saucy-amd64 chroot, use:

  $ schroot -c source:saucy-amd64 -u root
  $ apt-get install vim
  $ exit

  Or, to upgrade the schroot (you shouldn't do that with PPAs enabled unless you have a "special" chroot for a certain PPA setup):

  $ schroot -c source:saucy-amd64 -u root
  $ apt-get update
  $ apt-get upgrade
  $ exit

  Or, more simply:

  sbuild-update -udc saucy-amd64

See also Barry's chup script for an easy way to keep your chroots up-to-date.

Expiring active schroot sessions

• Sometimes needed if schroots are left hanging around due to a crash or reboot. You can expire all active schroot sessions with:

  $ schroot -e --all-sessions

Other hints, tips, and tricks

Mount your home dir

• To mount your home directory in the chroot, so that you can use files from there, add this entry to /etc/schroot/sbuild/fstab:

  /home/<username>                 /home/<username>  none  rw,bind  0  0

  Note: this is a read-write bind mount, so changes will be written to your real home directory.

Parallel builds

• To make use of all your machine's power, you can use the usual -jN option for sbuild:

  $ sbuild -d saucy-amd64 -j4

Temporarily adding PPAs

• To add a PPA without "polluting" the chroot, you can:

• Disable gpg key checking - add /etc/apt/apt.conf.d/80unauthenticated:

  $ schroot -c source:saucy-amd64 -u root
  $ echo "APT::Get { AllowUnauthenticated "1"; };" > /etc/apt/apt.conf.d/80unauthenticated
  $ exit

• Add the PPA, but don't update:

  $ schroot -c source:saucy-amd64 -u root
  $ echo "deb http://ppa.launchpad.net/ubuntu-unity/daily-build/ubuntu saucy main " > \
    /etc/apt/sources.list.d/ubuntu-unity-daily-build-saucy.list
  $ exit

  You can also access the source chroot directly in /var/lib/schroot/chroots/saucy-amd64/.

Local packages

• You sometimes need newer packages (libraries, whatnot) to build against. The Incredible BarryWarsaw did some nice things to help with that: lp:~barry/+junk/repotools.

• Branch repotools to your scratch:

  $ bzr branch lp:~barry/+junk/repotools ~/ubuntu/repo

• Install sponge:

  $ sudo apt-get install moreutils

• Modify clean.sh and scan.sh to set the repository path:

  ...
  where=/home/<username>/ubuntu/repo
  ...

• Add the following snippet to your ~/.sbuildrc:

  $external_commands = {
      "pre-build-commands" => [
          ['/home/'.$ENV{USER}.'/ubuntu/repo/scan.sh'],
      ],
      "chroot-setup-commands" => [
          ['/repo/prep.sh'],
      ],
  };

• And add the following lines to /etc/schroot/sbuild/fstab:

  # Expose local apt repository to the chroot
  /home/<username>/ubuntu/repo     /repo             none  ro,bind  0  0

• Drop packages into /home/<username>/ubuntu/repo and you're good to go!

bzr-debuild

• Install bzr-debuild (you already have it installed, right?) and create ~/.bazaar/builddeb.conf with those contents:

  [BUILDDEB]
  builder=sbuild -d saucy --arch=amd64 -c saucy-amd64-shm -j9
  quick-builder=debuild -j9

  This way calling `$ bzr bd` in a bzr tree will build in your shiny in-memory sbuild chroot. You can still call `$ bzr bd --quick` for the usual build with debuild.

  Read bzr-builddeb docs for more info.

CCache

• Read Debian's "Using ccache with sbuild" wiki entry.

ARM

ARM should basically Just Work, e.g. by creating an armhf chroot:

• $ mk-sbuild --target armhf trusty

Now, everything works just as if you were building i386 packages on amd64. Just use "armhf" as the host architecture. See CrossBuilding for more info.

Pre-populated chroots

A lot of time is spent on apt and dpkg installing build dependencies. You can just create a custom chroot, e.g. for trusty:

• $ mk-sbuild --target armhf --name foo-amd64-armhf trusty

Start a root session that makes persistent changes:

• $ schroot --chroot source:foo-amd64-armhf --user root 

Install all build dependencies for project foo:

• (foo-amd64-armhf)$ apt-get build-dep --host-architecture=armhf foo

Then, when building with that chroot, you can skip apt updates, and it will go straight into building:

• $ sbuild --chroot foo-amd64-armhf --no-apt-update --no-apt-distupgrade 

Dirty builds

If you want to save even more time, you can use "dirty" builds. Prepare a pre-populated chroot as above, and either build packages:

•  $ schroot --chroot foo-amd64-armhf
   (foo-amd64-armhf)$ dpkg-buildpackage -aarmhf -nc 

Or just configure and build yourself:

•  $ schroot --chroot foo-amd64-armhf
   (foo-amd64-armhf)$ export `dpkg-architecture -aarmhf`
   (foo-amd64-armhf)$ cmake
   (foo-amd64-armhf)$ make 

Juju

• TBD: maybe it'd be possible to deploy this locally with juju and lxc?

CategoryPackaging