Ubuntu Wiki

This is an extract from SecurityTeam/BuildEnvironment that tries to make it as quick and easy as possible to reduce your pbuilder / PPA / distro build failure rate and improve your build & turnaround time, as well as allow you to debug the builds in an environment close to the Ubuntu builders.

This has only been tested on current Saucy. If you are not using the stock Ubuntu kernel, you need to make sure your kernel supports overlayfs (the mainline ppa kernels do not).

The original wiki page deals with many more details (especially around security) that are not required on a daily basis, but make it quite cumbersome to actually follow the guide. If anything here doesn't work, or is unclear, or you want more detail, check the other page - maybe there's help on your issue.

Setting up and using sbuild

Creating the schroots

1. Install sbuild and schroot:

   $ sudo apt-get install sbuild debhelper ubuntu-dev-tools

2. Make sure you are in the 'sbuild' group:

   $ sudo adduser $USER sbuild

3. Setup for mounting $HOME and extracting the ddebs:

   • Append to /etc/schroot/sbuild/fstab:

     /home/<username>/ubuntu/scratch  /scratch          none  rw,bind  0  0

     If you use an encrypted home directory, your $HOME is mounted differently,

     so you will also have to add:

     /home/<username>                 /home/<username>  none  rw,bind  0  0

4. Create ~/.sbuildrc:

   # Mail address where logs are sent to (mandatory, no default!)
   $mailto = 'username';
   
   # Name to use as override in .changes files for the Maintainer: field
   # (mandatory, no default!).
   $maintainer_name='Your Name <user@ubuntu.com>';
   
   # Default distribution to build.
   $distribution = "saucy";
   # Build arch-all by default.
   $build_arch_all = 1;
   
   # When to purge the build directory afterwards; possible values are "never",
   # "successful", and "always".  "always" is the default. It can be helpful
   # to preserve failing builds for debugging purposes.  Switch these comments
   # if you want to preserve even successful builds, and then use
   # "schroot -e --all-sessions" to clean them up manually.
   $purge_build_directory = 'successful';
   $purge_session = 'successful';
   $purge_build_deps = 'successful';
   # $purge_build_directory = 'never';
   # $purge_session = 'never';
   # $purge_build_deps = 'never';
   
   # Directory for writing build logs to
   $log_dir="/home/<username>/ubuntu/logs";
   
   # don't remove this, Perl needs it:
   1;

5. Make the following directory (change if specified something different in ~/.sbuildrc):

   $ mkdir -p $HOME/ubuntu/logs

6. Create ~/.mk-sbuild.rc:

   SCHROOT_CONF_SUFFIX="source-root-users=root,sbuild,admin
   source-root-groups=root,sbuild,admin
   preserve-environment=true"
   SKIP_UPDATES="1"
   # if you have e.g. apt-cacher-ng around
   # DEBOOTSTRAP_PROXY=http://127.0.0.1:3142/

7. Generate a GPG keypair for sbuild to use:

   $ sbuild-update --keygen

8. Finally, create the schroots:

   $ sg sbuild # only needed if you were added to the sbuild group during this session
   $ mk-sbuild saucy

   See man mk-sbuild for details.

9. Optional: It can often be useful to perform builds in /dev/shm, which is a tmpfs directory in RAM. While there are limitations on what you can compile and the number of concurrent builds that can be performed (based on how much memory your build machine has), building in /dev/shm can lead to much faster build times and reduce disk I/O. A way to create a 'shared' source schroot like so:

   $ sudo sh -c "sed -e 's#]#-shm]#' \
     -e 's#^\(directory=.*\)#\1\nunion-overlay-directory=/dev/shm/schroot/overlay#' \
     /etc/schroot/chroot.d/* > /etc/schroot/chroot.d/shm-overlays.conf"
   $ schroot --list

   Please note that the directory you specify for 'union-overlay-directory' must exist before using the shm chroot (can add an entry to /etc/rc.local). It might also be useful to remount /dev/shm with more memory than the default, which is 50% of RAM. Can adjust like so: sudo mount -o remount,size=75% /dev/shm.
   On the other hand, it is possible for a tmpfs to fill up and cause problems in some fairly common cases, since builds can easily approach the order of the size of RAM. It is perfectly reasonable to skip this step and allow sbuild to use a disk-backed overlay, even though builds will be a bit slower as a result.

Deleting a schroot

1. Remove the stanza for the chroot:

   $ rm /etc/schroot/chroot.d/sbuild-saucy-amd64

2. Remove the chroot from the disk:

   $ sudo rm -rf /var/lib/schroot/chroots/saucy-amd64

3. Optional Run the shm snippet above.

Using the schroot

• Using a schroot is similar to the chroot command but you specify the chroot and user you want to use rather than the directory.

  $ schroot -c saucy-amd64 -u root

  Or without root with:

  $ schroot -c saucy-amd64

  Or building via sbuild directly:

  $ apt-get source foo
  $ cd ./foo-*
  # do work...
  $ dch -i
  $ update-maintainer
  $ sbuild -d saucy-amd64
  # or from a source package
  $ bzr bd -S
  $ sbuild -d saucy-amd64 ../foo_*.dsc

Maintaining the schroots

• Can see a listing of all your chroots with:

  $ schroot -l
  chroot:saucy-amd64
  chroot:saucy-amd64-shm
  source:saucy-amd64
  source:saucy-amd64-shm
  ...

  The 'source:' chroot is the pristine chroot and you shouldn't go into it unless you want to change something for all future schroots into the chroot. Eg, if you wanted to always have 'vim' installed in your saucy-amd64 chroot, use:

  $ schroot -c source:saucy-amd64 -u root
  $ apt-get install vim
  $ exit

  Or, to upgrade the schroot (you shouldn't do that with PPAs enabled unless you have a "special" chroot for a certain PPA setup):

  $ schroot -c source:saucy-amd64 -u root
  $ apt-get update
  $ apt-get upgrade
  $ exit

See also Barry's chup script for an easy way to keep your chroots up-to-date.

Expiring active schroot sessions

• Sometimes needed if schroots are left hanging around due to a crash or reboot. You can expire all active schroot sessions with:

  $ schroot -e --all-sessions

Other hints, tips, and tricks

Mount your home dir

• To mount your home directory in the chroot, so that you can use files from there, add this entry to /etc/schroot/sbuild/fstab:

  /home/<username>                 /home/<username>  none  rw,bind  0  0

  Note: this is a read-write bind mount, so changes will be written to your real home directory.

Parallel builds

• To make use of all your machine's power, you can use the usual -jN option for sbuild:

  $ sbuild -d saucy-amd64 -j4

Temporarily adding PPAs

• To add a PPA without "polluting" the chroot, you can:

• Disable gpg key checking - add /etc/apt/apt.conf.d/80unauthenticated:

  $ schroot -c source:saucy-amd64 -u root
  $ echo "APT::Get { AllowUnauthenticated "1"; };" > /etc/apt/apt.conf.d/80unauthenticated
  $ exit

• Add the PPA, but don't update:

  $ schroot -c source:saucy-amd64 -u root
  $ echo "deb http://ppa.launchpad.net/ubuntu-unity/daily-build/ubuntu saucy main " > \
    /etc/apt/sources.list.d/ubuntu-unity-daily-build-saucy.list
  $ exit

  You can also access the source chroot directly in /var/lib/schroot/chroots/saucy-amd64/.

Local packages

• You sometimes need newer packages (libraries, whatnot) to build against. The Incredible BarryWarsaw did some nice things to help with that: lp:~barry/+junk/repotools.

• Branch repotools to your scratch:

  $ bzr branch lp:~barry/+junk/repotools ~/ubuntu/repo

• Install sponge:

  $ sudo apt-get install moreutils

• Modify clean.sh and scan.sh to set the repository path:

  ...
  where=/home/<username>/ubuntu/repo
  ...

• Add the following snippet to your ~/.sbuildrc:

  $external_commands = {
      "pre-build-commands" => [
          ['/home/<username>/ubuntu/repo/scan.sh'],
      ],
      "chroot-setup-commands" => [
          ['/repo/prep.sh'],
      ],
  };

• And add the following lines to /etc/schroot/sbuild/fstab:

  # Expose local apt repository to the chroot
  /home/<username>/ubuntu/repo     /repo             none  ro,bind  0  0

• Drop packages (with .dsc) into /home/<username>/ubuntu/repo and you're good to go!

bzr-debuild

• Install bzr-debuild (you already have it installed, right?) and create ~/.bazaar/builddeb.conf with those contents:

  [BUILDDEB]
  builder=sbuild -d saucy-amd64-shm -j9
  quick-builder=debuild -j9

  This way calling `$ bzr bd` in a bzr tree will build in your shiny in-memory sbuild chroot. You can still call `$ bzr bd --quick` for the usual build with debuild.

  Read bzr-builddeb docs for more info.

CCache

• Read Debian's "Using ccache with sbuild" wiki entry.

ARM

• TBD: see how to cross-build or put it all on an ARM device with some persistency.

  $ mk-sbuild --arch armhf saucy

Juju

• TBD: maybe it'd be possible to deploy this locally with juju and lxc?

CategoryPackaging