Ubuntu Open Week - Packaging 101 - Jordan Mantha - Mon, Apr 23, 2007

see also Wednesday Session.

TZ UTC-4

(12:03:04 PM) LaserJock: Hello everyone, my name is Jordan Mantha and I'm a PhD Physical Chemistry student and a voluteer developer in Ubuntu.
(12:03:22 PM) LaserJock: Today we are going take a brief journey through packaging, which is about the most important aspect of a Linux distro. Ubuntu's sucess in the long run is largely dependent on the quality, updatedness, and availability of software in it's repositories. To get there requires lots of help from people like you and I in the community who want to give back to the distro that has given us much.
(12:03:49 PM) LaserJock: I'd like to start by saying that packaging can be frustrating at times, and a little confusing, but give it some effort and patience and you'll be teaching a couple hundred people Packaging 101 :-)
(12:04:18 PM) LaserJock: You don't need to be a programmer or super cool rockstar like Jono. ;-)
(12:04:58 PM) LaserJock: In the time that I have I really can't give a good in-depth tutorial on packaging. What I'd like to do instead is give you a overview of packaging for Ubuntu, point you to some further reading, and get you fired up about joining in and contributing to Ubuntu packages. And of course answer a bunch of questions.
(12:06:27 PM) LaserJock: One of the first things to be aware of in Ubuntu packaging is that we deal primarily with source packages
(12:06:58 PM) LaserJock: source packages are what gets uploaded to the Ubuntu build machines
(12:07:19 PM) LaserJock: a diff to a source package is what somebody uploading a fix for you will want to see
(12:08:03 PM) LaserJock: so the question then is "what the heck is a source package and where do I get it?"
(12:08:17 PM) LaserJock: well a source package is really made up of 2-3 files
(12:08:53 PM) LaserJock: in what's call a non-native package there are 3 files
(12:09:33 PM) LaserJock: 1) a .dsc file that is a description of the source package and also has checksums of the other source package files
(12:10:00 PM) LaserJock: 2) a .orig.tar.gz file which is the tarball that you get from the upstream authors
(12:10:11 PM) LaserJock: we want to keep the .orig.tar.gz as pure as possible
(12:10:15 PM) LaserJock: so we have:
(12:10:42 PM) LaserJock: 3) .tar.gz file which is a diff to the .orig.tar.gz which holds all the modifications to the source
(12:10:57 PM) LaserJock: including the packaging specific files and any patches we may need
(12:11:49 PM) LaserJock: Adri2000: ah thanks, I was thinking ahead of myself
(12:12:00 PM) LaserJock: 3) is a diff.gz file
(12:12:13 PM) LaserJock: a native package is one that is specific to the distro
(12:12:35 PM) LaserJock: and doesn't really exist outside of it
(12:12:48 PM) LaserJock: in that case preservation of a original tarball isn't needed
(12:13:06 PM) LaserJock: so there are just 2 files, the .dsc and a tar.gz file that holds all the source
(12:13:17 PM) LaserJock: ok, any questions so far?
(12:13:39 PM) LaserJock:  LaserJock: I filed a bug against librsvg2-2 a good while ago. the folks at gnome even say in the place librsvg lives in the ftp archives "DO NOT USE 2.16.0" yet its still not updated to 2.16.1... what gives? do some packages slip thru the update net?
(12:14:09 PM) LaserJock: well, things can certainly fall through the cracks at times
(12:14:25 PM) LaserJock: we rely (especially in Universe) on Debian quite a bit.
(12:15:03 PM) LaserJock: and given there are roughly 50-80 developers for ~15,000+ packages it can be difficult to keep up

QUESTION: When we create a patch how can we get the result into the current Ubuntu Release's Repositories or is that not possible?

• If you mean a stable release (like Edgy of Feisty) then that requires a Stable Release Update (SRU)

QUESTION: when in the release cycle is the best time to get a new package in?

• After the repos have opened and before the Feature Freeze(Main) or NewPackages Freeze (Universe).

QUESTION: could you give a brief description of what to do to get a package from say, feisty, into the backports repository for dapper?

• You'll want to file a bug with the backports team.

  <PriceChild> You could also ask for help with filing the bug in the backports forum on ubuntuforums.org where jdong should help

QUESTION: If I wanted to make a change to a package, I have hear the preferred format is "debdiff". What is this?

• a debdiff is a somewhat special diff that shows the difference between source packages. The tool used is niftily called debdiff.

QUESTION: If a package I have installed in one version has broken dependencies after an upgrade, I report a bug in Launchpad, then how do I go about getting it fixed or submitting a fix?

• You'll want to grab the source package (apt-get source <packagename>) and fix the dependencies in debian/control. Submit a debdiff of those changes to the bug report.

QUESTION: if I fix a bug in a package, when should I do a straight debdiff and when should I instead do a proper patch using some patch system?

• Good question. If you feel your patch is complete and won't take any additional work by a dev, do it as a debdiff. You'll get credit and experience. If you are just providing a patch to one file and it may need more work or you don't want to go to all the trouble of preparing a new source package and debdiff'ing then just submit a patch.

QUESTION: Is there a possibility of getting a package in for a particular Ubuntu Version well after release? Say for two cases. A "new" addition or an update to an already existing package.

• Once a release is "released" it is frozen. We don't include new packages (unless by rare exception to -updates repos). Updates are done several ways.
  • -updates is a repo for important updates -security is for security fixes -backports is for getting newer versions in (from the current development release)

QUESTION : If a package (lower version than upstream) is found to have a serious security vulnerability, do you prefer to upgrade to the new package version, or do you apply hotfixes to the old package?

• If it's for a stable release we backport the fix to the existing version.

QUESTION: Is there anything in the package files that give a URL to where the package's source was downloaded from? If, for example, I want to know where you get the source for desktop-effects from, is there anything in the .diff.gz or .dsc files that tell me this?

• The debian/copyright files should say where the source was downloaded from

QUESTION: are there any licenses that we should be wary of when thinking of packaging a piece of software? I am considering packaging software licensed under the mozilla public license.

• Well, the best advice I can give is to look at the Debian Free Software Guidelines (DFSG) and the debian-legal mailing list. If it's "free" enough for Debian it'll be free enough for Ubuntu.

QUESTION: How are end users assured that the packages they're installing in Ubuntu are authentic/untampered from upstream versions (especially if a developer becomes victim to a hacker who changes code inside a package they're making... are there safeguards in place?)

• Yes, there are safeguards in place. The first is limiting who can upload a package. Only people in the ubuntu-dev team on launchpad can upload to Ubuntu. That's roughly 80 people broken up into MOTUs (Universe/Multiverse devs) and core-devs (Main and Restricted) to become either of those you need to have a gpg encryption key that is "signed" by a Debian or Ubuntu developer or somebody in the "strong set" in order to sign somebodies key you must physically meet them and check their ID know, that is all to prove you are who you say you are to prevent hackers devs keep very careful track of their gpg key and have revocation certificates that if their gpg was for some reason compromised they can revoke it and render it useless.

<neuro_> QUESTION: How different is packaging for Ubuntu from Debian?

• It depends on the package You can spot a package Ubuntu has messed around with by a "ubuntu" in the version. Roughly 4,000 packages in Main and 4,000 packages in Universe. In terms of actual packaging, we really don't differ. We do things like have different python or gcc versions, but as far as actually 'how to package' they are equivalent.

QUESTION: Suppose I have a package that's neither in ubuntu or debian repos, would I be best sending it to debian, ubuntu or both? Do all packages in debian repos go through to ubuntu?

• Excellent question. It's honestly best to send a package to Debian first, since Ubuntu syncs it's packages from Debian. More people benefit from your work if it gets to Debian. That said, you can certainly develop your package and get feedback from the Ubuntu devs and then finally submit it to Debian.

QUESTION: how time consuming is packaging for newbies? I'd really like to contribute and even have a package(s) in mind (Kadu from kadu.net), but I'm kinda affraid of taking the responsibility and then failing due to lack of time to do it properly.

• Well, that's a bit of the joy of Ubuntu. We use a team maintenance model so you aren't necessarily committing to taking care of the package for all time. On the other hand, we do like people to stick around and help out

QUESTION: Can anyone submit updates to packages or do they have to coordinate with a maintainer before submitting anything? I know nothing yet about the processes involved in contributing.

• Anyone can contribute. What happens is that if you aren't a developer you'll need an additional step of sponsorship. In both Main and Universe we have sponsorship teams set up so for instance, in Universe, you can attache a debdiff to a bug report then subscribe the ubuntu-universe-sponsors team and they'll look it over and if it's ok they'll upload it for you.

QUESTION: A previous update to the X server infamously caused problems for a large number of users. There was talk about improving the package release system to prevent broken packages from being released. Could you please update us on what changes (if any) have been made in light of the past?

• Lots. We've instituted a -proposed repo where devs upload their packages for testing before they go to -updates. We also instituted Stable Release Update policies that govern how an update gets done and tested. Suppose that i've found a bug into upstream and i've created a patch. I need to submit that to original author as well as ubuntu mantainers? What is the optimal update process? It kind of depends on how fast the development of the upstream is. It's always awesome to send stuff upstream so more people can get the fix but sometimes it takes quite some time to for the fix to trickle back down to us. So you might want to do both if it's something important. Usually you can sort of coordinate with the Ubuntu devs through a bug report

QUESTION: (sorry if this has been covered its 2:30am and im eating yesterdays lunch and half asleep) Say if there is a program we want in the ubuntu repository does it need to go past someone to say yes include it or no we dont need it (say if you make the deb package yourself not need someone else to do)

• In Universe it needs to be approved by 2 MOTUs before it is uploaded. In Main it needs to have already existed in Universe and go through a Main Inclusion Report.

QUESTION: how do you read the version numbers eg. 1.01.6~ubuntu4

• Well, in general it goes like this: <upstream version>-<debian revision>ubuntu<ubuntu revision>. So in the case of 1.2-3ubuntu4. 1.2 is the software version, 3 is the Debian revision (which is 0 if it isn't in debian), and 4 is the ubuntu revision. Sometimes you'll see things like ~ or +. Those are ways of making sure that a version is lower or higher than another version.

QUESTION : can megapackages be build , for example the full works for multi media , or webdevelopers , or what ever comes to mind...( megapackage being a set of programs to be installed in one go )

• You can use metapackages, which are basically empty packages that just depend on all the other packages that you want. ubuntu-desktop is like that.

QUESTION: Thank you for your earlier comments regarding security/assurance of packages being uploaded by developers. Could you please clarify if there are any requirements in place to only allow new updates to the repositories after 2 or more people have verified a compiled package and signed off on it?

• https://wiki.ubuntu.com/StableReleaseUpdates

QUESTION: Where should you check if a package has already been made or is already being made before you start packaging something yourself?

• You can do an apt-cache search or packages.ubuntu.com. We also have a new tag in Launchpad, "needs-packaging"

http://www.debian.org/devel/wnpp/ for Debian

QUESTION: I've read that when an upstream source already contains a debian dir, I should ask them to remove it. Why that?

• well, 2 reasons. 1) it makes maintenance harder because it turns it into a native package and every time you have a change in the packaging (new dep or something) you have to have a new upstream release. 2) many upstreams don't provide packaging that's suitable for Debian/Ubuntu.

QUESTION: REVU- tell me more about it. all i know is its for beginner package makers and thats all

• It's the MOTUs system for reviewing packages (and getting the 2 votes needed to enter Universe). It was written by MOTUs. There's a wiki page that has more info: https://wiki.ubuntu.com/MOTU/Packages/REVU

QUESTION: How do we go about getting sponsored by a member of one of the sponsorship teams? And how generally should we start helping with packaging?

• The best way is to hop on #ubuntu-motu. We love to have people willing to learn and willing to help

QUESTION: In preparing packages for Ubuntu, do you ever consult with developers of other distributions (Gentoo, Redhat, etc) and work collaboratively to solve issues? Or are package maintainers fairly independent?

• It really depends on the software. I think we generally like to be as collaborative as we can but it's fairly difficult with non-Debian based systems because our packaging is different. I think the best place to go if it's more than just an Ubuntu/Debian issue is the upstream developers themselves.

QUESTION: "It's honestly best to send a package to Debian first." How long would it usually take for a small, straight-forward package to be accepted?

• That can really vary. My first package to Debian was uploaded in 2 days. That was after I already had it go through REVU so it didn't need much additional work. The debian-mentors mailing list and IRC channel is great for that.

QUESTION: Are there any server-side statistics (not collected from clients who enable the package reporting feature) for how many times each package is downloaded in a time period?

• Not that I know of. Canonical probably would know.

(12:54:30 PM) LaserJock: ok, I'm going to end the vast Q&A there
(12:54:44 PM) LaserJock: and want to address packaging real quick :-)
(12:55:12 PM) LaserJock: obviously I don't have time to give you a tutorial now
(12:55:22 PM) LaserJock: but we have many resources to help you learn
(12:55:53 PM) LaserJock: I taught a session like this for the MOTU School: https://wiki.ubuntu.com/MOTU/School/PackagingBasics
(12:56:28 PM) LaserJock: I also maintain the Ubuntu Packaging Guide: http://doc.ubuntu.com/ubuntu/packagingguide/C/index.html
(12:56:42 PM) pbx left the room.
(12:56:47 PM) LaserJock: there are some other good resource on that MOTU School wiki page
(12:57:11 PM) LaserJock: basically, grab a source package
(12:57:34 PM) LaserJock: make sure you have source repos enabled
(12:57:40 PM) LaserJock: and apt-get source <packagename>
(12:57:45 PM) LaserJock: then poke around in debian/
(12:58:01 PM) LaserJock: the debian/ directory is where all the packaging info is
(12:58:24 PM) LaserJock: it contains the dependencies and control information (debian/control)
(12:58:35 PM) LaserJock: copyright information (debian/copyright)
(12:58:49 PM) LaserJock: changelog (debian/changelog who would have guessed0
(12:59:32 PM) LaserJock: and very importantly the "instructions" needed to build the package into a .deb (debian/rules)
(01:00:09 PM) LaserJock: harrisony: thanks for all the good questions ;-)
(01:00:19 PM) LaserJock: one of my packages is plotdrop
(01:00:23 PM) LaserJock: it's fairly easy
(01:00:56 PM) LaserJock: well, basically we didn't get anything done that I wanted to :-)
(01:01:30 PM) LaserJock: but we had fun anyway
(01:01:30 PM) LaserJock: I'd really encourage everyone that's interested in learning more to hop on #ubuntu-motu
(01:01:45 PM) LaserJock: and I'll be here again, same time, on Wednesday where I'll give a better tutorial
(01:01:59 PM) LaserJock: thanks for coming, thanks for the questions, I'm done