Ubuntu Open Week - Kernel Team - Ben Collins - Mon, Apr 23, 2007

TZ UTC-4

(04:01:17 PM) BenC: Hello, and welcome to the Ubuntu Open Week session for the Kernel Team. I'm Ben Collins, the Ubuntu Kernel Team Technical Lead. In this session, I'll start with some background on the Kernel Team, and then open up for some Q+A.
(04:01:43 PM) BenC: I work directly for Canonical as a member of the Ubuntu Distro Team, and have been maintaining the Ubuntu Linux Kernel for a little more than 1.5 years now.
(04:01:59 PM) BenC: When I first started, the Ubuntu Kernel Team consisted of myself, having taken it over from Fabio M. Di Nitto (fabbione).
(04:02:24 PM) BenC: Since then, we have grown to a total of 4 paid kernel developers: Myself, Kyle McMartin (kylem), Tim Gardner (rtg) and Phillip Lougher (pkl).
(04:02:54 PM) BenC: Our daily routine is very mundane. We peruse the launchpad bugs for the kernel, working to confirm and evaluate the bug reports. On good days, we try to fix these bugs, and in between we do a little development on the kernel :)
(04:03:23 PM) BenC: Most of us have outside projects we work on. Kyle handles parisc patches, and Phillip is the SquashFS maintainer. I used to maintain the Linux IEEE-1394 stack, but that has since been passed on to someone else for lack of time on my part. For many years I've done trivial work on the UltraSPARC kernel port.
(04:04:32 PM) BenC: Since the team in its current state is very new, we haven't seen a lot of what we can do. We do plan to open up our roadmap a lot for the gutsy development cycle, and produce some new and exciting things. I'm looking forward to seeing what the great group of developers can do.
(04:05:09 PM) BenC: Some things are already coming open, like the newly developed gutsy kernel build system, which should allow for the community to better interact with us, and allow us to provide a better quality kernel for release.
(04:05:49 PM) BenC: And now I would like to open the Q+A portion of this session...

<Riddell> QUESTION: what is the gutsy kernel build system?

• The build system in question is the bulk of the debian/* directory in our source tree. Before gutsy, we had been using kernel-package (make-kpkg) to create the kernel deb's and handle the majority of the build process. This has proven to be very clunky, and hasn't adapted well for us. So for gutsy, we developed a new build system that can be see in the debian/* directory of the ubuntu-gutsy.git repo (kernel.org). It allows us to better customize the build, and allows for easier add-ons by external builds (people who want to make custom kernels available). Plus it cuts down the build time A LOT

<tsmithe> QUESTION: How should an eager user learn about kernel development?

• I knew this question would come up, but I failed to prepare for it. However, the best way to learn about kernel development is dive in. Most people find something that plucks they're nerves and get into the source and try to fix it. This can be anything, even as trivial as msg printed in kern.log. Just getting familiar with the source tree layout is a tremendous way to start working with the kernel.

<Riddell> QUESTION: why does the package name include the upstream version? Does this mean you loose bugs when there's a new upstream version?

• The package name has always included the version. Back when we had bugzilla, there was a linux meta target for all kernel bugs. When the big lp move occurred, it was not possible to do this. (lp == launchpad)

<joebaker> QUESTION: Realtime patches? I see some places where realtime could be useful. Do you see Ubuntu ever releasing a default kernel with the Realtime enabled?

• However, I kept the staus quo because the kernel bug list can easily get full of stale bugs, and starting with a clean slate every release is very helpful. Well, yes, I can see that happening when those patches are in upstream kernel. See the KernelTeam wiki, specifically the FAQ about why the -rt patches are not in our kernel.

<erstazi> QUESTION: how many volunteers are helping with the development of the kernel?

• We have about 3 very active volunteers. Would be nice for that to grow.

<unimatrix9> QUESTION : what are the most urgent topics that needs to be adressed by your team, that might need help at the moment?

• Our biggest issue is hw testing. We obviously can't have every bit of hw, and even if people file a bug on a driver/hw combo that doesn't work, it's very hard for us to remotely fix it on someone elses system. We definitely need more capable people testing hardware. Two biggest areas where we need testing, storage (IDE/SATA) and wireless. We always seem to hit problems with them on every release, and we are chasing our tails to fix the problems.

<rohan> QUESTION: there are many regressions from edgy kernel to feisty kernel, related to sound issues. is this related to kernel team ? if so, will the new build system help in anyway to test these type of regressions ?

• Part 1) Yes, it is kernel related. Part 2) No, the build system does not do run time testing. See answer to prior question on testing.

<Demon012> QUESTION: Can you recommend any reading material on developing the Linux Kernel and Linux Drivers?

• There is a Linux Device Drivers book from Oreilly, which is what I read when I first started with the kernel. It's a great intro to the kernel APIs. There's also the linux-doc meta package in Ubuntu, which has html of all the in-source documentation

<zul> QUESTION: what is the new features for gutsy? er...besides the build system. Anything you want to see on the roadmap?

• One area I'd like to concentrate on is the validation and testing infrastructure in the new build system. A lot of regressions from release-to-release (or even from one upload to the next) can be caught at build time. There are some ideas on the KernelTeam/Roadmap wiki

<joebaker> QUESTION: Ubuntu-Studio requires Low latency. Can you speak to this? I'd like the sound to be production quality. We use Linux thin clients and I'm pondering whether we may be able to some day add VOIP support at the thin client. What suffers when you use a low latency option?

• Low latency is a hot topic for us. However, we have an issue where the stock kernel doesn't support the needed features, and patches to do this are very invasive. But, the new build system is going to allow for kernels like this to be built in-tree and uploaded with the rest of the kernel images. We will be VERY picky on which ones get included for this feature. Lowlatency is most likely one, Xen is another candidate.

<danohuiginn> QUESTION: how is the kernel team using the information in the ubuntu hardware database?

• In theory we want to use it to gather information on use coverage for certain types of hw, but in reality we haven't made much use of the information at this point. We done some percentage checks on different things, but nothing of note.

<ompaul> Question: in the next eighteen months to two years do you think that wireless will start seeing more free drivers and less dependancy on blobs to get them working?

• I think we'll see more free drivers, but I don't think we'll ever get rid of things like firmware. Certain vendors (*cough* Intel) are leading the way by example in releasing open source drivers for wireless. We will even see a daemonless ipw3945 driver during gutsy cycle.

<Riddell> QUESTION: are there plans to backport more drivers to dapper? people occationally complain that it doesn't support hardware on new servers

• At UDS we plan to do a hw rev discussion for dapper, to create a set of updated drivers for it. Until recently, we haven't had the man power to do it.

<joebaker> QUESTION: HW testing. How about having a 60 MB boot CD that has a battery of Kernel tests that you can have people try as a live CD. Use the hardware database to email people who have specific hardware that needs the testing.

• That's an excellent suggestion. Did I also hear you volunteer to make it? Seriously, that's just the kinds of ideas we need to get implemented.

<ubuntu_demon> QUESTION : how's the future looking for suspend/hibernate on laptops ?

• The future always looks bright, it's finding the right path to get to get to the light that's hard....Suspend/Hibernate has been famously plagued by bad BIOS implementations, and broken drivers. There are efforts upstream to get rid for the latter, but the former will always be out of our control. People can do things like report problems from the linux firmware kit to their vendors to make them aware.

<el_isma> QUESTION: Does the kernel team need more help triaging the kernel related bugs in launchpad?

• YES! Triaging bugs is very time consuming. There's info on how you can help at various wiki pages on wiki.ubuntu.com, and specifically at https://wiki.ubuntu.com/KernelTeamBugPolicies. Note that kernel bug policy has some slight variations from most bugs in Ubuntu...it's a very specialized area. We don't turn down any help, but knowing how we operate helps us all.

<Loic> QUESTION: Since you can't have all the hardware in the world (even though you might have some nice rigs) wouldn't it be possible to have a set of normal users, one or two for each main hw, that would be responsible to report any regression and help testing fixes (maybe opening a page for each hw like there is a page for each bug)? <Loic> COMMENT : By main hw I'm also thinking scanners/printers etc.

• One or two isn't a problem...we get that just from ubuntu core-devs (collectively we cover every major bit of hw). It's the corner cases we have issues with, and finding people that have those corner cases, or even knowing what they are, is near impossible until the bug occurs. Yes, peripheral hw is very hard for us to test...something like that for those types of hw would be very helpful. At the same time, those sorts of things aren't as high of a priority as say wireless cards, or SATA controllers.

<nansub0111> QUESTION: Ben, you mentioned that the best way to get started with Kernel development is by diving in or reading the device drivers book. What is the best way to setup a development environment for kernel hacking. Specifically if one is interested in modifying the rt2500 wireless drivers. Thanks!

• For general hacking, it's easiest to just install: make, gcc and linux-headers-generic. Then you can build something like that modularly and test it easily on your stock system. That reduces the time and variance which would occur with building an entire kernel.

<rohan> QUESTION: how about merging suspend2 in the kernel ? it's more mature, and works beautifully out the box, as compared to swsusp

• Knew that was going to come up sooner or later. Suspend2 is another incredibly invasive patch that touches way too many stock source files, and makes our headaches worse, even compared to the benefit it may give you

  --> https://wiki.ubuntu.com/KernelPatches For patches like this, the push should be made upstream as opposed to the distro level...

<h4wk0> QUESTION: Why does the final version of feisty not support my ati card (however herd 2 did) - Its an ati x1400 More at: http://ubuntuforums.org/showthread.php?t=399913

• That sounds incredibly like an Xorg question as opposed to a kernel question.

  <PriceChild> ooops sorry

  not your fault...Xorg is the kernel nemesis

  <PriceChild> hehe i just clicked the link and saw that the fix required blacklisting a few modules...

(04:53:23 PM) BenC: If anyone has specific questions regarding bug reports or other issues, the kernel team is always active in #ubuntu-kernel
(04:53:46 PM) BenC: well, not _always_, but when we are, that's the place :)
(04:54:36 PM) BenC: If you are interested in the kernel team and what we are doing, I strongly recommend subscribing to the kernel-team AT lists DOT ubuntu DOT com mailing list and checking out https://wiki.ubuntu.com/KernelTeam

<torkiano> QUESTION: if a bug affect a driver what package is afected? kernel(linux-source-2.6.20) or the source of driver (rt2x00)

• I'm quite surprised the topic of kernel version for gutsy hasn't come up yet. In that case, since the source code for that driver came from linux-source-2.6.20, that's where the bug can be fixed. It's also very helpful to report the bug upstream, since they are the ones most familiar with the hardware. e.g. the rt2x00 project.

<Riddell> QUESTION: what version of linux will be used in gusty?

• Ah, someone saved the day. It's %99.99 sure to be 2.6.22.

<h4wk0> QUESTION: Can we see all the ubuntu team in dresses please?

• Prepayment for cross-dressing requests can be sent to my paypal account.

<Riddell> QUESTION: how far branched is the ubuntu linux from linus's mainline? and how much work needs to be done for each new upstream release?

• Oooh...it's actually branched lot more than I would like...we usually end up pulling in some other git tree like libata-dev#upstream for feisty. But the Ubuntu patches are very limited. It's usually not a lot of work since most "new" things we pulled in are already upstream by the time the next cycle starts.

(05:00:25 PM) BenC: Looks like we're out of time. Thanks everyone for joining, and enjoy the rest of the week!