Martin Pool's blog

Bazaar gedit integration

Javier Derderian is working on , the GNOME standard text editor, so that you can very easily record changes, push them to a server, and so on. Bazaar's model that a branch is just a directory with extra metadata fits pretty well here. He just made another exciting release (or should that be "excited"? :-)

Avoiding "not permitted to upload" errors from PPAs

Morten asked today on irc about an error I have hit before myself: you go to upload your new package to a PPA, and get an odd message of Not permitted to upload to the RELEASE pocket in a series in the 'CURRENT' state.

What this means is that your upload was trying to go into the Ubuntu distribution, rather than into a PPA, and you're not authorized to put it there. The underlying reason is that the command line for dput, the tool for uploading source packages, is

dput [options] [host] package.changes ...

It's easy to forget the optional host parameter and if it's omitted it uploads into the Ubuntu archive.

There is a pretty easy (if crude) way to disable this behaviour, by adding these lines to your ~/.dput.cf:

[DEFAULT]
default_host_main = notspecified

[notspecified]
fqdn = SPECIFY.A.PPA.NAME

Bundle Buggy, for list-based code review

My favourite thing about Bazaar is actually a tool built on top of it, Bundle Buggy, a tool for managing contributions and reviews.

When you send a patch or a Bazaar bundle to the developer mailing list, Bundle Buggy gets a copy of it, sees the proposed merge, creates a web page about it, linked back to the mail thread and to a bug number if there is one.

By catching all patches to the list it makes sure that things are not accidentally dropped. The important thing though is the getting-things-done workflow and the integration with Bazaar. Seeing a long list of mails can make it hard to decide where to begin, which is an invitation to procrastinate. BB tries to sort out patches which are ready to merge, which have been reviewed by others, which you ought to review yourself, which need other work and other states. It knows about our quasi-voting process, where we try to get at least one core contributor to review all patches, and you can do the reviews entirely over mail.

BundleBuggy is (just about?) all Aaron Bentley's work, and credits Jeremy Kerr's Patchwork tool for inspiration.

We could do a lot more here, I'm sure: a nicer review interface than just showing the diff, perhaps automatically firing off a merge if the patch is approved, telling the bug and its subscribers if we believe a fix has been merged.

Tips for Launchpad PPAs

Launchpad has relatively new (well, a few months old) feature called Personal Package Archives.

PPAs have been taken up quite a lot by people making variations of packages in Ubuntu. One common situation is that developers will put pacakages with a proposed fix for a bug up into their PPA to let people try it out and confirm the bug is fixed.

I have been using them this week to make packages of Bazaar.

It's a pretty nice service; with probably less hassle than setting it up ourselves we have a nice apt archive that can be uploaded to by anyone in our team, and that rebuilds across several supported architectures. I hope with a bit more scripting we will be building these automatically on each release.

One piece of advice is to create a team specifically for packaging. PPAs as the name implies are owned by people or teams, so the URLs people put into their apt sources are bound to that team. So if, maybe later on, you want to allow more people, or indeed fewer people, to upload to the archive that can most easily be accomplished by making a specific team for it in the first place. Team memberships is transitive so you can always add your whole project's team to the PPA team. I put ours under ~bzr but it would have been better to use ~bzr-ppa. Or indeed, to allow for several different types of archive, maybe ~bzr-ppa-stable.

(More later, it's after 6 and enough for a week.)

About Launchpad betas

Launchpad.net has a fairly complex system by which new code is gradually exposed to a larger audience. This is managed through Bazaar branches, and automatic merging between them.

As this has evolved there has been a little confusion between launchpad.net, edge, beta, staging and so on. So to explain this there is now a help page about Launchpad beta testing.

Version control that doesn't make your eyes bleed

Nice positive comments on the benefits of Bazaar's simple and usable interface from the Java-GNOME maintainer.

zsh xterm mouse support

I just discovered a nice extension for zsh: mouse.zsh, which lets you click within the input line to move the cursor, copy or paste.

lvm snapshots in ubuntu feisty

Ubuntu 7.04 has some pretty good support for LVM, the Linux Logical Volume Manager. You can set lvm on software raid from the installer - it's not quite obvious, but I guess given the variety of ways people might want to configure raid the configuration needs to reflect that complexity.

I thought I would be clever and take a snapshot of my root filesystem so I could recover if something went wrong. Bad move: the machine becomes unbootable.

It turns out that the dm_snapshot module that handles snapshots is not loaded by default, and without it you can't access the logical volume on which the snapshot is based. So there's no root filesystem block device, therefore no root filesystem and you're dumped into initramfs. (It makes some sense that you can't write without the copy-on-write code being loaded.)

lvm lvs shows this lv having the attributes -wi-do, where d means mapped (d)evice present without tables, a phrase that doesn't seem to be explained anywhere else on the web before today.

So the fix seems to be: from the initramfs shell, run lvm and lvremove the snapshot so you can boot again, and come into regular single user mode. Then add dm_snapshot to /etc/initramfs-tools/modules, and then update-initramfs -u and reboot, and then it should be ok.

Those not used to initramfs might be surprised that putting it into /etc/modules is not enough, that file isn't read (I think) until after the root filesystem is mounted.

(update) However, sadly this is not enough: bug 113713 means that snapshots on lvm on md don't work on the machine I tried it on.

Loggerhead: new bzr branch browser

Robey wrote a new and better web bzr branch browser, Loggerhead.

Launchpad as a directory for bzr

Launchpad is the Ubuntu bug tracker, and more than that. It handles bugs, translations, support requests and so on, and can relate them to the Ubuntu packages and the original upstream source. jamesh writes of his recent work to use Launchpad as a directory for Bazaar branches.

Also, the Jokosher audio editor is switching to the Launchpad bug tracker, and they have some interesting comments.

Introduction to dvcs and Bazaar for students

Presentation by Matthieu Moy for an introduction to version control for students, using Bazaar.

"Dogs and cats living together..."

The UNIX 03 Server specification requires Java support...

(OK, so Java is not really such a bad thing, but it doesn't seem like part of Unix as we know it.)

Insert a file as an email quotation in vim

In vim, to insert a file and turn it into an email quotation:

:r filename
:'[,']s/^/>

(The [ and ] markers are set to the region changed by the previous command, which is the entire inserted block.)

The Rise and Fall of CORBA

ACM Queue has an interesting article by Michi Henning, The Rise and Fall of CORBA (there's a lot we can learn from CORBA's mistakes.) CORBA was very big at DSTC when I was at the University of Queensland, to which it was attached. (Heh, my fingers always want to write 'distcc' rather than 'dstc' — in fact they just did it again.)

I once used their CORBA Notification package, which seemed like a bit of an abstraction inversion: asynchronous publish & subscribe notifications built on top of synchronous rpc built on top of streamed sockets.

A democratic process such as the OMG's is uniquely ill-suited for creating good software. Despite the known procedural problems, however, the industry prefers to rely on large consortia to produce technology. Web services, the current silver bullet of middleware, uses a process much like the OMG's and, by many accounts, also suffers from infighting, fragmentation, lack of architectural coherence, design by committee, and feature bloat. It seems inevitable that Web services will enact a history quite similar to CORBA's.

His new project, Ice, looks pretty interesting.

Commit from bzr into Subversion

Jelmer Vernoiij has been hacking on Bazaar-Subversion integration and produced some sexy results.

Basically what this is getting to is being able to make a checkout from Subversion using Bazaar, use all Bazaar's distributed features (make branches, commit, diff, etc, all offline) and then eventually merge back into Subversion. And do it all over again, remembering what was already merged, and without requiring any particular setup on the svn server.

zsh glob to match broken symlinks

Peter Stephenson suggests that one can make a zsh glob (wildcard) match only broken symlinks by appending the (N-@) glob qualifier. (You must have setopt EXTENDED_GLOB or BARE_GLOB_QUAL for this to work.)

So for example this shows all the broken links in the manpage directory:

ls -lGg /usr/share/man/*/*(N-@)

Parallel pair programming

Scott writes of the intersection of test-driven development and pair programming.

Unix secrets of billionaires

One reason why I wanted to write distcc is that when you have to wait for a computer, it costs more than just the time it actually takes for the job (compile, test whatever) to run. Nobody likes to just sit blankly waiting, so I tend to do mail or irc while the job is running, and sometimes stay distracted by that for a long time after it completed.

Mark has a simple but effective tool to reduce this lost time: a little script called alarum that plays a sound, so he can run make check; alarum and come straight back when it's done or failed. On my machine I have

% cat ~/bin/alarum
ogg123 /usr/share/games/neverdata/snd/jump.ogg

bazaar-ng 0.0.7; into Mandriva Cooker; plans for next release

We released bzr 0.0.7, which has several nice improvements, particularly to merge handling.

Michael Sherer put builds of bazaar-ng into Mandriva Cooker, their unstable development system.

Aaron started on a tool to display bazaar-ng branch histories as graphs, using dot. Still early days but could be fun.

A few more people have come across from working on the Arch-based Bazaar code. Robert Collins has a great intuitive grasp of OOP patterns and test-driven development and I'm very happy to have his help. James Blackwell, who was previously the GNU Arch release manager, is working on the wiki.

In our next release we plan to change to weave-based storage, which will be much more compact and also allow smarter merges. We've had code for this for a while but not turned it on because of working on the UI and framework, but the time has come.

Ian Lance Taylor on OSI

Ian Lance Taylor wrote an interesting post on governance of OSI, the Open Source Initiative.

Several years ago I agitated strongly about the lack of any semblance of democracy or transparency in the OSI. I stopped when I realized that the OSI didn't really matter. Since then the OSI has some to matter somewhat more--e.g., sourceforge.net looks to it to ratify licenses. But it still doesn't matter very much. And it is also still completely undemocratic and only slightly more transparent (the increase in transparency is thanks entirely to Russ's efforts to increase communication from the board). These are strange, indeed nearly incomprehensible, characteristics for an organization which claims to represent the community (compare to the FSF, for example, which makes no claim to represent anybody except itself).

The OSI also completely lacks any formal mechanism for correcting errors. Thus the argument that one should judge the results makes no sense; by the time there are results, it is too late.

Personally I think the OSI should drop any claims about representing the community, and instead describe itself as a group of self-selected experts who periodically issue opinions about open source licensing-- i.e., more or less the same as any NGO. I think that would be more honest and more helpful.

I'm in love with lftp

I've recently been delighted by lftp, a multiprotocol text-mode client. One very nice feature is that you can append & to commands to run them in the background, just like in a Unix shell:

lftp> get index.html&
[0] get index.html &
lftp> jobs
[0] Done (get index.html)
        4060 bytes transferred

lftp will automatically open new connections to continue the command while you continue to do other work. It will also reopen connections if they drop or timeout, without losing any user-visible state.

I saw a cool instance of that today: one of our servers hosting the bazaar-ng website was upgraded to new hardware while I was asleep. I still had an lftp session open to maintain the web site. When I typed ls the next day I could continue working just as before, without ever needing to know the whole server had changed. Truly a remarkable program.

Two nice Linux tools

lftp: amazing text-mode client for http, ftp, sftp and others. Does job control within the client so you can start a transfer of a file and keep doing things inside that client. Automatically reconnects as needed.

file-rc: replacement for the standard init program on Debian and Ubuntu. Rather than a mess of symlinks you just have one simple /etc/runlevel.conf file which shows in one place what programs should be run in what state.

If you make a little shell script called service containing

/etc/init.d/$*

then zsh will automatically tab-complete service names.

bzr in brazil

I haven't posted here for a long time, which is bad. It's funny how many people got through phases of just not wanting or getting around to blogging.

bzr (aka bazaar-ng) is coming along pretty well. We just passed 1000 commits onto the mainline since it started self-hosting in March of this year, and we're regularly pulling in changes from contributors branches using bzr.

The emphasis has changed a lot since I started. At the time it was meant to be a research prototype to explore ideas to move back into systems based on GNU Arch. Since then, bitkeeper has exploded, and many people have started picking up the pieces, and the original schedule looks rather leisurely in retrospect.

One important consequence is that we're trying to get bzr to a finished state in its own right.

The core versioning/branching functionality is there and working: branch, push, pull, merge, status, diff, add, commit etc. Some are rather nicely polished; some less so.

I'm trying to define some specific goals for the next few months, and focus on getting them done. Near the top of the list are more compact storage and smarter mesh merging.

One outcome of the research side is modules to do the fundamental vc algorithms of 3-way merge and weave in pure Python. A few people have suggested that merge3 in particular might usefully be contributed back into the Python standard library; after a bit more maturation I'll look into that. These have practical use in allowing it to be installed and run without any extra dependencies like GNU diff; this is moderately important on windows.

I've been in Brazil for the last couple of weeks with other folks from Canonical.com, mostly doing architecture-ish work on Launchpad, our application set for building and supporting a free software distribution. Not so much bzr hacking for now.

canonical-praia-vemilhia

Perhaps the most immediately interesting of these is Rosetta, a web-based software translation system that already used by a thousand translators. (If you consider the pool of people who are fluent in at least two languages and work on translating free numbers then a thousand is a pretty large number.)

There are some other good concepts in there — a bug tracker that really deeply understands the way code (and bugs) spread across different free software distributions and packages. They're going to need some UI love before it's easy to understand what's happening inside, but I think that will happen.

The other very cool thing is getting to spend time with Mark and other bright people here.

A flying office with incredible views

There's a Lufthansa ad on page 37 of this week's Economist advertising their new in-flight broadband service. The person in the picture seems to be running GNOME on his iBook. (It's a bit blurry.) Way to go!

Andrew Morton's keynote

Andrew's talk at lca was interesting, as ever. (I think he reads from a prepared text, which makes it a bit dry, but the text was good.)

One point is that every person who posts a patch to an open source project represents a person who not only has a problem, but cared enough to spend time attempting a fix. Andrew says every patch deserves at least some kind of response, and perhaps to be merged even if it's not the perfect fix. He takes the opposite position to Linus, who says his job is mostly to say no.

Development without a shared CVS server — whether by patches-over-mail or a distributed VCS — runs the risk of being too tough on people who want to put fixes in. I think I've made that mistake in the past.

Impatience

From a history of Windows NT:

The first two weeks of development were fairly uneventful, with the NT team using Microsoft Word to create the original design documentation... Finally, it was time to start writing some code.

(I wish I'd seen this line a couple of days ago and could have quoted it in my bzr talk; I spent at least six full weeks writing and reviwing design documentation for that. :-)

Bugzilla and Bazaar-NG

I was talking to Luis Villa at LCA and had an idea of an interesting application for the kind of distributed version control enabled by bzr.

Essentially every Bugzilla installation is a fork of the project: people install it once, tend to customize it to fit their own use, and only intermittently install updates from the maintainers. In a previous job, I was assigned the task of removing all references to "bugs" from the installation, on the grounds that our product only had "issues", not "bugs". I understand Bugzilla is now getting to be more parameterized but there is still some customization and in any case the customization needs to be merged with the new upstream defaults.

If bugzilla was distributed with a small amount of bzr metadata it might be much easier for people to either integrate their changes into new releases, or to submit improvements for upstream integration.

I stepped in to do a talk on bzr when another speaker dropped out. It was received pretty well, and sparked a lot of interesting conversations.

Patch queues in bzr

The response to bazaar-ng so far has been very positive. Many people are contributing bugs to fix platform problems or to add features they urgently want. It's a very pleasant experience.

One consequence is that, as for many other projects, there are patches submitted that I haven't tested and merged yet. One thing to do would be to put them in an issue tracking system of some kind, but not every project wants that. So I was thinking about what bzr can do to help within this source management problem.

Very shortly we plan to allow submission of changesets by email, as regular diffs with extra annotations. When those come in, they can be stored in a directory on my laptop or on the webserver. Because the changesets have universally unique IDs and descriptive metadata we can then ask interesting questions about the patches:

This queue could be filled by a robot that looks for patches on a mailing list, or in the public trees of past contributors. Although I call it a queue it's not necessarily a FIFO queue; the integrator (me) can pull things out in whatever order. As well as the branches that are normally captured by scm systems, bazaar-ng also helps you manage a bag of patches that don't necessarily fit together yet.

In general: human integrators have a valuable role but a tool can help make them more productive.

You can imagine extending this towards something like Rusty's trivial patch monkey, or perhaps towards filtering out security critical patches as candidates for merging into a stable series.

No more bitkeeper for kernel developers

KernelTrap, Slashdot.

The KernelTrap story seems to pretty much just print a Bitmover press release. I suppose the other side(s) of the story will come out soon.

Much of what I would say about it is already said by other people in the kerneltrap or slashdot comments. In particular This comment is spot on:

If BitMover stated up front that all licenses would be withdrawn from all Linux developers in the event that any single Linux developer tried to reverse engineer BitKeeper, then Linus was a total idiot for agreeing to that license.

If BitMover did not state those conditions up front, then they are being evil and manipulative in yanking licenses from unrelated parties in a fit of pique over what one person is doing in his own time.

To head off any FUD at the pass: there is copious documentation of where the bazaar-ng design ideas came from. Almost all are from places other than bitkeeper; where bitkeeper has had an influence it is through things people have said about it e.g. in the kernel BK usage docs.

I have heard from some kernel maintainers that even before this they weren't using bk except to sync from Linus.

I'd really like bazaar-ng to be a great kernel development system, and I think the performance will be good, but it's going to take a while to get enough features in place for even early adopters, maybe 2-3 months.

I suspect Larry understands that if he cut them off in 2006, no one would care so much, because the open systems would be good enough. Perhaps that's why it's been done now.

It's a great demonstration of the risks of putting critical data in a system whose licence can be revoked at any time.

I was rather amused by Larry's notion that it's unacceptable for a platform vendor to compete with their application vendors. Microsoft or Apple would never do that, oh no.

The fact is that any successful product is going to attract competition. It's a predictable outcome of a free market, and not any kind of moral failing of the open source community. There is no point whining about it.

If you've been left out in the cold by Bitkeeper I encourage you to check out bazaar-ng.

file renames in bazaar-ng

Renames are completely(?) working in bzr now, after the last couple of days work. There are two commands: move moves one or more files into a different directory, and rename renames a file (and optionally also moves it.)

(I think this is one case where the unix unification of both these things into mv really causes much more trouble than it's worth.)

There is also a renames command that shows which files have moved relative to the last revision.

Demo:

demo% ls -a
./  ../
demo% bzr init
demo% echo hello > one
demo% bzr rename one two
bzr: error: can't rename: old name 'one' is not versioned
demo% bzr add one
demo% bzr rename oen two
bzr: error: can't rename: old working file 'oen' does not exist
demo% bzr rename one two
one => two
demo% bzr status
A       two
demo% bzr commit -m 'add file'
demo% bzr rename two three
two => three
demo% bzr status
R       two => three
demo% bzr renames
two => three
demo% bzr commit -m 'renamed'
demo% mkdir subdir subdir/d2
demo% bzr add subdir subdir/d2
demo% bzr status
A       subdir/
A       subdir/d2/
demo% bzr move three subdir/d2/
three => subdir/d2/three
demo% cd subdir/d2
d2% bzr rename three ../four
subdir/d2/three => subdir/four
d2% bzr renames
three => subdir/four
d2% bzr status
A       subdir/
A       subdir/d2/
R       three => subdir/four
d2% cd ../../
demo% bzr status
A       subdir/
A       subdir/d2/
R       three => subdir/four

The code is not published yet; it'll be up later this week.

Archives 2008: May Apr Feb 2007: Jul May Feb Jan 2006: Dec Nov Oct Sep Aug Jul Jun Jan 2005: Sep Aug Jul Jun May Apr Mar Feb Jan 2004: Dec Nov Oct Sep Aug Jul Jun May Apr Mar Feb Jan 2003: Dec Nov Oct Sep Aug Jul Jun May