Martin Pool's blog

Mail me RSS Bazaar distcc Canonical is hiring! flickr

Lion

posted Sat 31 Jul 2004 in /photo | link

Kent Beck's "Economic Nonsense"

Kent Beck did a talk at OSCon about Open Source as economic nonsense.

Shows Maslow's hierarchy of needs: survival, love, ... self-actualization. But I think he misinterprets it. Maslow indicates that beyond a certain point, getting more money (food, housing, etc)

The market for software is incredibly distorted. If you do something innovative, there's a big company north of here that will just squash you. Open source is a way of saying "hey, we're just not going to play this game."

I think I agree. Possibly if Microsoft declines and we get a more competitive and functional software market then the pressure for open source would decline a bit too. Microsoft like to talk about ecosystems; Linux in part is a reaction to that.

Reckons open source goes against trend towards accountability.

But open source also empowers users, letting them both get in touch with developers and also take control of things if they don't trust the developers.

If you're not getting enough accountability for free, pay money. Get HP or Red Hat premium support.

The whole thing is a bit like the argument that it's physically impossible for a bumblebee to fly. Satisfying yourself that you proved this doesn't make bumblebees fall out of the sky. It just indicates a problem in your theory.

Why does OS seem to work? We live in a land of plenty; people are willing to sacrifice basic needs to get higher needs met.

I think the core falacy here is that everything ought to reduce to money. Money is fine. However, making the maximum money is almost nobody's goal: people spend money on random things they enjoy, rather than saving every cent. People spend time goofing off in the evening when they could be working a second job.

People spend excess energy on programming rather than art, sports, community groups, poker, etc.

Well, true. Some people prefer programming to playing soccer. It's not for Kent Beck to say how people should spend their time.

Want to see whole people working whole jobs: both financially and spiritually rewarding. Options: sponsorship, patronage, pay-per-use, licensing, complements.

Well, balance is good. But if you can't get a perfect job, then it's a reasonable tradeoff to have a slightly tedious job and to do something more fun on evenings and weekends.

Money is a way of attracting people to do jobs they wouldn't do otherwise. That's why they call it money.

posted Sat 31 Jul 2004 in /software/freedom | link

Loss of a server in Arch and Darcs

I wrote a while ago on some things I think are less than perfect in Arch.

I think the one that bugs me most is that branches are bound to a particular location, rather than being purely distributed. (I use the word branch here for the comfort of a general audience; in Arch they would strictly be called versions which I think is a bit misleading.) I want to try to explain this a bit more.

The machine hosting sourcefrog.net crashed because of hardware problems the other week and was offline for a couple of days. I wanted to work on two projects which are hosted there, librsync and distcc. Because I am a version-control gourmet, distcc is in Arch and librsync is stored in Darcs.

Because sourcefrog is quite close to where I live, I normally work directly against its repository from Arch. I would have the choice of making downstream repositories on each machine I work on, but that would introduce a lot of "noise" merges every time I moved code from those machines onto sourcefrog. Since there's only one distcc branch, and I'm the only person who commits, I'd rather just work directly to that branch.

A consequence of this is that when sourcefrog is down, I can't commit or update at all. I am stuck.

Or almost stuck. In fact, I can cheat: make an archive on my laptop and a new branch in that archive, and commit from my working copy onto that branch. When the main machine is back up, I can merge from my branch back to sourcefrog.

This is pretty neat. I don't think I could easily do it in either Subversion or CVS. With those systems, I'd probably keep hacking and just make one big commit at the end. (Which is not really such a bad thing, but not ideal.) At best, I could keep snapshots of the tree at different and commit each one by hand as a separate patch.

On the other hand, what I did is not documented, and I'm not sure it's entirely kosher. It does require a certain amount of understanding Arch internals and fiddling to get the merge to work back. It is a testament to the elegance and flexibility of the Arch design that it's possible to use it in this unintended way.

By contrast in Darcs having your server go down makes no difference at all, except that you can't publish to that particular server. Because everything is always committed locally and then pushed up the natural way of working means there's little dependency on anything but the local machine. All of this doesn't leave any major permanent record, because revision names don't depend on the machine to which they were originally committed. With the server offline you can make changes, record them, roll them back, and make branches. If the machine's going to be down for a while you can start committing to a different server, or email your changesets to someone else.

You can do this in Arch but it's more natural in Darcs.

I think at the moment I would compare them like this:

Arch has a lot of structure and metadata to let you see the history of every changeset and to organize large trees. That might be good for very large projects. It's good for small projects, though the sheer complexity can be a disincentive.

Darcs is much simpler. I think you can show someone all they need to know in ten minutes. It's naturally very distributed. I rarely or never need to wait for network traffic.

posted Fri 30 Jul 2004 in /software/vc/arch | link

Paul Graham on Hackers at OSCON

Paul Graham delivered an insightful keynote on hackers.

Great hackers also generally insist on using open source software. Not just because it's better, but because it gives them more control. Good hackers insist on control. This is part of what makes them good hackers: when something's broken, they need to fix it. You want them to feel this way about the software they're writing for you. You shouldn't be surprised when they feel the same way about the operating system.

posted Fri 30 Jul 2004 in /conf/oscon2004 | link

Let the games begin

Paul Hochman on shenanigans in the Olympic Village.

posted Fri 30 Jul 2004 in /random | link

Illogical Markup

Dave Thomas says

It seems to me that DocBook is falling in to the same trap as the rest of the XML world, confusing tedious verbosity for semantic information.

posted Fri 30 Jul 2004 in /software/xml | link

Crash-only software

No, not Windows. Rather, the idea that since serious software has to gracefully recover from abrupt shutdown, you might as well get rid of the orderly shutdown option. Occam's Razor, or OnceAndOnlyOnce says you should just pull the plug and rely on recovering properly.

USENIX paper.

(via Val Henson.)

posted Wed 28 Jul 2004 in /software/filesystems | link

CPU indicator light for GNOME

I wrote a little GNOME applet to show when the CPU is busy.

Certainly there are at least dozens of different system load indicators, but I don't really like the existing ones, for two usability reasons.

Many load indicators give you a graph showing historical load. The GNOME multiload applet is an example of this. That's fine, but I don't care very much about how busy the machine was in the past. What I mostly want to know is whether it's busy at the moment. That's useful in giving a feeling for the machine: is something unexpected happening? Has your compile finished yet?

In the current GNOME load applet (shown above), the current state of the CPU, which is the most important information, is confined to a one-pixel bar over on the right hand side. In fact, it's not even that, since the bar is only updated every ten seconds or so.

The other approach is something like gkrellm, which shows flcikering indicators of instantaneous load, as well as a graph. But I find it too distracting; load is something you want a background awareness of. It shouldn't be so intrusive as to have moving object in your field of view.

Hardware indicates what I feel is a good interface to do this: it makes a little bit of quiet noise, or lights up a small LED depending on state. Furthermore the LED is in your peripheral vision so you can be aware of it without actually looking at it. You can be easily aware of when your drive is busy and therefore get some idea of what the computer's doing without needing a massive moaning goat on your desktop.

What I would really like is a hardware CPU busy LED on my machines. Failing that, we can fake it in software:

This is meant to look like a hardware LED: when the CPU is idle, it's black. When the CPU is flat out, it lights up brightly, in between it is proportionally bright. It responds fairly promptly to changes in actual load, unlike things based on loadavg or that only update every ten seconds. I don't use loadaverage because it's not well suited to modern machines; instead this is the fraction of idle time in the last 0.5 second.

There's a little bit of hysteresis to make it look less flickery and more analog.

I think brightness is probably a better display than say a bar graph. It doesn't cause a visual sensation of motion unless it suddenly switches on or off, which is the most important event. Brightness is only qualitative, not quantitative, but that's highly appropriate for this.

I've been running it for a few days and I like it. You can see out of the corner of your eyes when a compile finishes or when a cron job starts up.

The implementation needs to read /proc/stat (via libgtop) every second, which is a bit unfortunate. It would be nice if it didn't need to wake up so often, but I think the impact is not necessarily much worse than showing a blinking cursor. It doesn't repaint unless the load changed significantly.

This was my first GNOME applet. It turns out to be pretty easy to write, except that the bonobo server registration stuff is just slightly hard to work out. Basically you need to copy the .server file into /usr/lib/bonobo/servers before the applet can get onto the panel at all. In GNOME 1.4 just running the program would make it pop up on some panel.

Probably the worst part of writing this is all the automake voodoo required to link against GNOME; that's deeply unfortunate. I think the only way to get it right is to copy and paste from existing code, and I hate to think how novice programmers would work out some of the automake error messages. I'd like to write predefined SCons builders so you could just say GnomeApplet('blink_applet', 'blink.c') and have it work.

Once it is there you can update the program by just killing the old task. The panel then asks if you want to restart it; if you say yes there then it runs the new version.

Get it and let me know what you think.

You can also get a more recent copy using darcs get http://sourcefrog.net/darcs/blink-applet/

posted Wed 28 Jul 2004 in /software/gnome | link

Stephen Kapp and Reapertech

I've been to a few interesting sessions in the last few days about infringements of copyright on free software. It reminds me of when Stephen Kapp and Reapertech ripped off my copyrighted code. Since I put that page up, several people have written to me saying he infringed copyright on other code too. I haven't checked it myself but it sounded plausible.

It's pretty sad. I guess as free software gets bigger there will be both more parasites more legal ruckus. I expect to see someone offer contingent-fee copyright suits in appropriate cases in the next few years.

posted Wed 28 Jul 2004 in /conf/oscon2004 | link

Airport signs

Seen in SFO:

NO CHANGE WITHOUT A PURCHASE

Very true.

posted Tue 27 Jul 2004 in /random | link

Airport signs, two

NO LITTERING.
DO NOT LEAVE CHILDREN UNATTENDED

posted Tue 27 Jul 2004 in /random | link

nwrite

nwrite is a pretty neat replacement for standard unix write(1) for talking to users on the same machine. It'd be nice if it could gain GNU readline support and simple spelling correction (or specifically teh-correction). I guess I should just run it under emacs.

posted Tue 27 Jul 2004 in /software/nwrite | link

UNUSED in gcc

Of course you build programs with warnings turned on? If not, write it out one hundred times...

-Wunused causes gcc to complain about unused parameters. This is often a good thing, if it makes you check twice whether you mistyped a variable name or forgot to implement something. But sometimes you really need to have a parameter which is not used, perhaps because the function needs to match a particular prototype.

gcc has a __attribute__((unused)) thing you can apply to paramters to quieten the warning. This actually means possibly unused: if you do use the parameter, gcc doesn't complain. So with unused alone, you can make errors in the opposite direction.

I use this little macro:

#ifdef UNUSED
#elif defined(__GNUC__)
# define UNUSED(x) UNUSED_ ## x __attribute__((unused))
#elif defined(__LCLINT__)
# define UNUSED(x) /*@unused@*/ x
#else
# define UNUSED(x) x
#endif

void dcc_mon_siginfo_handler(int UNUSED(whatsig))

This applies the unused attribute and also mangles the variable name so that you really can't use it. It also lets you just write UNUSED(x) regardless of compiler. Maybe there should be more branches for other compilers.

(Sometimes this won't do: you might have a parameter or variable which is only used when particular preprocessor variables are set. But in that case you might want to simplify the ifdefs anyhow.)

Tim points out that this helps shift up Rusty's interface simplicity continuum: the compiler won't let you get it wrong. The parameter is either used, or unused.

Of course you can also say

void a(int b) {
    (void) b;
    ...

I have a feeling there was some reason this was inferior to the unused attribute but I don't remember what it was.

posted Fri 23 Jul 2004 in /software/languages/C | link

Warm weather in Ottawa

I'm in Ottawa at OLS. It was very warm here yesterday, about 30C and humid. Meanwhile, back in Canberra they had the coldest morning this century, at a brisk -12C. Funny to be going to Canada for sunshine.

It's a nice city; charming architecture around south O'Connor St where I'm staying.

I was reading the National Post at breakfast, which seems to be considered a fairly conservative paper. Nevertheless, I read a very positive editorial about the government's legislation to decriminalize possession and use of personal amounts of marijuana, and calling for outright legalization. (I don't personally like it very much, but I think current laws are pretty silly.)

posted Fri 23 Jul 2004 in /conf/ols2004 | link

perl6

I'm in Damien Conway's Perl6 tutorial. He's a lively and entertaining speaker. It's amazing but it's awful.

I guess it makes more progress towards making any possible sequence of characters be a program that does something....

posted Fri 23 Jul 2004 in /conf/ols2004 | link

Wierd spam

Some people seem to have really wierd and specific fetishes:

From: "Hostelries F. Misadventure" <ricketiest@nesd.net>
Subject: Average Teen Cumming movies
Date: Wed, 14 Jul 2004 21:38:41 -0400

Precisely average teens?

posted Thu 15 Jul 2004 in /issues/spam/wierdness | link

Integrals and derivatives

The fundamental difference between Subversion and Arch/Darcs is that Subversion keeps snapshots of a tree, and the distributed systems collect changesets.

You can think of a tree at any point in time as being the sum of changesets to date. It is a bit like the sum or integral of those changesets. Conversely, the changesets are just the differences between the different states of the trees, like a derivative in calculus.

changesets = d/dt(snapshots)

snapshots = integral(changesets)

Obviously they form a kind of duality, we can transform from one to the other. We can, in fact, choose which one is most useful for solving a particular problem. In engineering we might choose to work in either the time or the frequency domain — and sometimes switching will make a previously intractable problem seem easy. A good notation can be more than half the solution.

The user model presented by Subversion is primarily one of snapshots of trees: I make some changes to r42, commit, and now I have r43. Darcs and Arch seem to work primarily in the changeset domain: I make some changes, commit, and now I have a new changeset in my history.

I'm speaking here only about the model presented to the user: internally, it is more complex. Subversion stores deltas to save disk space; Arch caches snapshots at various points to speed retrieval. I don't think my description here would be new to any of the people working on these systems.

Snapshots might be easier for novice users to understand because they correspond directly to what is in the working directory. It is easy to point at a directory and say it is equal to r23. It's also similar to the model used by CVS.

Working in terms of changesets, or at least having the option to do so allows more powerful operation.

For example, consider repeated merges among a related set of trees. Arch and Darcs handle this well, because they can easily remember which changesets have already come across. Subversion and CVS tend to handle it poorly, because merely tracking which version from the other tree has merged doesn't really capture the right information.

I'm not sure what the consequences of this are. I think it may mean Subversion is going to be a bit limited until it gets a more developed and natural notion of changesets.

Tom Lord said this in Diagnosing Subversion:

Suppose you have the same intuition that Walter expressed a while back, which I'll paraphrase as: "The first and most fundamental task of a revision control system is to take snapshots of working directories."

If you don't believe that that's a seductive (even though wrong) intuition, go back and look at how I replied. It took many, quite abstract paragraphs. What revision control is really about (archival, access, and manipulation of changesets) is subtle and _non_-intuitive. (Anecodtally, a few years before arch, I made an earlier attempt at revision control based on, guess what: snapshotting.) What's worse is that a set of working tree snapshots combined with a little meta-data is a kind of dual space to the kinds of records rev ctl is really about (they're logically interconvertable representations). Anything you say to a snapshotting fan about what you want to do with a changeset-librarian orientation they can reply to with "Yeah, but we could do that, too." So it's not even that the snapshot intuition is completely wrong: it's just putting an emphasis on the wrong details.

Now the transactional filesystem DB takes snapshots handily. It's ideal for that. So if you have the snapshot-intuition, and the transactional fs hammer -- you're apt to leap to a wrong conclusion: you've solved the problem!

Greg Hudson replies in Undiagnosing Subversion. I think the two mails together go a long way towards defining the different ideas of VC, though they require a bit of background.

[A] transactional FS, with a few annotations, is exactly the right hammer for version control as conceived of by Subversion ("taking snapshots of trees"). As much as one might fervently believe that this is the wrong conception of version control, it's a workable and very intuitive conception...

This is a subtle and important point, one which divides the centralized or tree-oriented version control systems (Perforce, Clearcase, CVS, Subversion) from the changeset-oriented ones (Bitkeeper, Arch). A full treatment of this issue could fill multiple journal articles, but one should recognize that it is an issue with two sides:

* Changeset-oriented version control is more powerful, but it is power which is largely unnecessary in all but the most chaotic of development projects.

* Changeset-oriented version control is harder to learn. In many environments, a shallow learning curve is the most important feature of a version control system.

* Changeset-oriented version control is hard to get right. Perhaps the best support for this statement can be found in a March 2003 note from Larry McVoy to the linux-kernel list:

http://www.ussg.iu.edu/hypermail/linux/kernel/0303.1/0130.html

* Changeset-oriented version control can be built on top of a tree-oriented foundation, although it will have all the disadvantages listed above. As Tom himself notes, tree-oriented storage is a dual to changeset-oriented storage. svk (http://svk.elixus.org/) serves as a working prototype of changeset-oriented version control implemented on top of Subversion.

I agree with Greg that changesets are harder to understand, much as calculus requires a certain a-ha moment to ascend beyond algebra. I think that suggests that all version control systems, if they want to be easy to learn, should let you do basic update-commit operations while pretending you're only working on snapshots.

I don't think you need a very complex project to get value out of changesets, though the more complex your project gets, the more you will want them. As soon as the project grows a second branch, the developers are going to want to merge between branches, and to maintain them going on. Changesets inherently handle this better than snapshots.

If the branch is stubby (one bugfix to a release), or infrequently merged, or there are few developers, then you can probably cope. People have coped in CVS for years, though with some swearing and sweating and some bugs that accidentally came in through incorrect merges. The whole point of changing is to get something which will make development more efficient and pleasant. I think the point where changesets can start to pay off is pretty low.

posted Wed 14 Jul 2004 in /software/vc | link

Google's opinion of Subversion

A lot of people seem to end up at my version control scribblings when asking Google for help with Subversion database crashes.

I guess I should put some content here, because mentioning this will probably increase the effect. So...

The immediate fix is to run svadmin recover /var/svn/fooproject, sigh, and get on with your life. It's gross that when Subversion is used as directed, people often need to run a operation which warns of the possibility of serious corruption of the repository.

The good news (well, less-bad news) is that although recovery is annoying, it rarely or never loses data. It just interrupts your work — possibly for a long time if you need a sysadmin's intervention.

A common cause of database crashes for svn+ssh is the permissions getting set incorrectly. (I don't understand why this isn't fixed in subversion; it shouldn't be all that hard.) If the permissions on the repository are screwed up then you might need to reset them. Typically on Unix the repository should have group set appropriately for the developers of the project, and should be g+w plus setuid directories.

The Subversion FAQ suggests that you avoid permission problems by not using Subversion over SSH, but using the Apache module or svn-over-tcp instead. I much prefer the model of using only SSH for access, but if you're on a closed network and permissions are biting hard you might want to change.

In the absence of proper handling of permissions in Subversion, you need to make sure all your users have a umask of something like umask 002 on the Subversion server. You can set this either in somewhere like /etc/profile or in a wrapper around svnserve.

If you're suffering Subversion database crashes, you're not alone! I've had it need recovery quite a number of times. Some people never get it though — I suppose it depends on how you share your repo, how often you interrupt operations and similar things.

It's good to make regular backups of your repository, just in case. We copy it several times a day, and make a complete dump of all repos every night. We also use the glorious rdiff-backup to keep snapshots of working directories going back three months.

The Subversion developers seem to acknowledge BDB crashes as a problem and are moving to a better storage system. Garrett tells me (30 seconds after I posted!)

The new non-berkeley db filesystem backend will actually be available in version 1.1 of Subversion (I use it now, and it's great), which should have it's first beta release any time now. Expect an actual release supporting the non-bdb back end in a few weeks, depending on how the beta goes.

In my personal experience Arch and Darcs don't get screwed up as often as Subversion does. (I don't pretend that my anyone else would necessarily have the same results.) I don't think the effect is so strong as to make you switch, but if you're thinking about moving from CVS to something else, you might keep that in mind.

posted Wed 14 Jul 2004 in /software/vc/subversion | link

Spam of the Day du Jour

From nqqap@loggain.nu Mon Jul 12 20:14:29 2004
From: "Yesenia Brooks" <nqqap@loggain.nu>
Subject: chat with assholes on the internet!

Thanks, but no. I do that enough already.

posted Wed 14 Jul 2004 in /issues/spam/wierdness | link

Growing Boies

Groklaw says:

Law.com has the most eye-opening article about David Boies and his firm. It is an older article, from February of 2003, which is just after SCO retained the firm, and the article explains some of the puzzling things we've noticed, like the abundance of typos in their legal pleadings. When you retain Boies, Schiller, you are expecting David Boies, that level of quality. According to the article, that isn't always what you get.

posted Mon 12 Jul 2004 in /issues/sco-vs-linux | link

Database Abstraction Layers

Jeremy Zawodny demands that database abstraction layers must die!

I know I've certainly spent many days writing layers above databases, GUIs, or whatever. In retrospect many of those days were wasted. Writing a generalized mapping from (say) SQL to Java is a chimera; I don't think there is any solution that is universal, efficient and simple. I think my default approach now would be to not write a database layer, but merely hold the database-related code to a single layer of the application. Forcing it through a single layer requires either losing information (doing filtering in Java because you don't want to write SELECT statements in the application) or making the layer enormous.

Linked from λ the Ultimate, who suggest the real solution is languages that can be extended to more comfortably express abstractions about data.

posted Fri 9 Jul 2004 in /software | link

GMapper

This is the coolest thing I've seen for a while: GMapper is a simple Python/GTK+ interface to the Expedia map server.

If Microsoft want to show that web services enable cool novel applications they should point to this. Expedia and GMapper share no code, no authors, are written in different programming languages (or so I assume) and run on different operating systems, located on different continents. You'd normally expect it to be very hard to interface them, not an afternoon's hack. But here through the magic of HTTP we have something far better than either developer could achieve along.

posted Fri 9 Jul 2004 in /software | link

Rusty on Interface Simplicity

Rusty's OLS 2003 keynote looks at the problem of designing and defining interfaces between software modules. For large programs, this tends to be a more important issue than the design of any one module, for a few reasons. Modules can be rewritten, but changing an interface can destabilize the whole system. You can get away with only one or a few programmers understanding the internals of a module, but almost by definition interfaces need to be understood by more people:

Once the code gets too big for one person, it's all about damage control. Interfaces make damage control possible... except when the interfaces themselves are the problem.

Rusty says the key criteria for an interface is how easy it is to use correctly or (by Murphy's Law) how hard it is to misuse. He delineates a spectrum from best to worst, which is so good I am going to shamelessly reproduce it here:

1. Compiler/linker won't let you get it wrong.

2. Compiler will warn if you get it wrong.

3. The simplest use is the correct one.

4. The name tells you how to use it.

5. Do it right or it will break at runtime.

6. Follow common convention and you'll get it right.

7. Read the documentation and you'll get it right.

8. Read the implementation and you'll get it right.

9. Read the correct mailing list thread and you'll get it right.

10. Read the documentation and you'll get it wrong.

11. Follow common convention and you'll get it wrong.

12. Do it right and it will break at runtime.

13. The name tells you how not to use it.

14. The obvious use is wrong.

15. Compiler will warn if you get it right.

16. Compiler won't let you get it right.

17. It's impossible to get right.

When you're inventing an interface, see if you can shift it up this list one or more steps. In particular, think about whether bugs are caused by interfaces that are too far down the list.

A favourite example of perverse interface design is bogofilter inverting the meaning of two flags from one release to the next.

Linus's law should have prevented that: if you change the meaning of an interface, change the name as well. In this case, if you feel the you assigned the wrong action to the -n option in the first release, then don't just change it. Make that option deprecated or invalid, and assign the new action to some other action. Years later you might reuse that option, once you're sure everyone has forgotten the original meaning, but even that is risky.

posted Wed 7 Jul 2004 in /software/aesthetics | link

distcc 2.15 released

I released distcc 2.15, which fixes a nasty bug in LZO compression which had been hanging around for a while. It turned out to be just a one line fix, but it was hard to find the problem because it only occurred intermittently and the failure happened a long time after the error. Basically I accidentally unmapped a too-long region of memory, clobbering whatever happened to be there.

I knew last time I changed it that this function was too long to be safe. I should have pulled things out to make it more obviously correct, which is what I have done now. It wasn't even a particularly complex function — certainly not unmaintainable — but large enough that it should have raised a warning.

posted Tue 6 Jul 2004 in /projects/distcc | link

CVS-style development with Darcs

Thread on CVS-style centralized development with Darcs, and strong and weak points when used in that mode.

posted Tue 6 Jul 2004 in /software/vc/darcs | link

Blogs and thought

Blogs might be defined as web sites that are regularly updated, have an author's voice, and have some kind of chronological flow. They contain hyperlinks to and excerpts from other blogs or web sites, with commentary and refinement.

In some ways this is a lot like human thought or internal monologue, perhaps more so than other types of writing.

I write articles when I think of them, and then more or less leave them aside from small revisions or corrections. This is a lot like the way I think: one might draft or redraft thoughts before expressing or acting on them, and then they're left behind. Thinking seems to be very associative, very much about taking things other people said or wrote and twisting or commenting on them. Threads happen across blogs, not within a single one, and they twist and spiral around and then fade away — much as a conversation takes place through several minds, each of which sees it differently. As you follow a conversation you might listen to different parts differently, and some parts you might not read at all. This is also like a real life conversation, and rather unlike an email or bulletin board thread.

posted Tue 6 Jul 2004 in /software/blogs | link

Which Linux distribution is the most powerful?

Mr Bad asks:

So, after seeing the umpteenth Debian package description mentioning what a powerful throbbing ur-package is barely contained within the bulging envelope of this particular .deb, I started wondering: how much of the software in Debian is actually POWERFUL? Like, so notably powerful that that's how you'd describe the software; it impresses its powerful powerness on the maintainer that much that they can't help mentioning its power.

A quick look at sid:

evan unicorn:~% apt-cache search powerful | wc -l
586

Almost 600 powerful packages is a lot of power, and I'd be afraid to install even one-tenth of those packages to my computer. Sixty mighty and powerful packages jockeying for full domination of my little ThinkPad? No way. I can't handle that kind of power. [more]

posted Fri 2 Jul 2004 in /software/linux | link

Is swap space obsolete?

There was a thread on the CLUG list recently about whether it was still useful to have swap space, now that it's quite affordable to have a gigabyte or more of memory on a desktop machine. I think it is.

Some people have the idea that touching swap space at all is a sign that the machine is very overloaded, and so you ought to avoid it at all costs, by adding enough memory that the machine never needs to swap. This may have been true on Unix ten years ago, and may still be true on some systems for all I know but it's not true for Linux.

The meaning of the term swap has changed over time. It used to mean that entire tasks were moved out to disk, and they'd stay there until it was necessary to run them again. You can do this on machines without paged MMUs, and perhaps it was simpler to implement. However, these days almost all machines have MMUs, and so we use paging instead, where particular chunks of the program (typically 4kB) can move in or out independently. This gets more use out of the memory you have, because many programs run quite happily with only part of their virtual memory in RAM. Linux doesn't implement old-style whole-program swapping at all, and there does not seem to be any reason to add it.

I'll recapitulate the way VM works, and in particular the ways it is different on Linux from in your average computer science textbook. The basic idea is that we have a relatively small fast RAM, and a slower larger disk, and we want to get the best performance out of the combination. I will skip some details and special cases for simplicity.

All memory pages on Linux can be grouped into four classes. Firstly, there are kernel pages which are fixed in memory and never swapped. (Some other systems have pageable kernels, but at the moment the Linux developers consider it too risky.) Then there is program text: the contents of /bin/sh or /lib/libdl-2.3.2.so. These are read-only in memory, and so are always exactly the same as the file on disk. There are file-backed pages, which might have changes in memory that haven't been written out yet. Finally there are memory pages that don't correspond to any file on disk: this includes the stack and heap variables of running tasks. When a program does a malloc(), it allocates memory of this type. Pages in this last category are called anonymous mappings, because they don't correspond to any file name.

There is no separate disk cache in Linux, like there is on old-Unix or on DOS. Instead, we try to keep the most useful parts of the disk in memory as cached pages. Linux usually doesn't directly modify the disk: rather, changes are made to the files in memory and then they're flushed out.

You'll notice that the free-memory measure on Linux machines is normally pretty low, even when the machine has plenty of memory for the tasks it's running. This is normal and intentional: the kernel tries to keep memory filled up with cached pages so that if those files are accessed again it won't have to go to disk. The free pool indicates just a few spare pages that are ready for immediate allocation. One time when the free memory will be large is shortly after bootup when the kernel just hasn't read in very much of the disk yet. Another time you'll have a lot of free memory is shortly after a large program exited: it had a lot of data pages in RAM, but those pages were deleted and so there's no useful information in them anymore.

We talk of pages as being clean when the in-memory version is the same as the one on disk, and dirty when they've been changed since being read in. Data pages need to get written back to disk eventually, and the kernel generally does this in the background. You can force all dirty pages to be written out using the sync system call.

The kernel can discard a clean page whenever it needs the memory for something else, because it knows it can always get the data back from disk. However, dirty pages need to be saved to disk before they can be reused. We call this eviction. So flushing pages in the background has two purposes: it helps protect data from sudden power cuts, and more importantly it means there are plenty of clean pages that can be reused when a process needs memory. So efficient is this flushing that at the moment my machine has only four dirty pages out of 256,000 (by grep nr_dirty /proc/vmstat).

As the kernel allocates memory, it firstly takes pages from the free pool. If that drops too low, it needs to free up more memory. Where does that come from? It needs to discard a clean page to make room. If there aren't any suitable clean pages then it needs to flush a dirty page to disk, then use it. This is very slow because the allocation can't continue until the disk write has finished, so the kernel tries very hard to avoid this by always having some clean pages around. (Remember the whole point of the VM algorithm is to avoid ever having to wait for a disk access to complete, by keeping pages in memory that are likely to be used again.)

File-backed pages can be flushed by writing them back to their file on disk. But anonymous mappings by definition don't have any backing file. Where can they be flushed to? Swap space, of course. Swap partitions or files on Linux hold pages that aren't backed by a file.

If you don't have swap space, then anonymous mappings can't be flushed. They have to stay in memory until they're deleted. The kernel can only obtain clean memory and free memory by flushing out file-backed pages: programs, libraries, and data files. Not having swap space constrains and unbalances the kernel's page allocation. However unlikely it is that the data pages will be used again — even if they're never used again — they still need to stay in memory sucking up precious RAM. That means the kernel has to do more work to write out file-backed pages, and to read them back in after they're discarded. The kernel needs to throw out relatively valuable file-backed pages, because it has nowhere to write relatively worthless anonymous pages.

Not only this, but flushing pages to swap is actually a bit easier and quicker than flushing them to disk: the code is much simpler, and there are no directory trees to update. The swap file/partition is just an array of pages. This is another reason to give the kernel the option of flushing to swap as well as to the filesystem.

As I write this, my 1024MB machine has 184MB of swap used out of 1506MB, and only 17MB of memory free. On old-Unix this would indicate a perilous situation: with numbers like this it would be grinding. But Linux is perfectly happy with these numbers: the disk is idle and it responds well.

The 184MB constitutes tasks that are running in the background, like the getty on the text console, or the gdm login manager. Neither of them has done anything much since I logged in days ago. From a certain overoptimizing point of view I ought to get rid of those tasks — although for the login manager it might be hard. But even then, there's probably lots of memory used for features of programs I am running that don't get invoked very often.

With swap, that memory is written to disk and costs very little. Without swap, it would be cluttering up RAM, as if I was down to only 840MB. Everything else would need to page a bit harder, but it wouldn't be obvious why.

Disk is cheap, so allocate a gigabyte or two for swap.

On BSD people used to advise allocating as much swap as memory, or maybe two or three times as much. Although the VM design is completely different, it's still a good rule of thumb. If anything, disk has gotten relatively cheaper over time: a typical developer machine now has 1GB of memory, but 200GB of disk. Spending one half or one percent of your disk on swap can probably improve performance.

If you are short on disk, as I am on my laptop, then use a swap file instead of a swap partition so that you can shrink or grow it more easily. (I think there is still a limit of 2GB per swap target, but you can create as many as you like.) Swap files might be slightly slower, but it's much better than not having it at all. If you ever see it get close to full, add some more.

Understanding the Linux Virtual Memory Manager has an enormous amount of detail on how this works in 2.4 (and I hope it doesn't contradict me!)

O'Reilly's System Performance Tuning approaches this from a sysadmin's point of view, but it mostly describes the way swap works under Solaris.

posted Fri 2 Jul 2004 in /software/linux-kernel | link

GNOME file selector broken

On upgrading to GNOME 2.6 you may find you have a pretty new file selector dialog. Sadly, it is missing a text entry field! There is no way to quickly navigate using the keyboard, and it's lost the beautiful Tab-ding feature.

In fact, it is still there, but it's bizarrely hidden. Press Ctrl+L to bring up a text field.

GNOME bug 136541

gtk-list thread about this

posted Fri 2 Jul 2004 in /software/gnome | link

Annotated Induce Act

Via Lessig, The Obsessively Annotated Introduction to the INDUCE Act.

Also wikipedia:Copynorm:

As used by copyright theorists, the term copynorm (or more frequently copynorms) is used to refer to a normalized social-standard regarding the ethical issue of duplicating copyrighted material.

posted Thu 1 Jul 2004 in /issues/copyright | link

--funroll-loops

I'm really fond of Gentoo. I'm running it on my laptop, and because distcc is very popular for speeding up whole-system compiles I deal with them fairly often. Like any group there are some pleasant and well-informed people and some who are less so. My impression is the overall level of experience (and maybe age) is lower than in Debian, but on the other hand you hear their project leaders calling each other assholes less often.

All that said, --funroll-loops is really funny:

This page is dedicated to the Linux Community's greatest ambassadors, Gentoo users. Like the annoying teenager next door with a 90hp import sporting a 6 foot tall bolt-on wing, Gentoo users are proof that society is best served by roving gangs of armed vigilantes, dishing out swift, cold justice with baseball bats to those fucking ricer bastards.

"I essentially started using Gentoo because my ....ing KDE clock would never show the right time in Red Hat."

"Watching shit scroll by for hours makes me a Linux expert overnight!"

I guess if it makes people happy and they learn something it's not all bad.

The whole thing kind of reminds of what Havoc said about metacity [img]:

Boring window manager for the adult in you. Many window managers are like Marshmallow Froot Loops; Metacity is like Cheerios.

posted Thu 1 Jul 2004 in /software/gentoo | link

Slightly used Lamborghini

eBay:

You are bidding on a slightly used Lamborghini. It only has 121 miles garage kept. Had a small brake problem.

posted Thu 1 Jul 2004 in /random/humour | link

Deryck Hodge's "Recovery"

Deryck Hodge wrote a good short story entitled Recovery:

While waiting for the 72 bus to take me to the University District, out to my job at Easy Loan, this guy, a few years younger than me and all decked out in leather jacket and jeans, said, "Hey, man. Nice day, huh? Where you headed?" [more]

posted Thu 1 Jul 2004 in /books | link

Copyright (C) 1999-2007 Martin Pool.