sourcefrog

For ever and ever

JohnHoward.blogspot.com:

Even when I'm dead, Australians will elect my corpse over the Australian Loser Party, 'cause they will know that my corpse has a proven track record and that it will keep interest rates low. Speaking of which, that was one of the best parts of this election. I reckon that the whole interest rate thing really showed how heaps smart Australians are. 'Cause like, even though every economist in the country said that they wouldn't rise under the Loser Party, Aussies were smart enough to listen to me instead. That's 'cause Australians know I'm honest and reliable and stuff.

posted Fri 29 Oct 2004 in /issues/politics | link

eBay's huge selection

Some people may find eBay's google ad offensive.

posted Thu 28 Oct 2004 in /random/humour | link

More notes on the hp msa1000 under Linux

Further random observations about using the hp msa1000 under Linux:

The msa1000 seems to have two internal SCSI channels, one for the left 7 disks and one for the right seven (DISK101-107 and DISK108-114 respectively.) It may be that spreading logical units across both channels gives better performance. (Or it may not; I haven't measured it or seen anything in the docs.)

Through the serial interface or the HP client you can allocate disks into logical units in various ways. There are a choice of raid levels, and you can assign hot spares. The raid configurations are relatively constrained though: you can only allocate whole disks, and existing units can be expanded but not shrunk. Therefore, as the manual suggests, you should think hard about your requirements before you start.

The simplest default setup is perhaps this: make a single unit containing DISK101-113 in raid level 5, with DISK114 as a hot spare.

Each logical unit appears to Linux as a separate SCSI disk, e.g. sda, etc. It works OK to just use this block device directly, but many programs (eg the RHEL installer) feel scared of devices with no partition table, so it seems like a good idea to create one, even if there's only a single partition.

The msa1000 will typically be attached via a FC HBA card such as the QLogic qla2300. Different kernels may discover this controller earlier or later than an on-board SCSI card, so the drive ordering may not be consistent: the drive may appear as sdb under a different kernel.

Given all the above, it seems like a good idea to me to use LVM, the Linux Logical Volume Manager, on top of the raw units. This has several advantages. LVM's discovery mechanism will hide any renumbering of units from the OS, giving you simple and stable names like "home". LVM allows you to shrink logical volumes, and to allocate volumes of arbitrary size, not just one disk at a time.

So what I've been doing is chopping off several disks as say UNIT0. That appears to Linux as sda. I make one big partition filling the whole unit, and then make an LVM physical volume on that. That PV gets added into one big volume group, from which storage can be flexibly allocated. If your requirements change, as they probably will, this lets you reallocate storage without needing to rebuild everything from scratch.

It probably doesn't make any sense to do raid in LVM; that can be more efficiently done by the msa1000. LVM may impose a performance cost — I haven't measured it — but if you're sure you just want e.g. one big filesystem it might be easier to do it straight on the device.

(Note that LVM has changed slightly under Linux 2.6; nodes appear by default under /dev/mapper. If you want to share volumes with Linux 2.4 you need to explicitly ask for metadata format version 1.)

The serial console works well. It runs at 19200 baud, rather than the more common 9600 8N1. A good way to get at it from Linux is as follows:

stty -F /dev/ttyS0 19200 
cu -l /dev/ttyS0

Make sure to use the right serial port. On hp ia64 machines, ttyS0 is typically the magic remote console, not the first external serial port. On laptops, ttyS0 is sometimes discovered as an infrared port. Look in /proc/tty/drivers/serial may indicate which ones are connected.

(Disclaimers: Read the manual first. Please appreciate our quality responsibly.)

posted Wed 27 Oct 2004 in /computers/storage | link

RHEL filesystems

Trap for young players: RHEL only supports only the ext2 and ext3 filesystems (not counting FAT etc.) If you happen to have a big-ass disk that's already set up in XFS this can be quite a pain.

(I wonder why? Lack of assurance in the others? Simplicity?)

The kernel-unsupported package contains reiserfs, jfs and some others but not XFS.

posted Mon 25 Oct 2004 in /software/linux | link

rsync trick of the day

I have all the ISOs for a previous release of software, and I want to get the next release from a remote machine. Since some packages probably have not changed, I want to use rsync to find the common parts and avoid transferring them.

mkchdir rhel4b1
cat ../rhel4a4/*.iso > nahant-ia64-AS-disc1.iso
cp nahant-ia64-AS-disc1.iso nahant-ia64-disc2.iso
cp nahant-ia64-AS-disc1.iso nahant-ia64-disc3.iso
cp nahant-ia64-AS-disc1.iso nahant-ia64-disc4.iso
cp nahant-ia64-AS-disc1.iso nahant-ia64-disc5.iso
rsync -avP remote::iso/rhel4b1/ ./

This uses a lot of disk while making the big temporary files but allows rsync to find matches even for data that has moved between disks.

If you do this, try not to interrupt rsync in the middle of the transfer. It will recover, but it will throw away all of the basis data for the iso that was in flight.

On reflection I think we can be even more clever by making hardlinks rather than copies for each disk. rsync will use the concatenated version for a basis version, and then break the link for the updated version.

posted Mon 25 Oct 2004 in /software/rsync | link

O'Gara's reportage

More from Groklaw on Maureen O'Gara, IBM, SCO, and the courts.

posted Sun 24 Oct 2004 in /issues/sco-vs-linux | link

The benefits of instability

There has been considerable discussion of Ubuntu by Debian developers, and LWN [sub], amongst others.

I have one additional observation: Debian's so-called inability to release may be a key to Debian's success.

Free software projects get the most benefit from community testing just before and just after a release. Few people other than the developers and the most hardcore users want to use an outright experimental branch, so those things never get much testing. Conversely, a truly stable release tends to find a decreasing number of bugs over time, and so get less benefit from the high number of users testing it.

Projects get the most benefit from having a large number of users trying out code that has lots of bugs to find, but that is not so buggy as to scare users away. Different projects have different systems for signalling this: the Linux kernel has "x.y.0" releases, others have "release candidate" snapshots, etc. Small projects like distcc can just try to stay in this state all the time, trying to make not too many and not too few changes in each release.

In recent years, Debian has had trouble getting stable releases out regularly. Many people run unstable/testing, even if they'd rather just have a timely stable release. This is often seen as a problem that should be fixed, but in an evolutionary sense, it's probably good for Debian: all those people are essentially roped into testing unstable.

If Ubuntu does manage to do timely 6 month releases, I may just use those releases. Ubuntu won't get the benefit of so many testers on their unstable builds.

(This is more of a thought experiment than an argument for either distribution or any release policy.)

posted Sun 24 Oct 2004 in /software/ubuntu | link

Votive

posted Sun 24 Oct 2004 in /photo | link

More on Wikipedia accuracy

Wikipedia is clearly not perfectly accurate; in some areas it is noticeably incomplete, inaccurate, or biased. I wouldn't want someone to rely solely on Wikipedia in making an important decision. I wouldn't want someone to rely on any single source on the net to make a truly important decision.

On the other hand, as background material, as an overview or summary of the subject, as entertainment, to merely remind you of the meaning of a word or location of a country, I think it's harmless.

In the future I think it is likely to continue to improve. I think there is strong self-regulation to make sure that good content is not lost or corrupted once it's added, and so the fraction of good bits will increase. Any one page at any one moment may be incorrect, but the probability will decrease over time. Critical reading, and applying Wikipedia-specific skills such as scrutinizing the history and cross-referencing can reduce the danger of false information.

As an example of this filtering in action, the Australian Gannet page was originally created as an attack on Wikipedia, but now it has useful content.

Wikipedia is both a product and a process. Even if the product is not yet perfect, the process ensures that at the end of every day, the encyclopedia is higher quality than it was at the beginning of the day. That doesn't ensure we will eventually attain perfection (if such a thing is even possible), but it's something to believe in.[...]

It should be noted that the three other leading online encyclopedias have disclaimers and provide no warranty as to their accuracy - Britannica, Encarta and Bartleby. Sometimes the staff of those encyclopedias forget this fact.

posted Sun 24 Oct 2004 in /software/wikipedia | link

The real reason for invading Iraq

The Economist's review of Bush's presidency gives this explanation for his foreign policy. [Sub required, mail me if you would like to see the whole text.]

It is not surprising that such a conservative president produced such a conservative response to September 11th. For a while the terrorist attacks both unified the country and turned Mr Bush into the most popular president since the second world war. Democrats and Republicans in Congress joined hands to sing "God Bless America"; the vast majority of the country supported Mr Bush's immediate decision to remove the Taliban from power in Afghanistan. But in the months after the felling of the Taliban, Mr Bush drove a wedge into the heart of American politics.
One reason why Mr Bush proved so divisive was that he embraced such a radical response to September 11th. He did not think that fighting terrorism was just a matter of bringing individuals to justice: that approach had been tried in the 1990s and resulted in catastrophic failure. He did not think it was just a matter of improving security at home: terrorists would always find a way to get through even the most cunning security systems. He argued that you need to take the battle to the enemy camp: first, by destroying terrorists in their home base and, second, by revolutionising the Middle East. America's traditional policy of cuddling up to the region's dictators and kleptocrats had turned the region into a breeding ground for Islamic extremism. What was now needed was a radically new approach, in which America would throw its weight behind the liberating force of democracy.

Mr Bush's decision to remove Saddam may have been highly controversial. But at least it sprang from a positive vision of regional transformation (people who say he took the huge risk of invading Iraq to improve his election chances are misjudging where the true political risks lay). Much less admirable is Mr Bush's willingness to exploit September 11th for partisan gain. In the mid-term elections in 2002 the Republicans relentlessly portrayed the Democrats as weak on terrorism. In Georgia they even campaigned successfully against Senator Max Cleland—a man who had lost three limbs in Vietnam—on the grounds that he was soft on homeland security.

Whether "revolutionising" the Middle East again will improve things has yet to be seen.

posted Sun 24 Oct 2004 in /issues/politics | link

Refactoring editor for English

I'd like an editor or word processor that understands enough English grammar to do intelligent refactoring, as can already be done in Java by things like IntelliJ.

For example, when I edit a sentence like this:

Please call Jack_; it's important that he be there.

it should automatically adjust the case when I change the object:

Please call Jack and Jill_; it's important that they be there.

There are grammar checkers that can detect the inconsistency but I don't know of any that will automatically fix it as you type.

Much more complex examples are possible.

posted Sat 23 Oct 2004 in /software/ui | link

Random C idea: limited integers

I wonder if this could be added to gcc

int a __attribute__((limited));

So if a every overflows or underflows, the machine will will check, perhaps by raising a SIGTRAP.

How much would it cost in performance? At the worst, one conditional branch operation after each manipulation of such a variable, to check the overflow status bit. But probably only one or two additional instructions, if we only care about limiting to the size of the variable and not an arbitrary range. Maybe something more efficient is possible.

The goal is to prevent the significant class of bugs caused by an integer overflow without necessarily having a following array overflow.

This probably needs to be optional per-variable so as not to break existing code.

Maybe this has already been done or it isn't feasible or it's just silly?

posted Sat 23 Oct 2004 in /software/languages/C | link

More opinions on 4/3

dpnow writes of the possible advantages and disadvantages of the Four Thirds standard.

posted Sat 23 Oct 2004 in /photo/tech | link

John Lott 0, Wikipedia 1

Tim Lambert writes on the resistance of Wikipedia to intentional attack, a topic I wrote about earlier.

John Lott has been anonymously editing a wikipedia article about his work, to remove information he felt is inaccurate. One of the sections that Lott dislikes discusses Lott's habit "pseudospoofing": — posting glowing reviews of Lott's own work under pseudonyms such as Mary Rosh.

Orin Kerr holds has a more pessimistic view of Wikipedia accuracy.

posted Sat 23 Oct 2004 in /software/wikipedia | link

Tool of the week: xdiskusage

xdiskusage is a great little tool to show what is using up space on a disk.

There is also Filelight which has a bit more eye-candy, but depends on KDE and so may be a bit counter-productive in freeing up disk space.

posted Fri 22 Oct 2004 in /software/tools | link

djbOS

Dan Kegel:

skadns is part of djbos, and using it means diving down a long rabbit hole and saying goodbye to Unix APIs in general. I wouldn't recommend it unless you are willing to drink the djb koolaid (which may be yummy but takes a lot of getting used to).

posted Thu 21 Oct 2004 in /software/djb | link

Free software isn't altruistic; be glad

Outsiders looking at free software tend to think that it is a kind of altruism or charity to make great software available gratis.

Some people are driven by a desire to, say, give good software to people in developing nations, or for students to have access to "the best software library in the world". That's well and good, but I think for most participants altruism is not the main motiviation. They want software freedom for themselves, and only secondarily for other people.

This is a good thing. If we were trying to persuade people to spend their evenings and weekends to write free software for the often-ungrateful general public, I don't know if we'd get very far. But if you can get people to do something that's fun, that produces a good result they can use themselves, and that incidentally improves the world, well... the results speak for themselves.

I don't know if I totally believe in the Invisible Hand theory that self-interest necessarily leads to good outcomes. The converse may be true though: most good things people do originate in some kind of self-interest, because most good or bad things people come from self-interest.

You can make a kind of evolutionary argument, however: In some societies, the behaviour caused by self-interest will tend to be constructive rather than destructive. Such societies tend to be more stable and to replicate.

[bonus Google serendipity: dissection of intelligent design creationist kook Michael Behe.]

posted Wed 20 Oct 2004 in /software/freedom | link

ipmiutil

ipmiutil seems to work OK on hp ia64 machines. You can list sensors, etc.

One limitation at the moment is that it does not correctly decode all sensor values; they're all assowed to be be "threshold" sensors. So sensors which are listed as "warn-lo" may actually be showing some other state entirely.

0000 SDR IPMB 12 15 dev: 20 00 07 01 Zircon BMC
0020 SDR Comp 02 29 20 sensnum 01 Chassis Intrus   = 00 c0 00 00 OK  
0050 SDR Comp 02 27 20 sensnum 02 Chassis Open     = 00 c0 01 00 Warn-lo
0080 SDR Comp 02 23 20 sensnum 03 Security         = 00 00 00 00 OK  
00b0 SDR Comp 02 27 20 sensnum 04 Power Button     = 00 c0 00 00 OK  
....

posted Wed 20 Oct 2004 in /software/ipmi | link

The city of software

My best friend is writing an essay about correspondances between urban planning and software/web design. She read the introduction to me last night and I wanted to hear more.

There is a navigational aspect to the web, but it hasn't worked out the way we might have expected. Gibson was very prescient, but all the ideas about net as cityscape have not come to pass. We don't represent different web sites as buildings of various sizes, and we don't ride light cycles between them. That may partially be an accident of technology, that VRML was too-wrong, too-soon. But it's also not clear that it would be useful: why would I want to perambulate when I can click straight to the destination?

And yet, different places on the web have feelings, as different parts of a city do. Some are hard and shiny; some are slums; some are street markets; some are homely.

Stephane's paper is much less rambly than this.

I'm going to take some photographs on the weekend for use as illustration.

posted Wed 20 Oct 2004 in /software/usability | link

Michael Banck on Ubuntu

Michael Banck writes on Ubuntu:

It seems Canonical managed to pull off with a tiny workforce what Debian was not able to do with a thousand volunteers. Of course, there is the mythical man month: about three dozen highly skilled and motivated developers working full time on Ubuntu can somewhat compensate for thousand volunteers of which only a tiny fraction care about releasing at all. However, Ubuntu also bravely decided to take new approaches to distribution development (at least compared to Debian) and try fundamentally different ideas, a couple of which were taken from how the GNOME community works.[...]
They have a set of rules that says they should be respectful and communicative between each other. Disputes are regulated by their technical board and community council. This warrants a good working climate between the Ubuntu maintainers, which makes Ubuntu fun to work on.
There is a rigid, time-based release schedule. Not only the release date itself is fixed well in advance, but also every major milestone along the release process.[...]
Everybody involved can upload any package (as long as the patch gets approved), there is no concept of NMUs (Non-Maintainer-Uploads). More precisely, there is no concept of package maintenance at all, different developers are just loosely appointed to specific parts of the archive, like X, GNOME, etc.

posted Wed 20 Oct 2004 in /software/ubuntu | link

Rxvt terminal in random colors

I have this script bound to a panel button. I find it helps me keep different windows straight.

exec rxvt -bg \#$[ $RANDOM % 4 ]$[ $RANDOM % 4 ]$[ $RANDOM % 4 ] "$@"

Screenshot of desktop with colored terminal
windows

posted Tue 19 Oct 2004 in /software/junkcode | link

Fixed again in 2.6.9

A different Minolta bug apparently caused breakage this time. There is a workaround by Matthew Dharm for 2.6.9-rc4. The necessary patch hunk is

diff -Nru a/drivers/usb/storage/transport.c b/drivers/usb/storage/transport.c
--- a/drivers/usb/storage/transport.c   2004-10-10 19:59:55 -07:00
+++ b/drivers/usb/storage/transport.c   2004-10-10 19:59:55 -07:00
@@ -908,6 +911,7 @@
   int result;
 
        /* issue the command */
+       us->iobuf[0] = 0;
        result = usb_stor_control_msg(us, us->recv_ctrl_pipe,
                                           US_BULK_GET_MAX_LUN, 
                                           USB_DIR_IN | USB_TYPE_CLASS | 
@@ -918,7 +922,7 @@
            result, us->iobuf[0]);
 
        /* if we have a successful request, return the result */
-       if (result == 1)
+       if (result >= 0)
           return us->iobuf[0];
 
        /*

This should apply against anything reasonably recent from 2.6.

I can't really recommend attaching this camera over USB though, because it runs down the NiMH batteries extremely quickly. It's better, if at all possible, to read direct from the card through a PCMCIA carrier or a printer or a USB attachment. It may be quicker too, if you have a better interface than the USB1 on the DiMAGE.

posted Tue 19 Oct 2004 in /software/linux-kernel/usb | link

Blame W

Phil Greenspun:

If we can say "we didn't vote for W" we are considered good citizens of the world. George W. Bush attracts all of the hatred.
Maybe we should take advantage of the fact that we have our scapegoat in place. We can make a list of all of the countries that we need to invade, install puppet governments in, or steal their natural resources. If W. loses the election we go on a big military spree until mid-January and then Kerry can come in and say "We had nothing to do with the fact that Bush kicked your asses but sadly the U.S. government never apologizes for anything or returns any loot."

posted Mon 18 Oct 2004 in /issues/politics | link

Ontology of bug reporters

Benjamin Drieu

If you are involved in the development or in the packaging of a program, sooner or later you'll get bug reports. Every bug report is unique, but most of them fall into one category. Based on my Debian or upstream experience, here are the most common families of bug submitters I had to deal with.

[from Joey]

posted Mon 18 Oct 2004 in /software/bugs | link

Interview with Olympus engineers and thoughts on 4/3

Olympus are running an extended interview with the engineers who worked on the Four Thirds system and the E-1 and E-300 cameras. It gives an uncommon insight into the design tradeoffs.

Most DSLRs available in 2004 are based on 35mm SLR cameras, which allows some reuse both by manufacturers and users. However, producing a sensor the size of a full 35mm frame is quite expensive, so cameras tend to have a "crop factor": only the center of the image plane is actually used to capture images.

Olympus have chosen to go with a new all-digital system based on a sensor somewhat smaller than that used in mid-level DSLRS from other manufacturers.

Manufacturers are fairly free to change sensors at will between fixed-lens camera models. However, for DSLRs customers expect to be able to reuse lenses between different generations of cameras. So by picking a particular sensor size and designing lenses to match, Olympus are committing to this size for perhaps ten years, which is an eon in electronics. Engineering tradeoffs can be tough, but particularly so when you're going to be stuck with them through four or more generations of technology.

Lenses specifically and bodies designed from size can be smaller than those based on the 35mm format, and therefore faster, lighter and cheaper at a given quality level. (Or the optical quality can be improved at a given price point, etc.)

On the other hand, a smaller sensor means that at a given resolution level, each sensor site must be smaller, and therefore possibly able to gather less pixels, and so less sensitive. At a given level of sensor technology, aperture and speed, larger sensors ought to have lower noise.

Or does it? Much of the light coming through a 35mm SLR lens falls outside an APS-sized sensor, and so is essentially wasted. The Olympus cameras should get more useful data bits per photon entering the lens.

To cut a long ramble short, the Olympus engineers seem confident that they will continue to be able to improve within 4/3 system, achieving quality comparable to medium-format and then large-format silver-halide cameras. Worth a read.

posted Mon 18 Oct 2004 in /photo/tech | link

Falling in the crack

Ed:

[Black and White is] too explicit to really be art, but not explicit enough to be porn. So, in the end, it's a miserable failure.

posted Mon 18 Oct 2004 in /photo | link

Survey of using statistical processes to model behavior for computer security

Interesting information, beautifully presented by the Ambient Empire.

[from: NTK]

posted Fri 15 Oct 2004 in /tech/math | link

Ubuntu versus...

Comparisons of the new Ubuntu Linux operating system to others.

(These comparisons are based on my limited experience. You may disagree. You can mail me if you like.)

Ubuntu vs Debian

+ Almost all Debian packages are available.

+ No idiotic debian-devel flamewars.

+ Six-month predictable upgrade cycles are far nicer than 2-year-old or bleeding-edge. I can get security and bug fixes through apt without worrying that it will break libc.

+ No dselect! dselect put me off Debian for about three years in the 90s.

+ Sane and simple installer. (I haven't tried the new Debian installer for a while, and I hear it's got better. Of course, most existing Debian users are so cowed that they install by cloning an existing system...)

+ No hand-hacking X configurations, at least on my machine.

− Can't upgrade in-place from Debian; presumably you need to move Debian's system files out of the way and install Ubuntu. Not terribly hard.

Ubuntu vs Gentoo

+ Both seem to be reasonably fresh. Both care about a default install that's reasonably pretty.

+ No pointless rebuilding of software that could be packaged as binaries.

+ Having seen the Gentoo install instructions, I think I'd rather circumcise myself — anonymous.

− Gentoo's ebuild system for defining new builds is far cleaner than that of dpkg.

Ubuntu vs Fedora

They're alike in many ways: Both are open source and free to download, have friendly installers and GNOME-based desktops. Both appeal to both traditional Linux power users and non-shell-oriented users. Both have a single commercial commercial sponsor: Canonical and Red Hat.

Both, in different senses, continue on from previous work: Debian and legacy Red hat respectively.

? I haven't used Fedora enough to really compare.

+ Last time I looked, Fedora was criticized for being insufficiently open to people outside Red Hat.

+ Red Hat use Fedora as an input to their for-money distributions, whereas Ubuntu is apparently an end to itself. Whether that affects the result I don't know.

+ Ubuntu installed automatically on my Evo N410c laptop and Fedora hung. (Fedora may have since fixed this.)

Ubuntu vs Red Hat Enterprise

Red Hat Enterprise is really a services package these days, not (just) a Linux distribution, so the comparison is inexact.

+ I'd rather buy a new camera if I had this much money. Even at work, where I theoretically am entitled to use RHAS for ia64, it's a big pain to deal with their quasi-licensing restrictions. In fact, it's far easier to get a new install of Windows from MSDN and keep it up to date. Something's wrong there.

− No commercial support for Ubuntu (yet); no indication of whether or how this will happen.

− No hardware certifications (yet). Expensive 64-CPU servers are going to be running mostly SLES or RHEL for the near future.

Ubuntu vs Mac OS X

(High bar, but very much in GNOME's sights.)

− Macintosh hardware integration is far better: they only have to support a few configurations, they have all the specs, the hardware is designed to work with the OS and it's attractive in its own right. Plug a TV into the composite port and it just works. Sleep/hibernate just works.

− The Mac GUI is really beautifully polished, both visually and ergonomically. GNOME is getting better but isn't there yet: fonts on X11 used to be awful and are now OK.

+ On the other hand, you don't need to buy new hardware, PC hardware is cheaper per cycle, and you can reboot into Windows to play games.

+ Linux is now the standard Unix. Ubuntu is more consistent with what's likely to be running on your servers. Linux is faster on some benchmarks, but all benchmarks are bogus, and particularly so for desktops.

+ Linux utilities tend to be free, not annoying $10 shareware. (OK, I know about Fink, but still.)

+ Better (though slightly less consistent) keyboard navigation.

+ Less Macintosh historical baggage. OS X suffers some friction between the Mac and Unix parts, in for example handling case sensitivity or resource forks. (Mac apps can be in the wierd state of running but having no windows open, which seems to me of very marginal benefit. It's debatable.)

− No single-source 1-800 support.

− No pretty PDF alpha-blended display (yet), but I'll live. Less consistency in the GUI (though it's coming). Not every program runs under GNOME, so there's likely to be some inconsistent GUI toolkits popping up for some time.

− Some functions just have no GUI configuration tools at all: configuring wireless or a VPN may require sudo vi .

+ If something annoys you, you can patch it yourself.

+ The defaults are good, but more of them can be changed; if you don't want Nautilus or Metacity you can just install something else. That's harder/more scary on Mac OS.

+ Both GNOME and OS X are more attractive than Windows. GNOME is probably a slightly easier transition for Windows users, though neither would be very hard.

posted Wed 13 Oct 2004 in /software/ubuntu | link

More on fragmentation of lisp

I complained previously about the relative fragmentation of Lisp into incompatible dialects. Each of several times I've tried to do something nontrivial I seem to find that the book or library I want to use doesn't correspond to the implementation I have available.

We're developing a networking library at work in Python. It opens sockets; it parses XML; it has a web and a graphical interface (using PyGTK). It's not enormous (about 5kloc), but useful. With relatively little effort Tim got not just the library but also the GUI to run on both Linux and Windows. If he wanted, he could probably get it to run on his Wince handheld as well. This is pretty cool, but not really remarkable: of course it runs everywhere.

Some of the data structure manipulations would probably be simpler in Lisp than in Python. Maybe it could be only 3kloc or less. That would be great. But I don't think we could find a single open Scheme or Lisp implementation which had the right libraries and ran on every platform.

The most-commonly-suggested Common Lisp implementation, SBCL, doesn't run on Windows, or even 64-bit Linux platforms. (I just discovered there is a GTK+ binding in prerelease, which is a start.)

Ten or more years ago people expected languages and environments to be a bit different between machines. You picked a system, then wrote in whatever dialect existed there. Adding a dependency on a library was a substantial risk. Open source and scripting languages have expanded expectations: everything ought to run everywhere by default, and there ought to be libraries for almost everything by default. (Seven years ago it was a big deal that Java supposedly ran everywhere. Now Python just does it.)

Python may be converging on Lisp, but it would be really nice if Lisp could converge on Python too.

posted Mon 11 Oct 2004 in /software/languages | link

Death by prettiness

Mike Johnson writes on photo.net of scenic fatigue, which is what I was trying to get at a while ago.

What instigated these thoughts was that I recently spent several hours poking around on a large picture-posting site (I won't name it explicitly, to try to avoid hurting anyone's feelings). For the most part what I looked at are amateur pictures, and not even polished, finished presentations — just a large mass of random pictures that people have taken, and, for some reason, liked. So it wouldn't be proper or fair to criticize too harshly just because what's presented isn't always art.

When I first start looking at large numbers of snapshots, I can get excited about possibilities, and my mind is full of comments I might make to the photographers. But after a while, a particular kind of fatigue sets in.[...]

I get this feeling quite strongly, and it has pretty much turned me off sites where people post random photos. Partly it is this fatigue, and partly it's just that other people far better photos of sunsets than me: they're more creative, more patient, more skilled, have better gear, better luck, or whatever.

When you choose to take a sunset, you have to really engage with sunsets — learn how to look at them, learn how to distinguish among them. What are the best sunset pictures? How to they work? How does yours stack up? Instead of taking one sunset and being satisfied, you ought to take a hundred sunsets and choose one.

What I want to do is make a good representation of how the moment felt to me. This is something I won't get from looking at any number of other photographs of similar scenes, however technically or aesthetically excellent. And it's still a challenge; merely taking a random photograph at a moment doesn't necessarily capture that moment.

posted Mon 11 Oct 2004 in /photo | link

Mandatory registration for free information

A number of news sites, such as The Age, have started requiring registration to read.

On this point consider the registration agreement for the wonderful bugmenot.com [from boingboing]:

What percentage of sites do you visit that require registration?
What percentage would you be comfortable with?

Annual Income (in US dollars)

Explain how a search engine like Google would function if no content was publicly accessible:

How many sexual partners have you had in your life?

To some extent it's a technical problem: I don't mind disclosing that I'm a 46yo Anguilan herpetologist, but I do mind filling out a clumsy and longwinded form &mdash just to read a story that they very likely lifted from AP or Reuters anyhow. So I pretty much just close the window on these things.

I can sympathize with publishers wanting more tightly targetted ads (or, as they call them these days, "articles".) So I'd like to propose a less awful way to find this out: put a little survey next to the articles, asking people for random information. Explain that it helps make the site sustainable. Ask one question at a time, and make it voluntary: "what state do you live in?", "how old is your car?", etc.

posted Mon 11 Oct 2004 in /issues/privacy | link

Theo de Raadt interview

Sam Varghese interviews Theo de Raadt.

posted Sun 10 Oct 2004 in /software/openbsd | link

Peter Norvig on learning

The best kind of learning is learning by doing. To put it more technically, "the maximal level of performance for individuals in a given domain is not attained automatically as a function of extended experience, but the level of performance can be increased even by highly experienced individuals as a result of deliberate efforts to improve." and "the most effective learning requires a well-defined task with an appropriate difficulty level for the particular individual, informative feedback, and opportunities for repetition and corrections of errors."

posted Sat 9 Oct 2004 in /software | link

Abuse of "literally"

Robert Fulford:

Writing in the Toronto Sun last week, Douglas Fisher said that the prime minister, when pushing through the bill to compensate some but not all of those infected with hepatitis by tainted blood, "literally laid on his whip."

posted Thu 7 Oct 2004 in /words | link

Canoe polo

I wouldn't have believed there was such a sport as Canoe Polo, but apparently there is.

canoe
polo
Canoe Polo, CISAC, Canberra, 2004-10-03

update: Ed says his favourite esoteric non-erotic watersport is underwater hockey. Wow.

posted Thu 7 Oct 2004 in /photo | link

Kernel hackers debate: Is Canberra boring?

LKML thread:

From: Alan Cox
Subject: Re: Loops in the Signed-off-by process

Linus Torvalds wrote:
> On Fri, 1 Oct 2004, alan wrote:
> > (Kind of like Mountain View without all the excitement
> Oohh-keey.. "Mountain View without all the excitement".
> ALL THE EXCITEMENT? MOUNTAIN VIEW?
> Brain overload.
Visit Canberra then you will understand.

From: Benjamin Herrenschmidt

> Visit Canberra then you will understand
I respectfully disagree :) Canberra at least has some good pubs :)

Ben.

posted Thu 7 Oct 2004 in /conf/lca2005 | link

The Lisp Paradox

Paul Graham wrote a while ago on the idea that good programming languages are converging on Lisp.

In it, he presents a programming language elegance microbenchmark: write a function that returns a function that accumulates values. To some extent this is skewed towards Lisp — it's impossible to write at all in pure functional languages, despite that some people find them very valuable. And it's slightly artificial in Python: if you wanted to sum the values in some sequence, you'd just do it without using a function.

But still, it does show Python as a language is perhaps not quite so elegant as Lisp. I'm inclined to agree. And yet I find Python far more useful.

Though Paul didn't say so, I think his results demonstrate the big problem with lisp:

Lisp: Arc (def foo (n) [++ n _])

Lisp: Common Lisp (defun foo (n) (lambda (i) (incf n i)))

Lisp: Goo (df foo (n) (op incf n _))

Lisp: Scheme (define (foo n) (lambda (i) (set! n (+ n i)) n))

Four rather different definitions, all for a nearly-trivial function. (Four different spellings of a single keyword!) And this is just for a pure algorithm, with no consideration of how to make the program run, or how to print the results.

We may leave aside the experimental Arc and Goo dialects, or concentrate on only Scheme or CL, but allow for the different ways of launching various lisp interpreters and compilers on Linux and you're probably up to a dozen variations. The most basic thing for a Unix language is to have a "shebang" line that can go at the start of a program to make it executable, but Lisp/Scheme doesn't even standardize this.

And this is to say nothing of the libraries you might need to do something useful, like split a string or display a dialog or parse web input. Not only do the libraries differ, but even the syntax for loading a library differs.

Of course you could just pick one dialect, but which one? If you say "I'm using Foo Lisp" in particular then you're using a very unusual language, with corresponding worries about it being dropped or inadequately supported in the future. Sometimes it's worthwhile.

The Python solution to the benchmark is longer and slightly klunky compared to Lisp, but at least it will run everywhere Python runs, including under the CLR and the JVM. There is a single standardized method for calling this function from another program.

Peter Norvig (who knows infinitely more about Lisp than me) says

Python can be seen as either a practical (better libraries) version of Scheme, or as a cleaned-up (no $@&%) version of Perl.

Pathological variation is an endemic disease for Lisp: it's fun and easy to write an interpreter, so everyone does, producing a maze of incompatible versions, eternally irrelevant. I say that sadly, not triumphantly, because I would really like a good Lisp that was widely useful, but I don't expect to ever get it.

posted Thu 7 Oct 2004 in /software/languages | link

Peter Norvig's important lessons

Peter Norvig lists 52 important programming lessons from his book Paradigms of AI Programming. My favourites, which apply to any language or domain:

Use the most natural notation available to solve a problem.
Whenever you develop a complex data structure, develop a corresponding consistency checker.
To solve a problem, describe it, specify it in algorithmic terms, implement it, test it, debug and analyze it. Expect this to be an iterative process.
Use data-driven programming, where pattern/action pairs are stored in a table.
The expert ~~Lisp~~ programmer eventually develops a good "efficiency model".
There are four general techniques for speeding up an algorithm: caching, compiling, delaying computation, and indexing.
Translating inputs to a canonical form is often a good strategy.
The first Lisp interpreter was a result of a programmer ignoring his boss's advice.

posted Wed 6 Oct 2004 in /software | link

Detecting undergrad CS plagiarism: "show your working"

Plagiarism is fairly rife in undergraduate computer science courses: code can be copied from the net, or co-developed between different students. Andrew wrote recently on algorithmic methods for detecting plagiarism. It is indeed rather ironic that people will put so much hard work into avoiding the work they're supposed to do.

Here is another approach: require students to show their working, by submitting a version-control history of their work. Inspection of the history will show whether the code was developed in a plausible way by gradual evolution and refinement, or whether it miraculously appeared fully-formed. If you expect people to do most of their work in a lab or on the network you might require them to use a central repository, which makes it fairly trustworthy. (Seeing a rush of commits through the night before the deadline may give tutors some entertainment.)

This imposes little or no additional burden on honest students, beyond making them learn rudimentary version control, which is easy and useful anyhow. I suppose it's possible for people to cheat by gradually developing towards somebody else's code, but it makes cheating more difficult. I think it might still show up.

None of this stops people sharing designs or test cases, but that's really a separate problem.

I suppose tutors might even give students feedback on whether they developed the software in a good way: was their top-down/bottom-up order appropriate, did they make avoidable false starts, and so on. (Detailed feedback is rare for cost reasons, but it'd be nice...)

This is similar to requiring working papers to be submitted with written exams in mathematics or science.

I wonder if this has ever been tried?

(related: Phil Greenspun is trying to work out what's wrong with undergraduate CS education.)

Responses:

Andrew likes the idea, but Byron is skeptical.

Certainly an appreciation of versioning is something that comes with time, but that's also true of comments and good structure, and we still ask students to use them.

What happens if someone really does just commit the whole thing? Pretty much the same as if they just write down the answer with no working on a maths exam: they lose points for not showing, and there is a heightened suspicion of cheating. Of course most markers don't check every line of working on every paper, but they check that it's there.

posted Wed 6 Oct 2004 in /software/education | link

Python rocks for testing

I'm testing a custom XML-object mapping library. Writing test cases in Python is incredibly pleasant:

class BaseParseTest(comfychair.TestCase):
    def runtest(self):
        tt = pywbem.tupletree.xml_to_tupletree(self.input_xml)
        result = pywbem.tupleparse.parse_any(tt)
        self.assert_equal(result, self.expected_obj)

class KeyBinding_Numeric_Test(BaseParseTest):
    input_xml = """<KEYBINDING NAME="answer">
        <KEYVALUE VALUETYPE="numeric">42</KEYVALUE>
    </KEYBINDING>"""
    expected_obj = {"answer": 42}

and that's it! Rinse, repeat. Add more as problems are discovered.

I'm using my comfychair test library here, which is yet another rewrite of the JUnit idea, specialized for the kind of tests I often write.

In particular:

It's easy to statically declare complex nested object structures. It would be in Perl or Ruby or Lisp too; it's hard in Java or C or C++. It's even hard to have an embedded multi-line XML document in those languages.
It's easy to compare those nested objects.
If something goes wrong, you're likely to get a nice traceback showing where things broke, and allowing use of the postmortem debugger.

It's only a matter of degree, but being 50% easier to create test cases has a compounding effect on the amount of test coverage.

Python has much weaker compile-time tests than say Java or C. The ease of writing tests tends to make up for this though, and to get you to a higher state than Java that merely compiles but is not yet tested.

posted Wed 6 Oct 2004 in /software/testing | link

Wikpedia vandalism experiment

Chris perpetrated what I'd call a grey-hat hack against Wikipedia — inserting some incorrect information to see what would happen. Rusty calls it vandalism. The specific change was an assertion that there is a colony of Australian Gannets at ANU, and they eat pizza. I removed it. (Sorry, Chris.) It was slightly funny, which is why I left it there for a week after I originally saw it.

So Chris proved that incorrect information can persist in Wikipedia for a matter of a few weeks, even when the page is reviewed by some people. OK, so you can't believe everything you read on the Internet — that's hardly news.

I'm sure there are errors in Wikipedia. There are probably errors in non-free encyclopedias too, though perhaps proportionally less. On the other hand, traditional encyclopedias move much more slowly and can't cover current events to the same extent. If they tried to do that, their error rate might increase.

I suspect the influx of accurate content willl dilute the errors over time. Errors are removed in a kind of stochastic process — any particular error may persist for days or months, but eventually they go. Accurate content increases monotonically — it's easy to detect and revert vandalistic deletion. (This is a key advantage of Wikipedia over the web as a whole: if a page has just one error or inaccuracy you don't need to write a whole new page.) So I expect the fraction of errors to slowly decrease.

If you want reliable information then cross-checking Wikipedia with a traditional encyclopedia and other materials will probably give better results than any source alone.

I don't think marking articles as "reviewed" or "draft" would really fix it. A critical reader can get a much better sense of reliability from cross-referencing, looking at the article history and Talk page, etc. (Anyhow, how can you trust the "reviewed" tag?) As Douglas Adams pointed out, humans are cunningly designed to do trust calculations in firmware.

posted Tue 5 Oct 2004 in /software/wikipedia | link

Writing tests will make you happy

Mark Dominus has wise advice on testing:

Writing tests will make you happy
If you make it into a big chore, you won't do it

So don't make it into a big chore

Effective tests can be extremely simple

posted Tue 5 Oct 2004 in /software/testing | link

Olympus E-300

Olympus have announced a new consumer DSLR, the E-300. It uses their 4/3-system sensor and a mirror viewfinder, so is remarkably compact for a camera at this level. I've wanted to switch to a DSLR for a while, and maybe this would be it. I have looked at the E-1, but it's just a bit too expensive for the amount I'd use it.

This seems to be the E-1 scaled down and with some more amateur features added, such as scene modes. (I'd personally be happy with just PASM, but I guess it's necessary for this market.) It does have a built-in pop-up flash, which plays well with the compact size. The RAM buffer might be smaller than that of the E-1 and it's missing the splash-proof triple seals.

The sensor is a bit smaller than on the competing Canon 300D and the Nikon D70, as a tradeoff for having a substantially smaller body and lenses. I wonder if the smaller sensor is going to cause substantially more noise at any given ISO. On my current Minolta 7i there is noticeable noise at ISO 400 and 800 is often barely usable.

So far (29 Sep) a few people have seen it at the Photokina expo, but there's not much on the net aside from the Olympus press release and spec sheet.

The physical design of the camera looks intriguing, and it looks like it will fix a few annoyances in digital cameras to date.

There's no top LCD, as there is on the E-1 and many other digital cameras today. All the information is available on the review TFT or through the viewfinder. That seems like a pretty good tradeoff: there's already a large display on the back, and information in the viewfinder, so I'm not sure a top display is really needed. If it keeps the price and size down, by all means pull it out; the only downside is probably higher current drain when the TFT is on.

Playback is just through a single button, so presumably you can pop back into shooting mode just by half-pressing the shutter. Putting playback on a mode dial always seemed pretty pointless to me.

There are hard buttons for all the common functions, plus a single rotating control: exposure, WB, resolution, ISO, autofocus mode and point, flash mode, meter mode, (maybe more). It's probably a good tradeoff between interface simplicity and quick access; I hope it will let the user keep out of the time-consuming menus. So all in all it looks like quite a step forward in interface design; let's hope the photo quality is as good.

Olympus have some opinions of Japanese pros on the E-1 — the automatic sensor cleaning and compact lenses seem like the strong points. It looks like the lens system and sensor have carried over from the E-1 to the E-300, so hopefully it will do just as well.

Ted's Cameras are now advertising it at a price of AUD 2000 (approx USD 1400) (to be confirmed), available in November. That would puts makes it a little more than the EOS 300D, and below the Nikon D70 and EOS 20D.

Links:

posted Sat 2 Oct 2004 in /photo/tech | link

Archives 2008: Apr Feb 2007: Jul May Feb Jan 2006: Dec Nov Oct Sep Aug Jul Jun Jan 2005: Sep Aug Jul Jun May Apr Mar Feb Jan 2004: Dec Nov Oct Sep Aug Jul Jun May Apr Mar Feb Jan 2003: Dec Nov Oct Sep Aug Jul Jun May