I happen to have a repository of xaraya going back to 2002 in git, so I thought I’d give it a shot.

…more time passes than originally planned…

Result

8 years of commit history in 10 minutes of video. Produced with gource from a git repository of Xaraya. The video traces the 2.x main branch back to its origin, repository wise. The start is the import of the CVS postnuke repository into Bitkeeper. Since then xaraya has switched to monotone.

Gource settings used:

gource --stop-position 1.0 \
       --camera-mode overview \
       --bloom-intensity 0.18 \
       --user-image-dir .git/avatar \
       --highlight-all-users \
       --output-framerate 60 \
       -s 0.5 \
       -720x576 \
       -a 0.3 \
       --hide filenames \
       --user-scale 1.4 \
       --date-format %Y-%m-%d \
       --disable-progress \
       --output-ppm-stream - \
| ffmpeg -y -b 3000K -r 60 -f image2pipe -vcodec ppm -i - -vcodec libx264 -vpre default gource.mp4

This produced a video of 11 min. 35 seconds. To bring it back to 10 minutes the framerate was increased until the total time fell just below 10 minutes.

I specifically wanted the whole history to be in one video of 10 minutes (the Youtube maximum) which means compromising a bit on the quality. If there are things I can do within these assumptions to improve the video, I’d like to hear them.

There were two things on the list which stopped me in my daily work. The first was was the need to recompile the complete macports collection. I won’t go into the details of everything, but executing a complete reinstall of macports apps takes about a day, not counting the time needed to fix compile errors (wtf, all *.la files missing now?).

The second was the disfunctioning of a number of Mail.app plugins. I guess it’s not really the plugin authors who have dropped the ball here, as Apple had an early release of SL and many developers were just not ready. However, that didn’t change the fact my *nix tools and mail were both severely crippled now.

Not looking forward to spending two days to recompile and reconfigure everything and nothing new to show for I decided that a long time wish could be sneaked in. Leaving Mail.app behind and switching to Claws as a mail client.

Specifically, these are the main gripes I have with Mail.app:

• keyboard navigation is absent for all practical purposes;
• tabbing order in compose window or absence of shortcut keys to go to subject/body at once;
• weird threading decision; (altho i have to admit it works for the daily chores mail)
• no way to configure top/bottom posting;
• no nntp support;
• IMAP support does not include subscriptions;
• somehow it keeps locking up on me with tenacious regularity, i suspect sqlite here;
• the direction with yellowy sticky notes, the stationary things are all useless to me;
• proprietary, closed program (which I could live with…)
• but worse, it has a non-documented API for plugins, which is an insult to developers.

I’ve always liked claws on linux. Tremendously fast, great keyboard support, a feature-set which is more than I can handle in most areas and a small enough codebase which makes it realistic for me to patch it or write a plugin for, should I feel inclined to do so. The only problem I really had was running OSX, so the lightweightedness would be kinda lost, as I would need a while slew of dependencies. Still, most of them I already had because of our internal implementation of OpenERP

I knew I would be running into issues, so the first couple of days I ran with claws-mail inside VirtualBox running an Ubuntu install. This would give me sort of a reference to get everything working without having to worry about compilation digging etc. Installing a new mail application is easy but deciding if it is good enough to replace a program which you use every minute of the day is something else.

I sort of knew after half a day I was going to like it. The slowness of the virtual machine was a little bit in the way, as was its instability, but overall things were looking good. Ok, time to repeat the excercise for real on OSX.

Here’s the configure step I used:

export CPPFLAGS=-I/opt/local/include
export LDFLAGS=-L/opt/local/lib

./configure \
  --disable-trayicon-plugin \
  --disable-manual \
  --enable-ipv6 \
  --disable-dillo-viewer-plugin \
  --enable-crash-dialog

The first two are perhaps redundant, but I wanted to make sure everything came from the ports collection and not from libraries supplied by Apple by default. Compiling afterwards gives you a running, but ugly claws-mail. So, first thing I did was install a bunch of gtk2 stuff to give me a bit of theme-ing options. I used Platypus to create a Claws.app; the OSX application bundle so I can have an icon in the Dock:

Apart from the icon, that does not give you much really. The app runs as a child below X11.app so it still misbehaves in many ways, but alas.

Next up was ‘mailto’ URI handling. I had thought this to be an easy thing. OSX refuses to accept a shell script as handler though, it must be an app bundle. So, what I needed was an app-bundle, as invisible as possible which can act as an URI handler for claws. The first thing I thought about was adapting MailToMutt to call out to claws instead of mutt. Looking at the source I saw it was pretty much suitable to mutt only, so I set out to write my own, How hard could this be?

As it turns out, such an app in its simplest form would require the following:

1. An info.plist file, as all app bundles have, with some special entries;
2. A declaration of a handler on how to react to the involved ‘Apple Events’
3. The code of that handler.

The plist has to look something like this:

There are 2 things in there which make this interesting. First, the highlighted line marks this application as an ‘Agent’ which just means to not show it anywhere on screen.

Second, a list of URL-types on which this URIhandler app should react. As you can see I threw a bunch in there which I thought would come in handy someday. What this does is make the app known to OSX as being capable of handling these types of URLs. This is used, for example by the Mail.app application for listing clients in its preference for default Email program:

The second file needed in the XCode project is URLHandler.scriptSuite:

{
    Name = URLHandler;
    AppleEventCode = "UrHD";

    Commands = {
        "GetURL" = {
            CommandClass = URLHandlerCommand;
            AppleEventCode = GURL;
            AppleEventClassCode = GURL;
        };
    };
}

which registers URLHandlerCommand as the handler for Events of type GURL (get an URL).

And finally, the third file, URLHanderlerCommand.m which implements that handler:

@implementation URLHandlerCommand

- (id)performDefaultImplementation {

    NSString *urlString = [self directParameter];
    NSURL    *url = [NSURL URLWithString: urlString];

    // Log what we got
    NSLog(@"url = %@", urlString);

    // Launch _handler script in path
    NSTask *task = [NSTask new];
    NSString *script = [NSString stringWithFormat:@"~/bin/%@_handler",[url scheme]];
    NSLog(@"launchtarget = %@", script);
    [task setLaunchPath:script];

    // Give the URI-string as parameter to that script
    [task setArguments:[NSArray arrayWithObject:urlString]];

    // Catch stdout / stderr
    [task setStandardOutput:[NSPipe pipe]];
    [task setStandardError:[task standardOutput]];

    // Run it
    [task launch];

    // Log the output, if any
    NSData* output = [[[task standardOutput] fileHandleForReading] readDataToEndOfFile];
    NSString* out_string = [[[NSString alloc] initWithData:output encoding:NSUTF8StringEncoding] autorelease];
    NSLog(@"%@",out_string);

    // If we are not there we cant do no harm, so quit.
    [[NSApplication sharedApplication] terminate:nil];

    return nil;
}
@end

In words: It takes the URL that the user clicked on “mailto:something@something.org”, turns it into an URL object and calls out to the file mailto_handler in the Users bin directory.

For a ‘feed’ URL the app would call the file feed_handler and likewise for the other protocols. The mailto_handler file is just a shell script containing:

/usr/local/bin/claws-mail --compose $1

and my ~/bin/feed_handler contains:

/usr/local/bin/claws-mail --subscribe $1

If you would like a copy of the source of this program, go here: http://mrblog.nl/files/2009/09/URLHandler.zip

The rest of the configuration went pretty smooth. I had a little hickup getting GPG signing to work with both PGP and S/MIME signatures. But a quick recompile of gpgme to include s/mime and a specific configuration which was not obvious to me for gpg-agent solved that.

Here’s a picture of the end result (click on it for large version):

So, is this install perfect? Is it even better than Mail.app? I’d say, not yet. But the big difference is that I have control now and there’s only a gap of time and energy between me solving any gripe I might have.

Which of the gripes have I solved with this? Let’s recap:

✓ keyboard navigation is absent for all practical purposes;
keyboard navigation is strong, but not complete, in claws. One thing which is missing is a keyboard way to collaps or expand a folder tree (and thread, for that matter)

✘ tabbing order in compose window or absence of shortcut keys to go to subject/body at once
This one is unsolved, at least on osx. There is no key combination which allows to go to subject and / or body directly.

✓ weird threading decision; (altho i have to admit it works for the daily chores mail)

✓ no way to configure top/bottom posting;
claws has templates, which allow you to configure what a reply (or forward) looks like. This includes the cursor position, so it allows for top-posting, bottom-posting or anywhere posting. And all this is per account.

✓ no nntp support;
claws has full nntp support.

✓ IMAP support does not include subscriptions;

? somehow it keeps locking up on me with tenacious regularity, i suspect sqlite here;
We’ll have to see how claws performs here.

✓ the direction with yellowy sticky notes, the stationary things are all useless to me;

✓ proprietary, closed program (which I could live with…)
claws is licensed under the GPL v3

✓ but worse, it has a non-documented API for plugins, which is an insult to developers.
i’m not sure the claws API for plugins is documented (i havent looked) but being open source, there’s enough to work with.

I’d say that is a pretty good score! If it feels like that in the real world? Talk to me in 3 months :D I’m also pretty sure there will be a whole set of new gripes waiting for me around the corner.

The first 25 years seem to deal mostly with getting the “looks” right. A minority succeeds. A substantially larger part of the population think they have succeeded.

In the next 25 years the brains (should) get some more attention. Possibly the earlier mentioned ratio of actual versus perceived success is even worse here. If there is a third or even a fourth quarter for people, looks are pretty much left behind, while in some cases the brains still can produce some occasional surprise.

The above is an intro to what I was really wondering about: How does a software package evolve with respect to “looks” and “brains” in its life? (As always i’m mostly focussing on the typical open source package and how that comes to be)

Being the person that I am, that brings roughly this image into my head:

Inevitably i tried to catch the question in some sort of equation. Here’s roughly the train of thought:

• as the analogy is a “natural” thing some sort of “natural logarithmic” component came to mind
• work being mostly done in (large) groups which means that “things inforce things” (bad or good) some exponential part should be in there
• it would be nice if we could have a simple number to characterize the curve, like a quality factor Q or something.

In the picture above the blue line would be the ideal(?) but rather boring mix where the looks and brains are always completely in balance. (The lifeline starts in the origin following the line with age). The red line would be a “code-quality” oriented progression where the green line focusses on looks. The rough idea is to get from the origin to some point on the blue line as far away from the origin as possible. By varying that Q factor that point can be placed anywhere on the blue line. The formula i came up with was this:

which corresponds to the red line. (the green line is the reciprocal where x and y switched places). The interesting part is to play with the value of Q, look at the graph and reason about the software product. Or vice versa, reason about your project and determine it’s Q factor.

I tend to view at an application development process like this:

1. make it work
2. make it right
3. make it nice
4. make if fast

For Xaraya, the 1.0 release signals to me we’ve been able to make it work. Now, let’s try to make it right. Some ideas on how follow.

If you view this article by itself (what we call the display view in Xaraya), you’ll notice an extra part on the right side “Related entries” which lists other entries which have the same keywords attached to them. This allows a reader to quickly read all entries which have the same or similar taxonomy. The block isnt fully functional yet, because it reacts to some keywords while it doesn’t for others, so there’s still some digging to do there. (FIXED)

Outstanding task list:

• making Xaraya react on the mt_tb_ping_urls member
• deal with the allow/disallow flags for trackback and comments
• find a home for mt_excerpt
• implement the mt.supportedTextFilters method to allow for markdown or textile input

The more advanced features of the MT API I havent implemented (yet), but one thing is supported from the beginning, which is the Extended Entry.

The metaweblog api in Xaraya is incomplete, but the first result is there: multiple category support, directly from the blog client. (This post is in two for example)

The combo of PHP and GTK makes up for a reasonable alternative (for java for example) for platform independent applications. Especially if combined with libglade. Glade is a tool which is used to build graphical interfaces and store this in a XML format. PHP-GTK is able to read those XML-files and dynamically build the interface. This way, the interface is separated nicely from the code and can be changed without changing the code. :-))

Example:

<?xml version="1.0"?>
<GTK-Interface>
<widget>
<class>GtkWindow</class>
<name>windowMain</name>
<width>150</width>
<height>80</height>
<title>PHP-GTK-Glade for world domination</title>
<type>GTK_WINDOW_TOPLEVEL</type>
<position>GTK_WIN_POS_NONE</position>
<modal>False</modal>
<allow_shrink>False</allow_shrink>
<allow_grow>True</allow_grow>
<auto_shrink>False</auto_shrink>
<widget>
<class>GtkButton</class>
<name>button</name>
<can_focus>True</can_focus>
<signal>
<name>clicked</name>
<handler>on_button_clicked</handler>
</signal>
<label>Exit</label>
<relief>GTK_RELIEF_NORMAL</relief>
</widget></widget></GTK-Interface>

And the code:

function on_button_clicked() {
echo "Clickedn";
}$gx = &new GladeXML('interface.glade');
$gx->signal_autoconnect();

This creates a window with a button on it and connects the clicked signal to the function on_button_clicked. With minimal effort this application can be deployed on any platform which supports PHP and GTK. (someone else may figure out how many, but lots).

Because the glade interface is in XML this can be deployed locally or remotely at will. For PHP we know that it can run both locally and on the server. This combined opens up some interesting possibilities. We now can deploy interface and code both locally and remotely.

The usual goodies which relate to XML are of course applicable to glade, like for example XSLT transformations on the interface file (for example filtering out interface elements which have no usefull meaning due to privilege restrictions).

The combo opens up interesting options for bringing desktop applications and web applications closer together. The whole suite reminds one of techniques used in XUL applications and of Java deployment strategies. I gather if we could hook up this combo to an object broker service like Corba and a PHP compiler we’d have a very powerfull platform.

It certainly warrants further investigation.

This list should at least include:

• goal: what are we trying to improve, a little bit more verbose than fix #bug 49712 is appreciated
• peer review: each refactoring task is performed by two people (at least) . Who’s coding and who’s reviewing is up to the duet
• tests: which tests should be done to check whether goal is reached. (this is actually the most important step)
• solution strategy: refactoring has some well known terms to work with, especially for object oriented systems. Try to classify the refactoring into one of these terms, so an objective reference is available

It would rock if we could use a refactoring browser to make it easier to perform the refactoring.

Note: refactoring is not adding functionality, you go from a working situation to a working situation, not from a broken situation to a (hopefully) working situation.

Pitfalls

refactoring backtrack explosion: suppose you want to fix a bug, the solution is apparent, you fix it, but you recognize the code can be improved by making it more general. You create a refactor sheet for it and start thinking about it. Almost immediately you see that there is a lower level change required to make the top level refactoring viable, so you create another refactoring sheet, pulling in an extra dependency for that refactoring. If this process repeats itself 2 times, you can be sure your code is not othogonal enough. Instead of making the refactorings smaller and smaller until the changes are properly localized and you can start implementing them, you pull in more and more dependencies. The process fall into a deadlock, suddenly you don’t know where to start and you’re not sure which dependencies to check anymore.

If you find yourself in this situation, let the orginal bugfix sit in the code (make a FIXME though!) and go walk the dog. Don’t think about the problem for a while. At a later time, return to the problem and try to solve it at a higher level, not changing API or interfaces, but generally making code more orthogonal without solving the original problem.

What you’re actually doing at that time is converting the system to a new state, so your refactoring stack will be different than the one described above. Naturally you use the knowledge you have about the old stack.

Practical application

How to organize this in real life? Use the bugtracker for the refactoring sheets? It’s a good start. Start simple at least, but make sure information is in a repository (but I say that with all information produced)