MrBlog

Sep 29

As a side-effect of Migrating to Claws I lost my OSX addressbook usage, at least for mail. In my company we use OpenERP for CRM, invoicing and other business needs. This means that the majority, if not all the email-addresses I need are in our OpenERP database.

So, it made sense to finish an effort I started earlier, which is to link OpenERP to our LDAP server and thus be able to query information from the OpenERP database through an LDAP interface and have every address available in the Claws addressbook (and the OSX addressbook too for that matter).

In an earlier version of OpenERP we used a specific module for this, which basically published an LDAP entry whenever we changed data in the OpenERP database. This worked, but was a less than ideal solution, not to mention it stopped working on an OpenERP upgrade. The solution I wanted was to have the data available in LDAP immediately. This meant making the LDAP server a “client” of the OpenERP database or, said another way, making the OpenERP database function as an SQL backend to the LDAP server.

Defining an SQL backend for LDAP

Creating an SQL backend for slapd is, albeit terse, documented. I mainly used the information at the OpenLDAP FAQ. The basic idea is that the LDAP-server connects through ODBC to the database, the OpenERP database in our case, and translates information found in relational tables to a subtree of the LDAP hierarchy.

To model this information, you have to create at least 3 tables in the database:

  1. ldap_oc_mappings: which objectClass of LDAP is stored in what table;
  2. ldap_attr_mappings: how attributeTypes of an objectClass are resolved from RDBMS data;
  3. ldap_entries: what’s the DN of an entry, and how the entry relates to its objectClass mapping and to its parent DN;

The FAQ mentions two other tables, which we do not need. The idea is to specify in these three tables how the LDAP server gets to the entries and what they mean. For addressbook-like entries for mail, the defacto objectClass to use for this is something referred to an ‘inetOrgPerson’. The collection of these objects will be below an objectClass ‘organizationalUnit’, giving the simplest ‘tree-relation’ we can think of.

Making the SQL backend use OpenERP

The table ldap_oc-mappings is queried by the LDAP server to map these object classes to tables, so the server knows in which tables to look for the attributes of these two classes. The next two statements insert two rows in that table, one for each object Class, mapping them to the tables ldap_inetOrgPerson and ldap_organizationalUnit, expecting a column id to contain the primary key for the objects.

INSERT INTO ldap_oc_mappings(name,keytbl,keycol) 
    VALUES('inetOrgPerson','ldap_inetOrgPerson','id');
INSERT INTO ldap_oc_mappings(name,keytbl,keycol) 
    VALUES('organizationalUnit','ldap_ organizationalUnit','id');

The ldap_inetOrgPerson is actually a view over the res_partner_address table in OpenERP, so it uses the data directly.

CREATE OR REPLACE VIEW ldap_inetorgperson AS 
  SELECT 
    a.id, 
    btrim((COALESCE(a.firstname,'')||' ') || a.lastname) AS cn, 
    btrim((COALESCE(a.firstname,'')||' ') || a.lastname) AS displayname, 
    a.phone AS telephonenumber, 
    a.lastname AS sn, 
    a.firstname AS givenname, 
    a.fax AS facsimiletelephonenumber, 
    a.mobile, 
    a.private_phone AS homephone, 
    lower(a.email) AS mail, 
    a.street, 
    a.zip AS postalcode
  FROM res_partner_address a
  WHERE 
    a.email  ''  AND 
    a.email  '"' AND 
    a.type  =  'contact';

This gives a dataset of all people who actually have an email-address registered in the OpenERP database. The column aliases are not needed as such, but make the construction of the ldap_attr_mappings table a bit easier. The second objectClass we registered in ldap_oc_mappings, organizationalUnit can be modelled with one simple row in the table ldap_organizationalUnit:

INSERT INTO ldap_organizationalUnit(name) VALUES ('addressbook');

With that row, we basically define one organizational unit in our simple tree named addressbook under which all our objects of type inetOrgPerson will be placed.

So, at this point we have 2 objectClasses registered, we have created the raw data for them. What’s left? Two things, first, we need to define how the attributes of the raw data relate to the object attributes. For this, the table ldap_attr_mappings contains a row for each attribute. For the telephoneNumber attribute, the data row is as follows:

INSERT INTO ldap_attr_mappings(
        oc_map_id,name,sel_expr,from_tbls,join_where
    ) 
VALUES(
    1,'telephoneNumber','telephoneNumber',ldap_inetOrgPerson,'1=1'
);

This says basically to the ldap server that in order to get to the telephoneNumber attribute for inetOrgPerson (oc_map_id 1 refers to the first row in the ldap_oc_mappings table), it needs to look in the table ldap_inetOrgPerson, use the same attribute name for the column and apply no special where clause. It is basically a recipe for the server to translate an LDAP request into an SQL query.

For each of the columns in the ldap_inetOrgPerson view, such a row needs to be present in the ldap_attr_mappings table.

Still here? The final step is to create the third meta table ldap_entries. This table is basically the lookup table to map ldap-index values to rdbms-index values. I have defined ldap_entries as a view on the raw data as follows:

 CREATE OR REPLACE VIEW ldap_entries AS 
   SELECT 
     0 AS id, 
     'ou=addressbook,dc=hsdev,dc=com' AS dn, 
     2 AS oc_map_id, 
     0 AS parent, 
     0 AS keyval
    UNION 
   SELECT 
     ldap_inetorgperson.id, 
     ('cn=' || ldap_inetorgperson.cn) || 
     ',ou=addressbook,dc=hsdev,dc=com' AS dn, 
     1 AS oc_map_id, 
     0 AS parent, 
     ldap_inetorgperson.id AS keyval
   FROM ldap_inetorgperson;

This does 2 things, it refers the ldap addres ou=addressbook,dc=hsdev,dc=com as the organizational unit (oc_map_id = 2) and assigns that ID 0. The second part of the UNION then formats each of the inetOrgPerson addresses as an LDAP address in the constructed tree (mine would be: cn=Marcel van der Boom,ou=addressbook,dc=hsdev,dc=com) and maps it to the ID in the raw data table (keyval) and puts the organizational unit as its parent.

With the above a minimal proof of concept can be constructed so that each partner address which has an email-address shows up in LDAP. After this, it’s a matter of configuring the email-client using the ou=addressbook,dc=hsdev,dc=com as a search base.

Here’s a picture of my record in the claws address book:

screen_011.png

The same concept can be applied to other data in OpenERP (the partner records themselves come to mind or user accounts). It would not be that hard to wrap the above into an OpenERP module to manage this. Once the LDAP server has a configuration to use an SQL backend, all configuration can be done in OpenERP itself, modelling access using the meta tables. Perhaps I’ll do that at some point, if some of my customers would benefit from this too.

Sep 18

After installing Snow Leopard I was left with a page of programs and plugins that needed attention because of the upgrade. Most of these were little nuisances and just needed a little reconfiguration. There were a couple of things where Apple decided to rename an app or hide it otherwise from view (Kerberos Ticket manager for example).

There were two things on the list which stopped me in my daily work. The first was was the need to recompile the complete macports collection. I won’t go into the details of everything, but executing a complete reinstall of macports apps takes about a day, not counting the time needed to fix compile errors (wtf, all *.la files missing now?).

The second was the disfunctioning of a number of Mail.app plugins. I guess it’s not really the plugin authors who have dropped the ball here, as Apple had an early release of SL and many developers were just not ready. However, that didn’t change the fact my *nix tools and mail were both severely crippled now.

Not looking forward to spending two days to recompile and reconfigure everything and nothing new to show for I decided that a long time wish could be sneaked in. Leaving Mail.app behind and switching to Claws as a mail client.

Specifically, these are the main gripes I have with Mail.app:

  • keyboard navigation is absent for all practical purposes;
  • tabbing order in compose window or absence of shortcut keys to go to subject/body at once;
  • weird threading decision; (altho i have to admit it works for the daily chores mail)
  • no way to configure top/bottom posting;
  • no nntp support;
  • IMAP support does not include subscriptions;
  • somehow it keeps locking up on me with tenacious regularity, i suspect sqlite here;
  • the direction with yellowy sticky notes, the stationary things are all useless to me;
  • proprietary, closed program (which I could live with…)
  • but worse, it has a non-documented API for plugins, which is an insult to developers.

I’ve always liked claws on linux. Tremendously fast, great keyboard support, a feature-set which is more than I can handle in most areas and a small enough codebase which makes it realistic for me to patch it or write a plugin for, should I feel inclined to do so. The only problem I really had was running OSX, so the lightweightedness would be kinda lost, as I would need a while slew of dependencies. Still, most of them I already had because of our internal implementation of OpenERP

I knew I would be running into issues, so the first couple of days I ran with claws-mail inside VirtualBox running an Ubuntu install. This would give me sort of a reference to get everything working without having to worry about compilation digging etc. Installing a new mail application is easy but deciding if it is good enough to replace a program which you use every minute of the day is something else.

I sort of knew after half a day I was going to like it. The slowness of the virtual machine was a little bit in the way, as was its instability, but overall things were looking good. Ok, time to repeat the excercise for real on OSX.

Here’s the configure step I used:

export CPPFLAGS=-I/opt/local/include
export LDFLAGS=-L/opt/local/lib

./configure \
  --disable-trayicon-plugin \
  --disable-manual \
  --enable-ipv6 \
  --disable-dillo-viewer-plugin \
  --enable-crash-dialog

The first two are perhaps redundant, but I wanted to make sure everything came from the ports collection and not from libraries supplied by Apple by default. Compiling afterwards gives you a running, but ugly claws-mail. So, first thing I did was install a bunch of gtk2 stuff to give me a bit of theme-ing options. I used Platypus to create a Claws.app; the OSX application bundle so I can have an icon in the Dock:

Creating a <a href='http://claws-mail.org' rel='external ' title='Claws email client'>Claws</a> icon

Apart from the icon, that does not give you much really. The app runs as a child below X11.app so it still misbehaves in many ways, but alas.

Next up was ‘mailto’ URI handling. I had thought this to be an easy thing. OSX refuses to accept a shell script as handler though, it must be an app bundle. So, what I needed was an app-bundle, as invisible as possible which can act as an URI handler for claws. The first thing I thought about was adapting MailToMutt to call out to claws instead of mutt. Looking at the source I saw it was pretty much suitable to mutt only, so I set out to write my own, How hard could this be?

As it turns out, such an app in its simplest form would require the following:

  1. An info.plist file, as all app bundles have, with some special entries;
  2. A declaration of a handler on how to react to the involved ‘Apple Events’
  3. The code of that handler.

The plist has to look something like this:

Plist file needed

There are 2 things in there which make this interesting. First, the highlighted line marks this application as an ‘Agent’ which just means to not show it anywhere on screen.

Second, a list of URL-types on which this URIhandler app should react. As you can see I threw a bunch in there which I thought would come in handy someday. What this does is make the app known to OSX as being capable of handling these types of URLs. This is used, for example by the Mail.app application for listing clients in its preference for default Email program:

App registered as URL handler

The second file needed in the XCode project is URLHandler.scriptSuite:

{
    Name = URLHandler;
    AppleEventCode = "UrHD";

    Commands = {
        "GetURL" = {
            CommandClass = URLHandlerCommand;
            AppleEventCode = GURL;
            AppleEventClassCode = GURL;
        };
    };
}

which registers URLHandlerCommand as the handler for Events of type GURL (get an URL).

And finally, the third file, URLHanderlerCommand.m which implements that handler:

@implementation URLHandlerCommand

- (id)performDefaultImplementation {

    NSString *urlString = [self directParameter];
    NSURL    *url = [NSURL URLWithString: urlString];

    // Log what we got
    NSLog(@"url = %@", urlString);

    // Launch _handler script in path
    NSTask *task = [NSTask new];
    NSString *script = [NSString stringWithFormat:@"~/bin/%@_handler",[url scheme]];
    NSLog(@"launchtarget = %@", script);
    [task setLaunchPath:script];

    // Give the URI-string as parameter to that script
    [task setArguments:[NSArray arrayWithObject:urlString]];

    // Catch stdout / stderr
    [task setStandardOutput:[NSPipe pipe]];
    [task setStandardError:[task standardOutput]];

    // Run it
    [task launch];

    // Log the output, if any
    NSData* output = [[[task standardOutput] fileHandleForReading] readDataToEndOfFile];
    NSString* out_string = [[[NSString alloc] initWithData:output encoding:NSUTF8StringEncoding] autorelease];
    NSLog(@"%@",out_string);

    // If we are not there we cant do no harm, so quit.
    [[NSApplication sharedApplication] terminate:nil];

    return nil;
}
@end

In words: It takes the URL that the user clicked on “mailto:something@something.org”, turns it into an URL object and calls out to the file mailto_handler in the Users bin directory.

For a ‘feed’ URL the app would call the file feed_handler and likewise for the other protocols. The mailto_handler file is just a shell script containing:

/usr/local/bin/claws-mail --compose $1

and my ~/bin/feed_handler contains:

/usr/local/bin/claws-mail --subscribe $1

If you would like a copy of the source of this program, go here: http://mrblog.nl/files/2009/09/URLHandler.zip

The rest of the configuration went pretty smooth. I had a little hickup getting GPG signing to work with both PGP and S/MIME signatures. But a quick recompile of gpgme to include s/mime and a specific configuration which was not obvious to me for gpg-agent solved that.

Here’s a picture of the end result (click on it for large version):

screen_010.png

So, is this install perfect? Is it even better than Mail.app? I’d say, not yet. But the big difference is that I have control now and there’s only a gap of time and energy between me solving any gripe I might have.

Which of the gripes have I solved with this? Let’s recap:

keyboard navigation is absent for all practical purposes;
keyboard navigation is strong, but not complete, in claws. One thing which is missing is a keyboard way to collaps or expand a folder tree (and thread, for that matter)

tabbing order in compose window or absence of shortcut keys to go to subject/body at once
This one is unsolved, at least on osx. There is no key combination which allows to go to subject and / or body directly.

weird threading decision; (altho i have to admit it works for the daily chores mail)

no way to configure top/bottom posting;
claws has templates, which allow you to configure what a reply (or forward) looks like. This includes the cursor position, so it allows for top-posting, bottom-posting or anywhere posting. And all this is per account.

no nntp support;
claws has full nntp support.

IMAP support does not include subscriptions;

? somehow it keeps locking up on me with tenacious regularity, i suspect sqlite here;
We’ll have to see how claws performs here.

the direction with yellowy sticky notes, the stationary things are all useless to me;

proprietary, closed program (which I could live with…)
claws is licensed under the GPL v3

but worse, it has a non-documented API for plugins, which is an insult to developers.
i’m not sure the claws API for plugins is documented (i havent looked) but being open source, there’s enough to work with.

I’d say that is a pretty good score! If it feels like that in the real world? Talk to me in 3 months :D I’m also pretty sure there will be a whole set of new gripes waiting for me around the corner.