Mar 22

In Working with Jekyll and Org-Mode a simple solution was given to use org-mode format files as posts.

That solution had quite a few limitations:

  1. the files still needed the ’---yaml header to be present on the first lines, making it invalid org-mode documents
  2. only posts were supported, not documents, collections and pages
  3. the #+key: value syntax of org-mode was not used for settings, or at least only partially
  4. working with liquid tags was cumbersome.

What follows are some notes on how I solved the issues.

Getting rid of the ’---’ requirement

Jekyll uses the ’---’ header to determine which files need to be processed. If such a header is not present, the file is either copied verbatim or ignored, depending on the settings. Having this header in an org-mode file makes it somewhat invalid. Not a bit deal, but luckily it’s rather easy to get rid of.

The function in standard Jekyll that determines this is has_yaml_header so by extending that function we can make sure org-mode files are treated to be processed like there was a header.

module Utils
  def has_yaml_header?(file)
    !!((File.open(file, 'rb') { |f| f.read(2) } =~ /^#\+/) or
       (File.open(file, 'rb') { |f| f.read(5) } =~ /\A---\r?\n/))
  end
end

The check is just for the file to start with #+ which is enough for me, for now.

Dealing with the yaml header settings

Getting rid of the ’---‘–marker is one thing, but between those markers are settings which are relevant for the document. At least title, tags and layout are usually present in the header. With the yaml-header gone we need a way to register those variables in some org-mode syntax.

The standard org-mode key/value pairs as mentioned above are suitable for that. The method is implemented statically in the OrgConverter class:

def self.process_options(content, data)
    org_text = Orgmode::Parser.new(content, {markup_file: "html.tags.yml" })

    org_text.in_buffer_settings.each_pair do |key, value|
      # We need true/false as booleans, not string.
      if key.downcase == 'published' #(any others?)
        value = value.to_b
      end
      data[key.downcase] = value
    end
    data
  end

The implementation is not very elegant, but it registers all org-mode buffer settings as values. An exception must be coded for values which must be boolean. (I use only the published property, but there may be more)

Handling liquid tags

Here’s where things get a bit hairy. Formally, when writing content which requires a liquid construct, it should be enclosed in the proper org-mode block delimiters, something like:

#+BEGIN_HTML
   {% liquid construct here %}
#+END_HTML

such that the whole document is a normal org-mode document. As my current documents do not have the ’BEGIN_HTML/END_HTML’ delimiters this would be annoying to change (again). So, the first thing I tried is to implement it to avoid all that changing again. That worked, a fully working implementation is at commit cc2081

However, there are significant issues with this:

  1. documents are still odd org-mode documents; no real improvement from the originals (just a bit less non-org-mode constructs)
  2. build performance of jekyll is unacceptable in the implementation (roughly 3 times slower!!)
  3. to solve the problems plugins need to be adapted, so it’ll quickly become a nightmare.
  4. the code already is more complex than it needs to be (include tags, quote replacement)

After realizing this, I started over and just coded the convert method as:

def convert(content)
  org_text = Orgmode::Parser.new(content, {markup_file: "html.tags.yml" })
  org_text.in_buffer_settings.delete("TITLE")  # We already captured the title on processing the options
  org_text.to_html
end

Given I made the change to all my orgmode files as mentioned above, which is not that much work with emacs’ dired-do-query-replace-regexp command, the conversion is complete. The only thing to make sure is that for each type of source file (post, page, document) the header options are processed with the process_options function above.

You can view the end result in commit 40dad2

May 23

Many people in the Emacs community use orgmode. Quite a few of them use the org-babel system to write and maintain their emacs configuration, and so am I. I find the biggest advantage to be able to document my thought process as well as the configuration details. Emacs configuration can get pretty big and maintaining it becomes a lot easier for me if I can read back what I was thinking 6 months ago.

The system borrows from literate programming concepts where you write code and documentation in one document and let tools tangle the code and documentation into separate documents.

The default way to publish an Emacs configuration with that system would be to let org-mode export the configuration document to html and publish that somewhere. What I wanted to do was to use the in-place org-mode converter I am using with Jekyll. In short, the ideal would be that jekyll fetches the authoritative version of my emacs configuration in org-mode syntax and treats that as /normal/ content and publish it.

I started off by defining a collection, a new feature of jekyll, which gave me a chance to use it. The collection is defined in the _config.yml file:

# Emacs collection contains emacs 'generated' documents
collections:
  emacs:
    # Render them, i.e. convert them to html. 
    output: true

This will render source files in the _emacs folder as if they were pages. The url generated for them will be prefixed by /emacs/.

In this folder I placed a file config.org with the following contents

---
layout: orgmode
--- 

[... text left out  ..]


{% include extern/mrb.org %}

That includes the file <root>/_includes/extern/mrb.org from the jekyll installation and renders its content. The only correction I had to make was to insert a raw...endraw block around some org-mode settings.

You can view the result of the rendering in /emacs/config.html

Nov 16

Most of my notetaking and task management is in org-mode so it makes sense to use that as the basic format of my postings too. This is usually done by publishing a project from org-mode into the location where jekyll keeps its files and let jekyll convert that into something publishable. I’m using a slightly different setup which is more effective for blog posting.

Publishing with org-mode turned out harder than I thought. The publish process is pretty demanding in org-mode and you only end up with raw documents that still need to be processed by jekyll. It occurred to me that http://github.com has the ability to render org-mode documents directly and perhaps that library could be used to turn it into a jekyll plugin so I could use the org-mode format directly.

Turns out there was a recent commit which mentioned an org-mode converter plugin; long story short: installed it and never looked back. Having jekyll convert org-mode files directly saves the whole publishing configuration step in Emacs which would otherwise be needed from within org-mode.

The converter is simple; it uses an org-ruby call to convert the org-file to html and that’s it really:

module Jekyll
  # Convert org-mode files.
  require 'org-ruby'
  class OrgConverter < Converter
    safe true

    def setup
      # No-op
    end

    def matches(ext)
      ext =~ /org$/i
    end

    def output_ext(ext)
      ".html"
    end

    def convert(content)
      setup
      Orgmode::Parser.new(content).to_html
    end
  end
end

I can now just write org-mode files with a frontmatter and they’ll end up automatically as blog postings. As yaml frontmatter needs to come first in the file to make jekyll happy this can’t be hidden in an org-mode construct like a comment block or something else that org-mode itself ignores. This makes it harder to use the blog postings for anything else than jekyll because the frontmatter will get in the way; exporting the file to PDF for example. There is obviously room for improvement, but this simple plugin directly gives a workable system.

To have a reference document for writing I created a test org-mode file, with the rendered result here. This file helps to check what org-mode constructs render into something useful and verifying visual layout of them. Not everything worked as I had hoped, but given the amount of complexity that got eliminated I’m quite happy with it.

Issues that I found in the rendering:

  • the headers start at level 1 which is probably 1 or 2 levels too high for my purpose; I haven’t found a way to correct this yet. I probably should file a feature request for this;
  • footnotes do not work, which I would use to keep links nicely at the bottom of an article.
  • some rendering is ugly (blockquotes for example), but that’s probably not a direct consequence of the org-mode converter
  • there are only a couple of org-mode environments supported;
  • the use of liquid tags that jekyll uses is somewhat cumbersome.

I was pleasantly surprised by the code highlighting though, which worked out of the box for me.

The next step is finding or making some helper functions in emacs lisp to support working with drafts and publishing.

Aug 30
Jun 03

Many people got attracted to bitcoin the last couple of months. This explosion of interest probably caused the monetary value of bitcoins to show the same shape as the google trend-line chart pictured below.