• # Process DMARC reports with sieve

Share:

I get a lot of DMARC reports because I host mail for a couple of domains. Most of these mails require no attention as they are just notifications that others use one of our domains. I want to separate these mails from my normal mail workflow and auto archive them if I haven't looked at them within, say, 2 weeks.

Doing this with sieve server-side has my preference, but apparently it's not trivial to determine the age of a message, which is the core logic needed here. Also, the processing of sieve rules is normally only during reception of messages, not ad-hoc or on some other event, although dovecot and pigeonhole have some options for this, among others the sieve-filter tool.

I really only found one implemenation online which roughly solves the same problem I was having, but this involved more than needed I think.

My solution consists of 3 parts:

1. the sieve script that handles DMARC reports on reception and age-ing;
2. use of an extension that calls an external program to evaluate expressions to determine age;
3. a daily job that runs the sieve script in the scope of the designated folder.

Here's the sieve script which deals with DMARC reports both in the normal INBOX flow and a special treatment after 14 days. The latter part is not automatic by dovecot on reception of emails, but triggered by a run of the sieve-filter program.

 1    require ["date","fileinto","relational","variables","environment","imap4flags",
2             "vnd.dovecot.execute", "vnd.dovecot.environment"];
3
4    # Parameters
5    set "dmarc_folder" "Folder.for.dmarc-reports";
6    set "purge_days" "14";
7
9    if environment :is "vnd.dovecot.default-mailbox" "INBOX" {
10      if anyof (
14        header :contains "From" "opendmarc@box.euandre.org"  )
15      {
17        fileinto "${dmarc_folder}"; 18 stop; 19 } 20 } 21 22 # When running in the dmarc_folder, archive when age is <purge_days> 23 if environment :is "vnd.dovecot.default-mailbox" "${dmarc_folder}"
24    {
25      if currentdate :matches "julian" "*"
26      {
27        # Run a simple bc expresssion to get <purge_days> ago from todays julian day
28        execute :output "purge_date" "bc" "${1} -${purge_days}";
29
30        # Compare this with Date header and archive when age reached
31        if date :value "le" "Date" "julian" "${purge_date}" 32 { 33 fileinto "Trash"; 34 stop; 35 } 36 } 37 } The first part of the sieve script just moves the mails into the dmarc-reports folder and is a normal sieve processing rule. The second part runs if the default folder is the dmarc-reports folder. If so, it uses the ext_program extension of the sieve interpreter to let the bc program evaluate the expression for the age of the message. This uses a tiny script in the configured sieve execute bin directory of the ext_programs extension 1 #!/bin/sh 2 echo${1} | /usr/bin/bc

which just pipes the input given by the sieve line into the bc program. On returning, stdout is put into the purge_date variable. I'm using execute because I do not need to pipe the whole message into the external program, but specify input specifically.

With the above configuration I can set a cron job in the crontab of the vmail user to run

1  sieve-filter -We -u <mymailaccount> \
2               /path/to/vmail/mymailaccount/sieve/dmarc-archiver.sieve \
3               Folder.for.dmarc-reports

which executes the sieve script mentioned above in the IMAP folder <dmarc_folder> only.

I'm not sure why sieve makes it so difficult to get the age of an email (unless I'm missing something). Protonmail solves this by having a custom extension 'vnd.proton.eval' which does something similar like the above, but in the scope of the sieve language itself without having to shell out to an external program explicitly. (I think; I have not seen their implementation)

My approach above obviously has some drawbacks:

• the bc external program is called for every mail that matches, fine for 10 or 20 I guess, but rather inefficient if the amount of matched messages is big. For now, not a problem.
• unsure what sort of security consequences this has, the execution scope and environment is very limited, but we're still giving control to a script calling other programs.

Share:
• # Hike: Katwijk aan Zee - De Zilk LAW 05-2 (04)

Share:

This post is the result of a little hike along one of the Long Distance walks in the Netherlands: "LAW 05-2 (04) Katwijk aan Zee - de Zilk", part 4 of "Nederlands Kustpad"

I recorded the hike with my Suunto Traverse watch and exported this as a GPX file. I used that gpx file as example data to create a little gpx viewer which can be embeded in html pages. With Leaflet this is almost trivial to do.

The details of the hike are on my Movescount page for this move.

If you zoom in on the map, you'll see that the recorded path is not an exact match with the followed path (the dotted lines). This is because I used a long interval of GPS fixes to record the path. I think it was a 10 second interval, so there's some tuning to do for this in future.

The exported GPX has information about temperature, height, energy, speed and possibly some other stuff. My intention is to have this information incorporated in the map later on, similar to the way this is displayed on the movescount site.

If you're interested, the GPX file is here:

• # Further work on jekyll org-mode support

Share:

In Working with Jekyll and Org-Mode a simple solution was given to use org-mode format files as posts.

That solution had quite a few limitations:

1. the files still needed the '---' yaml header to be present on the first lines, making it invalid org-mode documents
2. only posts were supported, not documents, collections and pages
3. the #+key: value syntax of org-mode was not used for settings, or at least only partially
4. working with liquid tags was cumbersome.

What follows are some notes on how I solved the issues.

##### Getting rid of the '---' requirement

Jekyll uses the '---' header to determine which files need to be processed. If such a header is not present, the file is either copied verbatim or ignored, depending on the settings. Having this header in an org-mode file makes it somewhat invalid. Not a bit deal, but luckily it's rather easy to get rid of.

The function in standard Jekyll that determines this is has_yaml_header so by extending that function we can make sure org-mode files are treated to be processed like there was a header.

1  module Utils
3      !!((File.open(file, 'rb') { |f| f.read(2) } =~ /^#\+/) or
4         (File.open(file, 'rb') { |f| f.read(5) } =~ /\A---\r?\n/))
5    end
6  end

The check is just for the file to start with #+ which is enough for me, for now.

##### Dealing with the yaml header settings

Getting rid of the '---'–marker is one thing, but between those markers are settings which are relevant for the document. At least title, tags and layout are usually present in the header. With the yaml-header gone we need a way to register those variables in some org-mode syntax.

The standard org-mode key/value pairs as mentioned above are suitable for that. The method is implemented statically in the OrgConverter class:

 1  def self.process_options(content, data)
2      org_text = Orgmode::Parser.new(content, {markup_file: "html.tags.yml" })
3
4      org_text.in_buffer_settings.each_pair do |key, value|
5        # We need true/false as booleans, not string.
6        if key.downcase == 'published' #(any others?)
7          value = value.to_b
8        end
9        data[key.downcase] = value
10      end
11      data
12    end

The implementation is not very elegant, but it registers all org-mode buffer settings as values. An exception must be coded for values which must be boolean. (I use only the published property, but there may be more)

##### Handling liquid tags

Here's where things get a bit hairy. Formally, when writing content which requires a liquid construct, it should be enclosed in the proper org-mode block delimiters, something like:

#+BEGIN_HTML
{%raw%}{% liquid construct here %}{%endraw%}
#+END_HTML


such that the whole document is a normal org-mode document. As my current documents do not have the 'BEGIN_HTML/END_HTML' delimiters this would be annoying to change (again). So, the first thing I tried is to implement it to avoid all that changing again. That worked, a fully working implementation is at commit cc2081

However, there are significant issues with this:

1. documents are still odd org-mode documents; no real improvement from the originals (just a bit less non-org-mode constructs)
2. build performance of jekyll is unacceptable in the implementation (roughly 3 times slower!!)
3. to solve the problems plugins need to be adapted, so it'll quickly become a nightmare.
4. the code already is more complex than it needs to be (include tags, quote replacement)

After realizing this, I started over and just coded the convert method as:

1  def convert(content)
2    org_text = Orgmode::Parser.new(content, {markup_file: "html.tags.yml" })
3    org_text.in_buffer_settings.delete("TITLE")  # We already captured the title on processing the options
4    org_text.to_html
5  end

Given I made the change to all my orgmode files as mentioned above, which is not that much work with emacs' dired-do-query-replace-regexp command, the conversion is complete. The only thing to make sure is that for each type of source file (post, page, document) the header options are processed with the process_options function above.

You can view the end result in commit 40dad2

• # Publishing your Emacs configuration with Jekyll

Share:

Many people in the Emacs community use orgmode. Quite a few of them use the org-babel system to write and maintain their emacs configuration, and so am I. I find the biggest advantage to be able to document my thought process as well as the configuration details. Emacs configuration can get pretty big and maintaining it becomes a lot easier for me if I can read back what I was thinking 6 months ago.

The system borrows from literate programming concepts where you write code and documentation in one document and let tools tangle the code and documentation into separate documents.

{%pullquote right %} The default way to publish an Emacs configuration with that system would be to let org-mode export the configuration document to html and publish that somewhere. What I wanted to do was to use the in-place org-mode converter I am using with Jekyll. In short, {"the ideal would be that jekyll fetches the authoritative version of my emacs configuration in org-mode syntax"} and treats that as normal content and publish it. {% endpullquote %}

I started off by defining a collection, a new feature of jekyll, which gave me a chance to use it. The collection is defined in the _config.yml file:

1  # Emacs collection contains emacs 'generated' documents
2  collections:
3    emacs:
4      # Render them, i.e. convert them to html.
5      output: true

This will render source files in the _emacs folder as if they were pages. The url generated for them will be prefixed by /emacs/.

In this folder I placed a file config.org with the following contents

1---
2layout: orgmode
3---
4
5[... text left out  ..]
6
7{% raw %}
8{% include extern/mrb.org %}
9{% endraw %}

That includes the file <root>/_includes/extern/mrb.org from the jekyll installation and renders its content. The only correction I had to make was to insert a raw...endraw block around some org-mode settings.

You can view the result of the rendering in /emacs/config/