Process DMARC reports with sieve

I get a lot of DMARC reports because I host mail for a couple of domains. Most of these mails require no attention as they are just notifications that others use one of our domains. I want to separate these mails from my normal mail workflow and auto archive them if I haven't looked at them within, say, 2 weeks.

Doing this with sieve server-side has my preference, but apparently it's not trivial to determine the age of a message, which is the core logic needed here. Also, the processing of sieve rules is normally only during reception of messages, not ad-hoc or on some other event, although dovecot and pigeonhole have some options for this, among others the sieve-filter tool.

I really only found one implemenation online which roughly solves the same problem I was having, but this involved more than needed I think.

My solution consists of 3 parts:

1. the sieve script that handles DMARC reports on reception and age-ing;
2. use of an extension that calls an external program to evaluate expressions to determine age;
3. a daily job that runs the sieve script in the scope of the designated folder.

Here's the sieve script which deals with DMARC reports both in the normal INBOX flow and a special treatment after 14 days. The latter part is not automatic by dovecot on reception of emails, but triggered by a run of the sieve-filter program.

 1    require ["date","fileinto","relational","variables","environment","imap4flags",
 2             "vnd.dovecot.execute", "vnd.dovecot.environment"];
 3
 4    # Parameters
 5    set "dmarc_folder" "Folder.for.dmarc-reports";
 6    set "purge_days" "14";
 7
 8    # Move DMARC notifications when received
 9    if environment :is "vnd.dovecot.default-mailbox" "INBOX" {
10      if anyof (
11        header :contains "From" "dmarcreport@microsoft.com",
12        header :contains "From" "noreply-dmarc-support@google.com",
13        header :contains "From" "opendmarc@mail.arctype.co",
14        header :contains "From" "opendmarc@box.euandre.org"  )
15      {
16        addflag "\\Seen";
17        fileinto "${dmarc_folder}";
18        stop;
19      }
20    }
21
22    # When running in the dmarc_folder, archive when age is <purge_days>
23    if environment :is "vnd.dovecot.default-mailbox" "${dmarc_folder}"
24    {
25      if currentdate :matches "julian" "*"
26      {
27        # Run a simple bc expresssion to get <purge_days> ago from todays julian day
28        execute :output "purge_date" "bc" "${1} - ${purge_days}";
29
30        # Compare this with Date header and archive when age reached
31        if date :value "le" "Date" "julian" "${purge_date}"
32        {
33          fileinto "Trash";
34          stop;
35        }
36      }
37    }

The first part of the sieve script just moves the mails into the dmarc-reports folder and is a normal sieve processing rule. The second part runs if the default folder is the dmarc-reports folder. If so, it uses the ext_program extension of the sieve interpreter to let the bc program evaluate the expression for the age of the message.

This uses a tiny script in the configured sieve execute bin directory of the ext_programs extension

1  #!/bin/sh
2  echo ${1} | /usr/bin/bc

which just pipes the input given by the sieve line into the bc program. On returning, stdout is put into the purge_date variable. I'm using execute because I do not need to pipe the whole message into the external program, but specify input specifically.

With the above configuration I can set a cron job in the crontab of the vmail user to run

1  sieve-filter -We -u <mymailaccount> \
2               /path/to/vmail/mymailaccount/sieve/dmarc-archiver.sieve \
3               Folder.for.dmarc-reports

which executes the sieve script mentioned above in the IMAP folder <dmarc_folder> only.

I'm not sure why sieve makes it so difficult to get the age of an email (unless I'm missing something). Protonmail solves this by having a custom extension 'vnd.proton.eval' which does something similar like the above, but in the scope of the sieve language itself without having to shell out to an external program explicitly. (I think; I have not seen their implementation)

My approach above obviously has some drawbacks:

• the bc external program is called for every mail that matches, fine for 10 or 20 I guess, but rather inefficient if the amount of matched messages is big. For now, not a problem.
• unsure what sort of security consequences this has, the execution scope and environment is very limited, but we're still giving control to a script calling other programs.

Sunset on dutch beach

Hike: Katwijk aan Zee - De Zilk LAW 05-2 (04)

This post is the result of a little hike along one of the Long Distance walks in the Netherlands: "LAW 05-2 (04) Katwijk aan Zee - de Zilk", part 4 of "Nederlands Kustpad"

I recorded the hike with my Suunto Traverse watch and exported this as a GPX file. I used that gpx file as example data to create a little gpx viewer which can be embeded in html pages. With Leaflet this is almost trivial to do.

The details of the hike are on my Movescount page for this move.

If you zoom in on the map, you'll see that the recorded path is not an exact match with the followed path (the dotted lines). This is because I used a long interval of GPS fixes to record the path. I think it was a 10 second interval, so there's some tuning to do for this in future.

The exported GPX has information about temperature, height, energy, speed and possibly some other stuff. My intention is to have this information incorporated in the map later on, similar to the way this is displayed on the movescount site.

If you're interested, the GPX file is here:

Move_2016_11_06_09_16_34_Hiking.gpx

Further work on jekyll org-mode support

In Working with Jekyll and Org-Mode a simple solution was given to use org-mode format files as posts.

That solution had quite a few limitations:

1. the files still needed the '---' yaml header to be present on the first lines, making it invalid org-mode documents
2. only posts were supported, not documents, collections and pages
3. the #+key: value syntax of org-mode was not used for settings, or at least only partially
4. working with liquid tags was cumbersome.

What follows are some notes on how I solved the issues.

1  module Utils
2    def has_yaml_header?(file)
3      !!((File.open(file, 'rb') { |f| f.read(2) } =~ /^#\+/) or
4         (File.open(file, 'rb') { |f| f.read(5) } =~ /\A---\r?\n/))
5    end
6  end

 1  def self.process_options(content, data)
 2      org_text = Orgmode::Parser.new(content, {markup_file: "html.tags.yml" })
 3
 4      org_text.in_buffer_settings.each_pair do |key, value|
 5        # We need true/false as booleans, not string.
 6        if key.downcase == 'published' #(any others?)
 7          value = value.to_b
 8        end
 9        data[key.downcase] = value
10      end
11      data
12    end

#+BEGIN_HTML
   {%raw%}{% liquid construct here %}{%endraw%}
#+END_HTML

1  def convert(content)
2    org_text = Orgmode::Parser.new(content, {markup_file: "html.tags.yml" })
3    org_text.in_buffer_settings.delete("TITLE")  # We already captured the title on processing the options
4    org_text.to_html
5  end

Publishing your Emacs configuration with Jekyll

Many people in the Emacs community use orgmode. Quite a few of them use the org-babel system to write and maintain their emacs configuration, and so am I. I find the biggest advantage to be able to document my thought process as well as the configuration details. Emacs configuration can get pretty big and maintaining it becomes a lot easier for me if I can read back what I was thinking 6 months ago.

The system borrows from literate programming concepts where you write code and documentation in one document and let tools tangle the code and documentation into separate documents.

I started off by defining a collection, a new feature of jekyll, which gave me a chance to use it. The collection is defined in the _config.yml file:

1  # Emacs collection contains emacs 'generated' documents
2  collections:
3    emacs:
4      # Render them, i.e. convert them to html. 
5      output: true

This will render source files in the _emacs folder as if they were pages. The url generated for them will be prefixed by /emacs/.

In this folder I placed a file config.org with the following contents

1---
2layout: orgmode
3--- 
4
5[... text left out  ..]
6
7{% raw %}
8{% include extern/mrb.org %}
9{% endraw %}

That includes the file <root>/_includes/extern/mrb.org from the jekyll installation and renders its content. The only correction I had to make was to insert a raw...endraw block around some org-mode settings.

You can view the result of the rendering in /emacs/config/