Appendix I. Mail-Scanning-Lite

In order to demonstrate several of the techniques which have been described in this document we've put together a "mail-scanning-lite" package.

This package is completely new only borrowing ideas from our experience running the mail-scanning.com site. We have no intention to support it as a commercial venture, indeed the termination of our main service suggests that would be an unwise thing to do.

If you would like to use it you're entirely welcome to do so at your own risk. The authors will endeavour to answer questions but all support is discretionary and liable to be limited.

The package consists of a collection of qpsmtpd plugins which each work together to filter incoming email for an arbitrary number of domains hosted upon a single system. (There is no concept of running in a distributed fashion.)

As with the real service these plugins operate around a hierarchy of directories and files. In the real service these files, and their contents, could be manipulated via the online control panel, but in this simplified system you'll need to maintain them manually.

The reason these plugins work together in a cohesive fashion, processing and rejecting SPAM, is because they each follow a consistent approach. Each incoming message is handled from start to finish by multiple plugins, but plugins communicate with each other via a small list of fixed notes, which matches the way our real system worked as documented in Section 5.1.

From start to finish there are four phases which an incoming message will pass through:

1. Determining Sender/Recipient/Domain

The plugin global/test_recipient is responsible for responding to the "MAIL FROM" and "RCPT TO" phases of the SMTP dialog.

When the connecting client announces the recipient of the incoming mail the plugin can determine if that domain is hosted locally by looking for the directory /srv/$domain-name on disk. If the directory is present than the domain is hosted locally and the mail will be processed (the settings for the domain live beneath that directory).

If the mail is addressed to a host which isn't hosted locally then it is an attempt to relay mail and so it can be immediately rejected.

Using the transaction object the recipient domain will be associated with the incoming SMTP transaction. The recorded domain-name will be used by later plugins to determine which tests are enabled, disabled, or configured for that particular domain.

2. Running the antispam tests

Once the recipient verification has been completed succesfully each of the testing plugins will be invoked in turn to test whether the message is SPAM or not. Each plugin will ensure that it only runs if it needs to:

  • If a message has already been determined to be SPAM the plugins will skip all further processing.

  • If the test that the plugin implements is not enabled for the particular domain then the plugin will skip all further processing.

  • Otherwise the plugin will execute, and perform its tests. A SPAM result will be recorded in the transaction object.

As described above the structure of each plugin is identical, because each will attempt to ensure that the message hasn't already been rejected, and that the testing task it is to complete is actually enabled for the domain. For that reason each plugins differs only in the testing task it performs. Due to their simplicity the following two plugins are probably the simplest to understand and example:

3. Running the whitelist/blacklist.

The whitelist plugin is executed right at the end of the process and operates in a similar fashion to the testing plugins: If a message has not been judged to be SPAM it skips all work.

However if the message is currently listed as SPAM then the plugin will examine the per-domain list of whitelisted senders, recipients, and hostnames. If any of those attributes match the currently rejected message then the note recording that rejection will be removed.

4. Queueing for delivery or archiving as SPAM.

The final phase of the system to either archive the message as SPAM or accept it for delivery.

If the mail should be rejected it is archived to disk and a DENY result is returned to the sender, otherwise the message is merely piped to the local copy of exim for final delivery.

I.1. Obtaining The Code

You may checkout or view the code from the following github repository:

There is also some brief external documentation available on the capabilities and configuration of this software.