Chapter 5. An introduction to qpsmtpd

The core of our filtering service was the qpsmtpd, which implements little more than the core of an SMTP server. All the decisions and handling related to incoming email are delegated by qpsmtpd to external plugins.

To implement our service we developed a collection of custom plugins which read per-domain configuration settings and could be used to apply various tests to each incoming email. qpsmtpd makes the coding of new plugins an extremely straightforward process, allowing external plugin code to be executed at almost every stage of an SMTP dialog via a clever hook implementation.

A full introduction to qpsmtpd is outside the scope of this document, but installing a new plugin involves two steps:

Any plugins named in the /etc/qpsmtpd/plugins file are loaded when the daemon starts up and each plugin is executed in a chained fashion in the order in which they're mentioned in the file upon the receipt of incoming email. (For example there might be four plugins which examine the incoming client's HELO data, and ten plugins which look at the DATA submitted.)

Each plugin is called once, and once only, at the appropriate part of the incoming SMTP transaction, and the return code passed back to the qpsmtpd process may be used to skip the execution of further plugins. When a plugin is called it is given arguments that are related to the phase of the SMTP dialog it has been invoked at and almost all plugins are called with a transaction object which is the cornerstone of allowing communication between different plugins as they otherwise share no state.[2]

The transaction object has several properties and methods, but the one that is most useful is the notes method. The notes method may be called with one or two arguments like this:

$transaction->notes("test", "me")

Set the value of the property "test" to be "me" for this transaction.

$transaction->notes("test")

Return the data stored under the key "test" for this transaction.

Now that we've explained at a high-level how the qpsmptd plugin system works we can look at a working example plugin, albeit a very basic one:

Example 5-1. The most basic plugin


 =begin doc

 This function will be called after the remote host sends the command
 HELO foo.example.com

 =end doc

 =cut

 sub hook_helo
 {
    my ($self, $transaction, $host) = ( @_ );

    # skip if we've already rejected this connection
    return DECLINED if ( ( $transaction->notes("rejected" ) || 0 ) == 1 );

    if ( $helo eq "aol.com" )
    {
        $transaction->notes("rejected", 1 );
        $transaction->notes("plugin", plugin_name() );
        $transaction->notes("reason", "Forged AOL HELO" );
    }

    return DECLINED;
 }

 1;
 

This example code demonstrates most of the things that we've attempted to introduce:

1. The use of the naming convention

Because the plugin has a method named hook_helo qpsmtpd will ensure the code is called at the correct part of the SMTP dialog, which is just after the remote client has sent their HELO greeting command.

There are some exceptions, and extra methods, but for every command FOO available in an SMTP transaction you can ensure your plugin is called at that point by defining the method hook_foo.

2. The use of a transaction object

This plugin uses the transaction object to skip running if it finds a value already stored in the transaction object with a name of "rejected" and a value of 1.

If it determines that the incoming connection is from a SPAM sender (even America Online would never send a HELO parameter of "aol.com") it will store some notes of its own.

3. The use of a return code

This plugin returns with the magical value of DECLINED which instructs qpsmtpd that remaining plugins should continue to be invoked.[3]

5.1. Our testing framework

Although the qpsmtpd plugin framework allowed us to perform tests at every point in the SMTP dialog we had to receive each incoming email in its entirity so that we could archive it in the quarantine if it were SPAM. However we didn't want to waste resources performing tests that weren't required: If an email was judged to be SPAM at the HELO stage all the tests relating to the DATA section should be skipped.

The use of the transaction object was key to intra-plugin communication, we just needed to be consistent in each plugin, and that notion of consistency was what our testing framework was built around.

We limited our list of note keys to a very small and well-defined set:

Table 5-1. The transaction note names we used.

NameMeaning
domain

This note would contain the domain name of the email's recipient. As our tests were each per-domain it was useful to store this here for easy access.

rejected

This note would contain "1" if the mail had been judged to be SPAM by a prior plugin, and be empty otherwise.

reason

If a mail were rejected this human-readable explanation would be set.

plugin

For accounting purposes we'd store the name of the plugin which rejected each email in our logs.

The use of the rejected transaction note was the key to ensuring we didn't perform needless work - as suggested by the sample code already displayed.

The testing process we used for each individual plugin was thus very similar, regardless of the actual specific tests made. This is what we mean by a common framework - each plugin was based around a common method of operation, with only rare exceptions:

The following sample code demonstrates this approach, and is a slightly simplified version of the virus-scanning we performed against incoming email:

Example 5-2. The use of consistancy in our plugins.


 # /mf/plugins/av

 =begin doc

 This function will be called after the remote host sends DATA.

 =end doc

 =cut

 sub hook_data_post
 {
    my ($self, $transaction) = ( @_ );

    #
    # Get the recipient domain.
    #
    my $domain = $transaction->notes("domain");

    #
    # skip if we've already rejected this message
    #
    my $declined = $transaction->notes("rejected" ) || undef;
    return DECLINED unless defined( $declined );

    #
    # skip if virus scanning disabled for this domain
    #
    my $virus = MF::Domain::Virus->new( domain => $domain );
    return DECLINED unless( $virus->enabled() );

    #
    #  Scan.
    #
    my ( $infected, $vname ) = $virus->scanMsg( $self->get_body() );

    #
    # If infected record in the notes.
    #
    if ( $infected )
    {
        $transaction->notes("rejected", 1 );
        $transaction->notes("plugin", plugin_name() );
        $transaction->notes("reason", "Viral mail " . $vname );
    }

    # all done, let the rest of our DATA handlers proceed.
    return DECLINED;
 }

 1;
 

At the end of the SMTP transaction each incoming email would either be rejected as SPAM or passed to exim for final delivery (in either case the message would archived to local disk), the decision of which action to take rested soley on the presence or absence of the rejected transaction note.

Notes

[1]

It is possible to register hooks manually, such that subroutine names aren't used to decide when code is excluded, but our simplification is probably sufficient.

[2]

The exceptions are those hooks that are called before the SMTP dialog has been started, but these hooks are outside the scope of this chapter.

[3]

If, for some reason, the plugin decided it had done enough and that no further plugins should be called at HELO-time then it would instead return DONE.