Mail gateway with amavisd-new

Documentation of a mail gateway with postfix, amavisd-new, clamav, spamassassin and SMTP authenticated against the internal IMAP, with the configuration files.

Translation of the original in Catalan. Read it in: CatalàCastellano

This page was on the oriol.joor.net and oriolrius.cat wiki (2006–2012). I recover it here as it was.

Comments, questions and requests

Anything you want to ask me, tell me, suggest, or simply complain about because it doesn't work for you, you can do it in the article mailgateway: postfix+amavisd+clamav+spamassassin+sasl

Changelog

Introduction and Goal

Some companies have mail servers with really complex internal infrastructures and, following the golden rule of computing, the best we can do is: if something works, don't touch it. The problem is that these complex structures often have no malicious mail filters. That is, they are not able to run antivirus on the mail, filter spam and control other malicious mail deliveries. Moreover, trying to add software inside the server itself to do these tasks on top of the Mail Transport Agent (MTA) we already have running is an unnecessary and often unprofitable risk.

A solution that is often used is to add a server only to filter the mail that comes in and goes out of the company and to try to remove with this mail-gateway all the unwanted mail. Basically then what we do is to place the SMTP (Simple Mail Transport Protocol) of the mail-gateway as the entry point for external mail. Then we modify the internal mail server so that all the mail it has to send outside goes through the mail-gateway. That is, we set the address of the gateway MTA as the fixed relay server of the internal server.

In this scenario users don't have to change anything at all, since we are only intervening on port 25/tcp, that is, SMTP. Moreover, on the internal mail server we can create an account for managing malicious mail, for example mailgateway@exemple.com (exemple.com being our domain). In this account we create two folders, one called spam and the other called ham. What we will do is synchronise the mail of these accounts with a folder on the mail-gateway (if the accounts are IMAP it is a good idea to synchronise them with offlineimap). Once we have these mails on the mail-gateway we will tell the bayessians filters of the antispam to learn what the malicious mail it is not able to filter looks like and what the mail it filtered, and that was legitimate, looks like.

This mail account (mailgateway@exemple.com) that we have created on the internal server we can set up as a secondary mail account of a network administrator. That way they will be able to manage the maintenance without having to touch the mail-gateway. If any user of the network wants to locate a mail that has been filtered the administrator will be able to do it from this account.

Moreover, for users who are outside the company, branches or similar, we may be interested in authenticating SMTP so that these users can use the mailgateway SMTP to send mail. So we will take advantage of the IMAP (Internet Message Access Protocol) service of the internal server to validate the authentication in SMTP. It sounds a bit strange but the method is quite simple. It is about taking advantage of the account, user and password that users have on the internal IMAP server; then the MUA is configured to use this data as information to authenticate to SMTP, and the MTA server, when it receives an authentication via SMTP, opens an IMAP connection against the internal server and tries the user and password that the client has sent to SMTP; if that is correct the MTA allows sending mail to the MUA with whatever origin and destination it wants.

Network diagram

Network diagram
Mail-gateway network diagram

Diagram of the services and mail handling

Diagram of the services

In this diagram you can see the 3 TCP ports that we will publish to filter the mail and the main configuration options of the services that publish the ports.

Mail flow

When an SMTP server, either from our network or from an external network, sends mail to the mail-gateway, logically it does so on port 25/TCP to the SMTP server, in our case postfix. This, as we can see in the diagram, forwards the mail to port 10024 of the localhost IP. Moreover, this port is only able to receive requests from localhost to avoid security problems. The port we are talking about, if we look at the diagram, you can see that it is being published by amavis.

amavis is the program in charge of supervising the antispam and antivirus filters; it is the one that will make the relevant calls to third-party programs and the one that will capture the errors and send the log, email and other messages to whoever it corresponds to. If no anomaly is detected in the mail, it will be forwarded to the port localhost:10025/TCP.

According to the diagram, port 10025/TCP is also being published by postfix. Specifically, what SMTP will do in this case is to look for who the mail is addressed to, whether a local destination (local_relay) or a remote destination (MX dns lookup).

Configuring the services

postfix

The SMTP server basically has two main configuration files. Which in our case are not especially long. So it will be relatively easy for us to know if something is not working.

File /etc/postfix/master.cf:

The content of the file is at the end of this page.

As you can check in this file, this is the one that explains what the mail flow will be; we tell postfix what the order will be in which it will try to process the mail and who will handle it at each moment.

File /etc/postfix/main.cf:

The content of the file is at the end of this page.

As we can see in this file, above all we apply basic filters to avoid having the mail relay open. We allow sending mail from and to wherever we want, as long as we are inside our own network. But SMTPs connecting from other networks have quite a few restrictions as we can see from the comment Anti-spam filters. The rest of the configuration is quite typical. We have to realise that if we don't want the mail-gateway to keep any mail we have to make its domain differ from the domain of the internal mail server. Then, for mail generated locally, if we want to override this domain we can use the myorigin variable.

From these configuration files two more files are produced, one where we say which are our local domains (those of our network, not those of the mailgateway) and the other where we say which is the server that must explicitly handle the mail for the destination domain, whatever the DNS says.

File /etc/postfix/relay_delays:

joor.net	OK
oriol.joor.net	OK

File /etc/postfix/transport:

local		smtp:IP_SERVIDOR_INTERN:25
oriol.joor.net 	smtp:IP_SERVIDOR_INTERN:25
joor.net	smtp:IP_SERVIDOR_INTERN:25
oriolrius.cat	smtp:IP_SERVIDOR_INTERN:25 

The local domain refers to the domain defined in the mydomain configuration variable. So no mail will ever be stored on the mailgateway.

amavisd

amavisd-new basically has one important configuration file. This file is extremely long and very hard to analyse. The previous diagram summarises its most important options. Nevertheless, to configure it properly you have to read it carefully and test everything you enable to check that the behaviour is the expected one.

configuration file (/etc/amavis/amavisd.conf)

Here we define things as important as deciding what to do when we receive a mail with a virus: should we notify the person who sent it to us? should we notify the recipient? the system administrator? and if the mail was spam? and if it was simply a badly formatted mail?

There are also other things defined such as: which port amavis publishes to receive the mail from postfix (127.0.0.1:10024/tcp). Which antivirus program we will use, clamav in our case. And spamassassin, how we will pass it the mail so that it checks whether the mail we are handling carries a virus.

Moreover, from here most of the spamassassin parameters are also defined, such as the whilelist, blacklist and spam_lovers list. Specifically we have forwarded them to external hash files: /etc/amavis/whitelist, /etc/amavis/blacklist and /etc/amavis/spam_lovers.

It is good advice to spend a good while playing with this file, since it is the one that will mark the most important behaviour of the mailgateway.

clamav-daemon or clamd

The antivirus configuration is quite simple; once the daemon is started it publishes a unix socket; if we run netstat -lpn|grep clamd we will be able to see exactly where the socket is, in our case:

# netstat -lpn|grep clam
unix  2      [ ACC ]     STREAM     LISTENING     644309   29823/clamd         /var/run/clamav/clamd.ctl

It is important to know where this unix socket is because the previous /etc/amavis/amavisd.conf file refers to it. Here we can see that we have it at /var/run/clamav/clamd.ctl.

After this we only have to start the clamav-freshclam daemon, which takes care of updating the virus definitions with the periodicity we tell it and from the repository that suits us best.

spamassassin

It is called from amavisd-new but no port is published. But we also have our own configuration file where we tell it where the data it needs to work is:

# This is the right place to customize your installation of SpamAssassin.
#
# See 'perldoc Mail::SpamAssassin::Conf' for details of what can be
# tweaked.
#
###########################################################################
#
# rewrite_header Subject *****SPAM*****
# report_safe 1
# trusted_networks 212.17.35.
# lock_method flock

report_safe 0

use_bayes 1
bayes_path /var/lib/amavis/.spamassassin/bayes

skip_rbl_checks 0
use_razor2 1
use_dcc 1
use_pyzor 0

dns_available yes

header LOCAL_RCVD Received =~ /.*\(\S+\.domain\.com\s+\[.*\]\)/
describe LOCAL_RCVD Received from local machine
score LOCAL_RCVD -50

## Optional Score Increases
score DCC_CHECK 4.000
score SPF_FAIL 10.000
score SPF_HELO_FAIL 10.000
score RAZOR2_CHECK 2.500
score BAYES_99 4.300
score BAYES_90 3.500
score BAYES_80 3.000

Testing the server

Sending a mail via SMTP through telnet/netcat

Sometimes we are interested in testing directly against the SMTP port and not having to reconfigure the mail client (MUA) every time. An example of how to do it:

 $ nc -vvv IP_SERVIDOR 25
(UNKNOWN) [IP_SERVIDOR] 25 (smtp) open
220 Mail Gateway
helo exemple.com
250 
mail from:<oriol@oriol.joor.net>
250 Ok
rcpt to:<oriol@oriol.joor.net>
250 Ok
data
354 End data with <CR><LF>.<CR><LF>
provant
.
250 Ok: queued as 0281C874FD
quit
221 Bye
 sent 101, rcvd 140

At this point the server is already routing the mail to its destination, since it has informed us that it has queued it.

Looking at the logs

To view the logs of the mail arriving at and leaving the server you have to look at the file /var/log/mail.log; if we want to view the file in real time we can use the command:

$ tail -f /var/log/mail.log

To interpret the log you only need a little patience and to pay close attention to who is the source of the entry. Whether the trace has been sent by postfix, amavis, etc. Then we will be able to see the details of what this application has done and we will be able to follow all the hops between applications described earlier in this document.

Reading the email headers

Being able to read the email headers allows us to know where a mail has been. Moreover, in our case it helps us to see the hops it has made inside the mailgateway; specifically, if we look at the email headers that have crossed our antispam server we can see something like:

Received: from localhost (localhost.localdomain [127.0.0.1])
	by mailgateway (Postfix) with ESMTP id 61D7E187508
	for <desti@exemple2.com>; Tue,  2 May 2006 14:58:35 +0200 (CEST)
Received: from mailgateway ([127.0.0.1])
	by localhost (mailgateway.subdomini.exemple1.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id 07387-13 for <desti@exemple2.com>;
	Tue, 2 May 2006 14:58:33 +0200 (CEST)
Received: from mini2 (unknown [IP_PUBLICA_PROVES])
	by mailgateway (Postfix) with ESMTP id 69D44187506
	for <desti@exemple2.com>; Tue,  2 May 2006 14:58:33 +0200 (CEST)

Obviously the mail has more headers but these always show up. Which are those of the filters we explained earlier. Moreover, inside the header TAGS we will also be able to see this:

X-Virus-Scanned: by amavisd-new-20030616-p10 at subdomini.exemple1.com

It is a watermark that amavisd-new leaves on the mails it examines.

To know how to interpret email headers there are a huge number of web pages that explain how to do it; below I attach a few:

Active services

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:10024         0.0.0.0:*               LISTEN     6353/amavisd (maste
tcp        0      0 127.0.0.1:10025         0.0.0.0:*               LISTEN     9371/master
tcp        0      0 127.0.0.1:783           0.0.0.0:*               LISTEN     6370/spamd.pid
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN     10176/named
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN     9371/master
tcp6       0      0 :::22                   :::*                    LISTEN     8444/sshd
udp        0      0 0.0.0.0:53              0.0.0.0:*                          10176/named
udp        0      0 127.0.0.1:53            0.0.0.0:*                          10176/named
udp6       0      0 :::39827                :::*                               10176/named
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     2598578  9371/master         private/smtp-amavis
unix  2      [ ACC ]     STREAM     LISTENING     2609624  10176/named         /var/run/ndc
unix  2      [ ACC ]     STREAM     LISTENING     2598589  9371/master         public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     2598596  9371/master         private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     2598600  9371/master         private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     2598604  9371/master         private/defer
unix  2      [ ACC ]     STREAM     LISTENING     2598608  9371/master         public/flush
unix  2      [ ACC ]     STREAM     LISTENING     2598612  9371/master         private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     2598616  9371/master         private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     2598620  9371/master         private/relay
unix  2      [ ACC ]     STREAM     LISTENING     2598624  9371/master         public/showq
unix  2      [ ACC ]     STREAM     LISTENING     644309   29823/clamd         /var/run/clamav/clamd.ctl
unix  2      [ ACC ]     STREAM     LISTENING     2598628  9371/master         private/error
unix  2      [ ACC ]     STREAM     LISTENING     2598632  9371/master         private/local
unix  2      [ ACC ]     STREAM     LISTENING     2598636  9371/master         private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     2598640  9371/master         private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     1623569  32625/acpid         /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     2598644  9371/master         private/maildrop
unix  2      [ ACC ]     STREAM     LISTENING     2598648  9371/master         private/trace
unix  2      [ ACC ]     STREAM     LISTENING     2598652  9371/master         private/verify
unix  2      [ ACC ]     STREAM     LISTENING     2598656  9371/master         private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     2598660  9371/master         private/anvil
unix  2      [ ACC ]     STREAM     LISTENING     2598664  9371/master         private/scache
unix  2      [ ACC ]     STREAM     LISTENING     2598668  9371/master         private/discard
unix  2      [ ACC ]     STREAM     LISTENING     2596380  8961/saslauthd      /var/run/saslauthd/mux

Managing the internal account (mailgateway@exemple.com)

Needs of the internal account

  • Create two folders in the account:
    • ham - we will put the mail flagged as a false positive
    • spam - we will put the mail that was spam and the server did not detect
  • Being able to query it over IMAP

We synchronise the IMAP account with the mailgateway

We will do this with offlineimap; we can run it as a user.

$ cat .offlineimaprc
[general]
accounts = servidor_intern

[Account servidor_intern]
localrepository = Local
remoterepository = Remote

[Repository Local]
type = Maildir
localfolders = ~/mailgateway
sep=/

[Repository Remote]
type = IMAP
remotehost = IP_SERVIDOR_INTERN
remoteuser = mailgateway
remotepass = ___

We can see that the email will be saved in the home (~) of the user we choose, specifically in the mailgateway folder.

The file is quite simple; you can see how we define the servidor_intern account and then we define the local and remote repository of this account. The local one will be a simple maildir and the remote one an imap.

Misc

Let's educate the anti-spam filter

spamassassin, the more example spam mails it has the better it is at making detections because these are done through bayessian functions. That means that the more data it has in the DB the more accurate it is. The command used to feed the spamassassin DB is sa-learn.

How to create a hash file (.db)

Often the configuration files of amavis and postfix are in hash format; to transform a normal configuration file into its hash version you only have to do the following:

postmap fitxer_de_configuració

and after this we will get, in addition to our configuration_file, another file called configuration_file.db, that is, the file in hash format.

References

  • IMAP protocol cookbook on how to use the IMAP protocol from a TCP session.

Configuration files

master.cf file

/etc/postfix/master.cf:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
#smtp      inet  n       -       n       -       -       smtpd 
smtp      inet  n       -       n       -       -       smtpd
	-o content_filter=smtp-amavis:[127.0.0.1]:10024
smtp-amavis unix        -       -       n       -       10       smtp
	-o smtp_data_done_timeout=1200
	-o disable_dns_lookups=yes
127.0.0.1:10025 inet    n       -       n       -       -       smtpd
	-o local_recipient_maps=
	-o smtpd_restriction_classes=
	-o smtpd_client_restrictions=
	-o smtpd_helo_restrictions=
	-o smtpd_sender_restrictions=
	-o smtpd_recipient_restrictions=permit_mynetworks,reject
	-o strict_rfc821_envelopes=yes
#smtps	  inet	n	-	n	-	-	smtpd
#  -o smtpd_tls_wrappermode=yes -o smtpd_sasl_auth_enable=yes
#submission	inet	n	-	n	-	-	smtpd
#  -o smtpd_enforce_tls=yes -o smtpd_sasl_auth_enable=yes
#628      inet  n       -       n       -       -       qmqpd
pickup    fifo  n       -       n       60      1       pickup 
cleanup   unix  n       -       n       -       0       cleanup 
qmgr      fifo  n       -       n       300     1       qmgr
#qmgr     fifo  n       -       n       300     1       nqmgr
#tlsmgr	  fifo	-	-	n	300	1	tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
smtp      unix  -       -       n       -       -       smtp 
relay     unix  -       -       n       -       -       smtp
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual 
lmtp      unix  -       -       n       -       -       lmtp
#
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# maildrop. See the Postfix MAILDROP_README file for details.
#
maildrop  unix  -       n       n       -       -       pipe
  flags=R user=vmail argv=/usr/bin/maildrop -d ${recipient}
#  flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#  flags=R user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# The Cyrus deliver program has changed incompatibly, multiple times.
#
#old-cyrus unix  -       n       n       -       -       pipe
#  flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
# Cyrus 2.1.5 (Amos Gouaux)
#cyrus     unix  -       n       n       -       -       pipe
#  user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
#uucp      unix  -       n       n       -       -       pipe
#  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#ifmail    unix  -       n       n       -       -       pipe
#  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
#bsmtp     unix  -       n       n       -       -       pipe
#  flags=Fq. user=foo argv=/usr/local/sbin/bsmtp -f $sender $nexthop $recipient
trace     unix  -       -       -       -       0       bounce
verify    unix  -       -       -       -       1       verify
tlsmgr    unix  -       -       -       1000?   1       tlsmgr
anvil     unix  -       -       -       -       1       anvil
scache    unix  -       -       -       -       1       scache
discard   unix  -       -       -       -       -       discard

main.cf file

/etc/postfix/main.cf:

mail_owner = postfix

# nom de la maquina local http://www.postfix.org/basic.html#myhostname
myhostname = srvgat
# domini local, posem un subdomini pq no es quedi els correus que volem fer-los relay
# http://www.postfix.org/basic.html#mydomain
mydomain = subdomini.exemple1.com
# domini dels correus que genera la maquina local http://www.postfix.org/basic.html#myorigin
myorigin = exemple1.com
# dominis que rep el servidor local http://www.postfix.org/basic.html#mydestination
mydestination= $myhostname, localhost.$mydomain, localhost, $mydomain
# xarxes locals http://www.postfix.org/basic.html#mynetworks
mynetworks = 172.16.0.0/16, 127.0.0.0/8, ********

biff = no
# Si queden menys de 120M de disc lliures no rebem emails
# http://www.postfix.org/postconf.5.html#queue_minfree
queue_minfree = 120000000
# Message max_size 20M
message_size_limit = 20480000

relayhost = mailserver.exemple1.com
relay_domains = hash:/etc/postfix/relay_domains
transport_maps = hash:/etc/postfix/transport

# http://www.postfix.org/postconf.5.html#notify_classes
#notify_classes = delay, policy, protocol, resource, software
notify_classes = delay
# http://www.postfix.org/postconf.5.html#disable_vrfy_command
disable_vrfy_command = yes
# http://www.postfix.org/postconf.5.html#smtpd_banner
smtpd_banner = ESMTP Mail Gateway
#

# Auth rIMAP
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain = 
broken_sasl_auth_clients = yes

#
# Restriccions anti-spam
#
# http://www.postfix.org/postconf.5.html#smtpd_delay_reject
smtpd_delay_reject = no
# http://www.postfix.org/postconf.5.html#smtpd_helo_required
smtpd_helo_required = yes
# http://www.postfix.org/postconf.5.html#strict_rfc821_envelopes
strict_rfc821_envelopes = yes
# http://www.postfix.org/postconf.5.html#smtpd_client_restrictions
smtpd_client_restrictions = 
	permit_mynetworks,
	permit_sasl_authenticated
#	reject_rbl_client opm.blited.org,
#	reject_rbl_client relays.orbdb.org,
#	reject_rbl_client sbl.spamhous.org
# http://www.postfix.org/postconf.5.html#smtpd_sender_restrictions
smtpd_sender_restrictions =
	permit_mynetworks,
        reject_unknown_sender_domain,
        reject_non_fqdn_sender,
	permit_sasl_authenticated 
smtpd_recipient_restrictions = 
        permit_mynetworks,
	permit_sasl_authenticated,
	reject_unauth_destination
# no l'uso per res però esta requerit
alias_maps = hash:/etc/aliases