DSPAM is a statistical spam filter. It is intended to be scalable, content-based spam filter for large multi-user systems. Documentation for dspam.

NOTE: This is the only component I’ve not found to NOT speak SSL over SMTP/LMTP. It does connect to PostgreSQL over SSL just fine.

Create/populate the database

First, we will need to create a database and then populate it. Creating is straight forward. Connect to Postgres and type in:

CREATE ROLE dspam WITH LOGIN ENCRYPTED PASSWORD 'password';
CREATE DATABASE dspam WITH OWNER = dspam;

Now its time to populate it. You can use the following to populate your dspam database:

psql -U dspam -W dspam < /usr/local/share/examples/dspam/pgsql/pgsql_objects.sql
psql -U dspam -W  dspam < /usr/local/share/examples/dspam/pgsql/virtual_users.sql

You’ll get some warnings about permissions for analyzing. These can safely be ignored. We will setup a daily script that will perform database maintenance including analyization.

If you get errors like this (This doesn’t happen right away, but it may happen after things have been running for days/weeks/months):

Dec 12 03:41:29 pgsql.cmhome postgres[92640]: [126-2] DETAIL:  Key (uid, token)=(28, -7501006800599240569) already exists.
Dec 12 03:41:29 pgsql.cmhome postgres[92640]: [126-3] STATEMENT:  PREPARE dspam_update_plan (bigint) AS UPDATE dspam_token_data SET last_hit=CURRENT_DATE,innocent_hits=innocent_hits+1 WHERE uid=28 AND token=$1;PREPARE dspam_insert_plan (bigint,int,int) AS INSERT INTO dspam_token_data (uid,token,spam_hits,innocent_hits,last_hit) VALUES (28,$1,$2,$3,CURRENT_DATE);

You can add the following to postgres. It won’t fix dspam, but it will work around the problem by telling postgres to attempt to do an update, and if the update fails, redo the same thing as an insert. It also avoids the round-trip time of sending the error back to the mail server, and having dspam issue another sql statement.

CREATE FUNCTION update_rec(TEXT,INTEGER,INTEGER) RETURNS VOID AS $$
BEGIN
UPDATE dspam.table SET f1=$2, f2=$3 WHERE k=$1;
IF NOT FOUND THEN
INSERT INTO dspam.table (k,f1,f2) VALUES ($1,$2,$3);
END IF;
END;
$$ LANGUAGE plpgsql;

Configuring DSPAM

Now we can edit the dspam.conf (/usr/local/etc). Most of it is default. Of note, I’ve told it to store preferences in the database. I’ve also added a long list of headers to ignore (we don’t want somebody setting a header that causes us to not scan something, or that sends confusing messages to the user. Here is what mine looks like:

## $Id: dspam.conf.in,v 1.103 2011/11/10 00:27:34 tomhendr Exp $
## dspam.conf -- DSPAM configuration file
##

#
# DSPAM Home: Specifies the base directory to be used for DSPAM storage
#
Home /var/db/dspam

#
# StorageDriver: Specifies the storage driver backend (library) to use.
# You'll only need to set this if you are using dynamic storage driver plugins
# from a binary distribution. The default build statically links the storage
# driver (when only one is specified at configure time), overriding this
# setting, which only comes into play if multiple storage drivers are specified
# at configure time. When using dynamic linking, be sure to include the path
# to the library if necessary, and some systems may use an extension other
# than .so (e.g. OSX uses .dylib).
#
StorageDriver /usr/local/lib/dspam/libpgsql_drv.so

# Trusted Delivery Agent: Specifies the local delivery agent DSPAM should call
# when delivering mail as a trusted user. Use %u to specify the user DSPAM is
# processing mail for. It is generally a good idea to allow the MTA to specify
# the pass-through arguments at run-time, but they may also be specified here.
#
TrustedDeliveryAgent "/usr/local/sbin/sendmail" # FreeBSD

# Untrusted Delivery Agent: Specifies the local delivery agent and arguments
# DSPAM should use when delivering mail and running in untrusted user mode.
# Because DSPAM will not allow pass-through arguments to be specified to
# untrusted users, all arguments should be specified here. Use %u to specify
# the user DSPAM is processing mail for. This configuration parameter is only
# necessary if you plan on allowing untrusted processing.
#
#UntrustedDeliveryAgent "/usr/bin/procmail -d %u"
#UntrustedDeliveryAgent "/usr/libexec/mail.local"

#
# SMTP or LMTP Delivery: Alternatively, you may wish to use SMTP or LMTP
# delivery to deliver your message to the mail server instead of using a
# delivery agent. You will need to configure with --enable-daemon to use host
# delivery, however you do not need to operate in daemon mode. Specify an IP
# address or UNIX path to a domain socket below as a host.
#
# If you would like to set up DeliveryHost's on a per-domain basis, use
# the syntax: DeliveryHost.example.org 1.2.3.4
#
DeliveryHost		127.0.0.1
DeliveryPort		24
DeliveryIdent		localhost
DeliveryProto		LMTP

# Quarantine Agent: DSPAM's default behavior is to quarantine all mail it
# thinks is spam. If you wish to override this behavior, you may specify
# a quarantine agent which will be called with all messages DSPAM thinks is
# spam. Use %u to specify the user DSPAM is processing mail for.
#
#QuarantineAgent	"/usr/bin/procmail -d spam"

# DSPAM can optionally process "plused users" (addresses in the user+detail
# form) by truncating the username just before the "+", so all internal
# processing occurs for "user", but delivery will be performed for
# "user+detail". This is only useful if the LDA can handle "plused users"
# (for example Cyrus IMAP) and when configured for LMTP delivery above
#
EnablePlusedDetail	on

# Character to use as seperator between user names and address extensions.
# If you change this value then please adjust QuarantineMailbox to use the
# new specified character. The default is '+'.
#
PlusedCharacter	+

# Turn this feature on if you want to force DSPAM to lowercase the "plused
# users" username.
#
PlusedUserLowercase	on

# Quarantine Mailbox: DSPAM's LMTP code can send spam mail using LMTP to a
# "plused" mailbox (such as user+quarantine) leaving quarantine processing
# for retraining or deletion to be performed by the LDA and the mail client.
# "plused" mailboxes are supported by Cyrus IMAP and possibly other LDAs. If
# you don't set/change PlusedCharacter then the mailbox name must have the +
# since the + is the default used character.
#
QuarantineMailbox	+Junk

# OnFail: What to do if local delivery or quarantine should fail. If set
# to "unlearn", DSPAM will unlearn the message prior to exiting with an
# un successful return code. The default option, "error" will not unlearn
# the message but return the appropriate error code. The unlearn option
# is useful on some systems where local delivery failures will cause the
# message to be requeued for delivery, and could result in the message
# being processed multiple times. During a very large failure, however,
# this could cause a significant load increase.
#
OnFail unlearn

# Trusted Users: Only the users specified below will be allowed to perform
# administrative functions in DSPAM such as setting the active user and
# accessing tools. All other users attempting to run DSPAM will be restricted;
# their uids will be forced to match the active username and they will not be
# able to specify delivery agent privileges or use tools.
#
Trust root
Trust dspam
Trust virtual
Trust dovenull
Trust mail
Trust daemon
Trust postfix

# Debugging: Enables debugging for some or all users. IMPORTANT: DSPAM must
# be compiled with debug support in order to use this option. DSPAM should
# never be running in production with debug active unless you are
# troubleshooting problems.
#
# DebugOpt: One or more of: process, classify, spam, fp, inoculation, corpus
#   process     standard message processing
#   classify    message classification using --classify
#   spam        error correction of missed spam
#   fp          error correction of false positives
#   inoculation message inoculations (source=inoculation)
#   corpus      corpusfed messages (source=corpus)
#
#Debug *
#DebugOpt process

# Training Mode: The default training mode to use for all operations, when
# one has not been specified on the commandline or in the user's preferences.
# Acceptable values are:
#     toe     Train on Error (Only)
#     teft    Train Everything (Trains on every message)
#     tum     Train Until Mature (Train only tokens without enough data)
#     notrain Do not train or store signatures (large ISP systems, post-train)
#
TrainingMode teft

# TestConditionalTraining: By default, dspam will retrain certain errors
# until the condition is no longer met. This usually accelerates learning.
# Some people argue that this can increase the risk of errors, however.
#
TestConditionalTraining on

# Features: Specify features to activate by default; can also be specified
# on the commandline. See the documentation for a list of available features.
# If _any_ features are specified on the commandline, these are ignored.
#
Feature noise
Feature whitelist

# Training Buffer: The training buffer waters down statistics during training.
# It is designed to prevent false positives, but can also dramatically reduce
# dspam's catch rate during initial training. This can be a number from 0
# (no buffering) to 10 (maximum buffering). If you are paranoid about false
# positives, you should probably enable this option.
#
#Feature tb=5

# Algorithms: Specify the statistical algorithms to use, overriding any
# defaults configured in the build. The options are:
#    naive       Naive-Bayesian (All Tokens)
#    graham      Graham-Bayesian ("A Plan for Spam")
#    burton      Burton-Bayesian (SpamProbe)
#    robinson    Robinson's Geometric Mean Test (Obsolete)
#    chi-square  Fisher-Robinson's Chi-Square Algorithm
#
# You may have multiple algorithms active simultaneously, but it is strongly
# recommended that you group Bayesian algorithms with other Bayesian
# algorithms, and any use of Chi-Square remain exclusive.
#
# NOTE: For standard "CRM114" Markovian weighting, use 'naive', or consider
#       using 'burton' for slightly better accuracy
#
# Don't mess with this unless you know what you're doing
#
Algorithm graham burton

# Tokenizer: Specify the tokenizer to use. The tokenizer is the piece
# responsible for parsing the message into individual tokens. Depending on
# how many resources you are willing to trade off vs. accuracy, you may
# choose to use a less or more detailed tokenizer:
#   word    uniGram (single word) tokenizer
#           Tokenizes message into single individual words/tokens
#           example: "free" and "viagra"
#   chain   biGram (chained tokens) tokenizer (default)
#           Single words + chains adjacent tokens together
#           example: "free" and "viagra" and "free viagra"
#   sbph    Sparse Binary Polynomial Hashing tokenizer
#           Creates sparse token patterns across sliding window of 5-tokens
#           example: "the quick * fox jumped" and "the * * fox jumped"
#   osb     Orthogonal Sparse biGram tokenizer
#           Similar to SBPH, but only uses the biGrams
#           example: "the * * fox" and "the * * * jumped"
#
# In general the reccomendation is to use 'osb' for new installations.
# The default value of 'chain' remains here as not to surprise anyone upgrading
# that has not changed from the default value.
#
Tokenizer osb

# PValue: Specify the technique used for calculating Probability Values,
# overriding any defaults configured in the build. These options are:
#    bcr         Bayesian Chain Rule (Graham's Technique - "A Plan for Spam")
#    robinson    Robinson's Technique (used in Chi-Square)
#    markov      Markovian Weighted Technique (for Markovian discrimination)
#
# Unlike the "Algorithms" property, you may only have one of these defined.
# Use of the chi-square algorithm automatically changes this to robinson.
#
# Don't mess with this unless you know what you're doing.
#
PValue bcr

#
# WebStats: Enable this if you are using the CGI, which writes .stats files
WebStats on

#
# ImprobabilityDrive: Calculate odds-ratios for ham/spam, and add to
# X-DSPAM-Improbability headers
#
ImprobabilityDrive on

#
# Preferences: Specify any preferences to set by default, unless otherwise
# overridden by the user (see next section) or a default.prefs file.
# If user or default.prefs are found, the user's preferences will override any
# defaults.
#
Preference "trainingMode= TOE"		# { TOE | TUM | TEFT | NOTRAIN } -> default:teft
Preference "spamAction=tag"	# { quarantine | tag | deliver } -> default:quarantine
Preference "spamSubject="		# { string } -> default:[SPAM]
Preference "statisticalSedation=5"	# { 0 - 10 } -> default:0
Preference "enableBNR=on"		# { on | off } -> default:off
Preference "enableWhitelist=on"		# { on | off } -> default:on
Preference "signatureLocation=headers"	# { message | headers } -> default:message
Preference "tagSpam=on"			# { on | off }
Preference "tagNonspam=on"		# { on | off }
Preference "showFactors=on"		# { on | off } -> default:off
Preference "optIn=off"			# { on | off }
Preference "optOut=off"			# { on | off }
Preference "whitelistThreshold=10"	# { Integer } -> default:10
Preference "makeCorpus=off"		# { on | off } -> default:off
Preference "storeFragments=off"		# { on | off } -> default:off
Preference "localStore="		# { on | off } -> default:username
Preference "processorBias=on"		# { on | off } -> default:on
Preference "fallbackDomain=off"		# { on | off } -> default:off
Preference "trainPristine=off"		# { on | off } -> default:off
Preference "optOutClamAV=off"		# { on | off } -> default:off
Preference "ignoreRBLLookups=off"	# { on | off } -> default:off
Preference "RBLInoculate=off"		# { on | off } -> default:off
Preference "notifications=off"		# { on | off } -> default:off

#
# Overrides: Specifies the user preferences which may override configuration
# and commandline defaults. Any other preferences supplied by an untrusted user
# will be ignored.
#
AllowOverride enableBNR
AllowOverride enableWhitelist
AllowOverride fallbackDomain
AllowOverride ignoreGroups
AllowOverride ignoreRBLLookups
AllowOverride localStore
AllowOverride makeCorpus
AllowOverride optIn
AllowOverride optOut
AllowOverride optOutClamAV
AllowOverride processorBias
AllowOverride RBLInoculate
AllowOverride showFactors
AllowOverride signatureLocation
AllowOverride spamAction
AllowOverride spamSubject
AllowOverride statisticalSedation
AllowOverride storeFragments
AllowOverride tagNonspam
AllowOverride tagSpam
AllowOverride trainPristine
AllowOverride trainingMode
AllowOverride whitelistThreshold
AllowOverride dailyQuarantineSummary
AllowOverride notifications

# --- PostgreSQL ---

# For PgSQLServer you can Use a TCP/IP address or a socket. If your socket is
# in /var/run/postgresql/.s.PGSQL.5432 specify just the path where the socket
# resits (without .s.PGSQL.5432).

PgSQLServer		pgsql.cmhome
PgSQLPort		5432
PgSQLUser		dspam
PgSQLPass		password
PgSQLDb			dspam

# If you're running DSPAM in client/server (daemon) mode, uncomment the
# setting below to override the default connection cache size (the number
# of connections the server pools between all clients).
#
PgSQLConnectionCache	2

# UIDInSignature: PgSQL supports the insertion of the user id into the DSPAM
# signature. This allows you to create one single spam or fp alias
# (pointing to some arbitrary user), and the uid in the signature will
# switch to the correct user. Result: you need only one spam alias

#PgSQLUIDInSignature	on

# If you're using vpopmail or some other type of virtual setup and wish to
# change the table dspam uses to perform username/uid lookups, you can over-
# ride it below

PgSQLVirtualTable		dspam_virtual_uids
PgSQLVirtualUIDField		uid
PgSQLVirtualUsernameField	username

Notifications	off

# TxtDirectory: the directory that holds the templates for notification
# messages (see Notifications) and tagging (see tagSpam/tagNonspam).
#
TxtDirectory /var/db/dspam/txt

# Purge configuration: Set dspam_clean purge default options, if not otherwise
# specified on the commandline
#
# Purge configuration for SQL-based installations using purge.sql
#
PurgeSignature	off	# Specified in purge.sql
PurgeNeutral	90
PurgeUnused	off	# Specified in purge.sql
PurgeHapaxes	off	# Specified in purge.sql
PurgeHits1S	off	# Specified in purge.sql
PurgeHits1I	off	# Specified in purge.sql

#
# Local Mail Exchangers: Used for source address tracking, tells DSPAM which
# mail exchangers are local and therefore should be ignored in the Received:
# header when tracking the source of an email. Note: you should use the address
# of the host as appears between brackets [ ] in the Received header.
# By default DSPAM is considering the following IPs always as LocalMX:
#	10.0.0.0/8	- Private IP addresses (RFC 1918)
#	127.0.0.0/8	- Localhost Loopback Address (RFC 1700)
#	169.254.0.0/16	- Zeroconf / APIPA (RFC 3330)
#	172.16.0.0/12	- Private IP addresses (RFC 1918)
#	192.168.0.0/16	- Private IP addresses (RFC 1918)
#
LocalMX 127.0.0.1 198.18.0.0/20

#
# Logging: Disabling logging for users will make usage graphs unavailable to
# them. Disabling system logging will make admin graphs unavailable.
#
SystemLog	on
UserLog		on

#
# TrainPristine: for systems where the original message remains server side
# and can therefore be presented in pristine format for retraining. This option
# will cause DSPAM to cease all writing of signatures and DSPAM headers to the
# message, and deliver the message in as pristine format as possible. This mode
# REQUIRES that the original message in its pristine format (as of delivery)
# be presented for retraining, as in the case of webmail, imap, or other
# applications where the message is actually kept server-side during reading,
# and is preserved. DO NOT use this switch unless the original message can be
# presented for retraining with the ORIGINAL HEADERS and NO MODIFICATIONS.
#
# NOTE: You can't use this setting with dspam_trian; if you're going to use it,
#       wait until after you train any corpora.
#
TrainPristine off

#
# Opt: in or out; determines DSPAM's default filtering behavior. If this value
# is set to in, users must opt-in to filtering by dropping a .dspam file in
# /var/dspam/opt-in/user.dspam (or if you have homedirs configured, a .dspam
# folder in their home directory).  The default is opt-out, which means all
# users will be filtered unless a .nodspam file is dropped in
# /var/dspam/opt-out/user.nodspam
#
Opt out

#
# TrackSources: specify which (if any) source addresses to track and report
# them to syslog (mail.info). This is useful if you're running a firewall or
# blacklist and would like to use this information. Spam reporting also drops
# RABL blacklist files (see http://www.nuclearelephant.com/projects/rabl/).
#
TrackSources spam nonspam virus

#
# ParseToHeaders: In lieu of setting up individual aliases for each user,
# DSPAM can be configured to automatically parse the To: address for spam and
# false positive forwards. From there, it can be configured to either set the
# DSPAM user based on the username specified in the header and/or change the
# training class and source accordingly. The options below can be used to
# customize most common types of header parsing behavior to avoid the need for
# multiple aliases, or if using LMTP, aliases entirely..
#
# ParseToHeader: Parse the To: headers of an incoming message. This must be
#                set to 'on' to use either of the following features.
#
# ChangeModeOnParse: Automatically change the class (to spam or innocent)
#   depending on whether spam- or notspam- was specified, and change the source
#   to 'error'. This is convenient if you're not using aliases at all, but
#   are delivering via LMTP.
#
# ChangeUserOnParse: Automatically change the username to match that specified
#   in the To: header. For example, spam-bob@example.org will set the username
#   to bob, ignoring any --user passed in. This may not always be desirable if
#   you are using virtual email addresses as usernames. Options:
#     on or user	take the portion before the @ sign only
#     full		take everything after the initial {spam,notspam}-.
#
ParseToHeaders on
ChangeModeOnParse off
ChangeUserOnParse full

#
# Broken MTA Options: Some MTAs don't support the proper functionality
# necessary. In these cases you can activate certain features in DSPAM to
# compensate. 'returnCodes' causes DSPAM to return an exit code of 99 if
# the message is spam, 0 if not, or a negative code if an error has occured.
# Specifying 'case' causes DSPAM to force the input usernames to lowercase.
# Specifying 'lineStripping' causes DSPAM to strip ^M's from messages passed
# in.
#
#Broken returnCodes
Broken case
Broken lineStripping

#
# MaxMessageSize: You may specify a maximum message size for DSPAM to process.
# If the message is larger than the maximum size, it will be delivered
# without processing. Value is in bytes.
#
MaxMessageSize 5000000

# --- ClamAV ---

#
# Virus Checking: If you are running clamd, DSPAM can perform stream-based
# virus checking using TCP. Uncomment the values below to enable virus
# checking.
#
# ClamAVResponse: reject (reject or drop the message with a permanent failure)
#                 accept (accept the message and quietly drop the message)
#                 spam   (treat as spam and quarantine/tag/whatever)
#
ClamAVPort		3310
ClamAVHost		127.0.0.1
ClamAVResponse		reject

# --- CLIENT / SERVER ---

#
# Daemonized Server: If you are running DSPAM as a daemonized server using
# --daemon, the following parameters will override the default. Use the
# ServerPass option to set up accounts for each client machine. The DSPAM
# server will process and deliver the message based on the parameters
# specified. If you want the client machine to perform delivery, use
# the --stdout option in conjunction with a local setup.
#
# ServerHost: Not enabling ServerHost will bind DSPAM server to all available
# interfaces.
#
ServerHost		127.0.0.1
ServerPort		2424
ServerQueueSize	32
ServerPID		/var/run/dspam/dspam.pid

#
# ServerMode specifies the type of LMTP server to start. This can be one of:
#     dspam: DSPAM-proprietary DLMTP server, for communicating with dspamc
#  standard: Standard LMTP server, for communicating with Postfix or other MTA
#      auto: Speak both DLMTP and LMTP; auto-detect by ServerPass.IDENT
#
ServerMode auto

# If supporting DLMTP (dspam) mode, dspam clients will require authentication
# as they will be passing in parameters. The idents below will be used to
# determine which clients will be speaking DLMTP, so if you will be using
# both LMTP and DLMTP from the same host, be sure to use something other
# than the server's hostname below (which will be sent by the MTA during a
# standard LMTP LHLO).
#
#ServerPass.Relay1	"secret"
ServerPass.client	"password"

# If supporting standard LMTP mode, server parameters will need to be specified
# here, as they will not be passed in by the mail server. The ServerIdent
# specifies the 250 response code ident sent back to connecting clients and
# should be set to the hostname of your server, or an alias.
#
# NOTE: If you specify --user in ServerParameters, the RCPT TO will be
#       used only for delivery, and not set as the active user for processing.
#
ServerParameters	"--deliver=innocent, spam -d %u"
ServerIdent		"mx.cryptomonkeys.com"

# If you wish to use a local domain socket instead of a TCP socket, uncomment
# the following. It is strongly recommended you use local domain sockets if
# you are running the client and server on the same machine, as it eliminates
# much of the bandwidth overhead.
#
#ServerDomainSocketPath	"/var/run/dspam/dspam.sock"

#
# Client Mode: If you are running DSPAM in client/server mode, uncomment and
# set these variables. A ClientHost beginning with a / will be treated as
# a domain socket.
#
#ClientHost	/var/run/dspam/dspam.sock
#ClientIdent	"secret@Relay1"
#
ClientHost	127.0.0.1
ClientPort	2424
ClientIdent	"password@client"

# ProcessorURLContext: By default, a URL context is generated for URLs, which
# records their tokens as separate from words found in documents. To use
# URL tokens in the same context as words, turn this feature off.
#
ProcessorURLContext on

# ProcessorBias: Bias causes the filter to lean more toward 'innocent', and
# usually greatly reduces false positives. It is the default behavior of
# most Bayesian filters (including dspam).
#
# NOTE: You probably DONT want this if you're using Markovian Weighting, unless
# you are paranoid about false positives.
#
ProcessorBias on

# StripRcptDomain: Cut the domain (including the at sign) from recipients.
# This is particularly useful if the recipient name is equal to real user
# accounts as recipients with domains tend to cause permission issues with
# dspam-web.
#
StripRcptDomain off

# GroupConfig: The configuration file for groups. See the README file
# for details on how to enable users to combine their training data to
# get better results.
GroupConfig /var/db/dspam/group


IgnoreHeader Accept-Language
IgnoreHeader Approved
IgnoreHeader Archive
IgnoreHeader Authentication-Results
IgnoreHeader Cache-Post-Path
IgnoreHeader Cancel-Key
IgnoreHeader Cancel-Lock
IgnoreHeader Complaints-To
IgnoreHeader Content-Description
IgnoreHeader Content-Disposition
IgnoreHeader Content-ID
IgnoreHeader Content-Language
IgnoreHeader Content-Return
IgnoreHeader Content-Transfer-Encoding
IgnoreHeader Content-Type
IgnoreHeader DKIM-Signature
IgnoreHeader Date
IgnoreHeader Disposition-Notification-To
IgnoreHeader DomainKey-Signature
IgnoreHeader Importance
IgnoreHeader In-Reply-To
IgnoreHeader Injection-Info
IgnoreHeader Lines
IgnoreHeader List-Archive
IgnoreHeader List-Help
IgnoreHeader List-Id
IgnoreHeader List-Post
IgnoreHeader List-Subscribe
IgnoreHeader List-Unsubscribe
IgnoreHeader Message-ID
IgnoreHeader Message-Id
IgnoreHeader NNTP-Posting-Date
IgnoreHeader NNTP-Posting-Host
IgnoreHeader Newsgroups
IgnoreHeader OpenPGP
IgnoreHeader Organization
IgnoreHeader Originator
IgnoreHeader PGP-ID
IgnoreHeader Path
IgnoreHeader Received
IgnoreHeader Received-SPF
IgnoreHeader References
IgnoreHeader Reply-To
IgnoreHeader Resent-Date
IgnoreHeader Resent-From
IgnoreHeader Resent-Message-ID
IgnoreHeader Thread-Index
IgnoreHeader Thread-Topic
IgnoreHeader User-Agent
IgnoreHeader X--MailScanner-SpamCheck
IgnoreHeader X-AV-Scanned
IgnoreHeader X-AVAS-Spam-Level
IgnoreHeader X-AVAS-Spam-Score
IgnoreHeader X-AVAS-Spam-Status
IgnoreHeader X-AVAS-Spam-Symbols
IgnoreHeader X-AVAS-Virus-Status
IgnoreHeader X-AVK-Virus-Check
IgnoreHeader X-Abuse
IgnoreHeader X-Abuse-Contact
IgnoreHeader X-Abuse-Info
IgnoreHeader X-Abuse-Management
IgnoreHeader X-Abuse-To
IgnoreHeader X-Abuse-and-DMCA-Info
IgnoreHeader X-Accept-Language
IgnoreHeader X-Admission-MailScanner-SpamCheck
IgnoreHeader X-Admission-MailScanner-SpamScore
IgnoreHeader X-Amavis-Alert
IgnoreHeader X-Amavis-Hold
IgnoreHeader X-Amavis-Modified
IgnoreHeader X-Amavis-OS-Fingerprint
IgnoreHeader X-Amavis-PenPals
IgnoreHeader X-Amavis-PolicyBank
IgnoreHeader X-AntiVirus
IgnoreHeader X-Antispam
IgnoreHeader X-Antivirus
IgnoreHeader X-Antivirus-Scanner
IgnoreHeader X-Antivirus-Status
IgnoreHeader X-Archive
IgnoreHeader X-Assp-Spam-Prob
IgnoreHeader X-Attention
IgnoreHeader X-BTI-AntiSpam
IgnoreHeader X-Barracuda
IgnoreHeader X-Barracuda-Bayes
IgnoreHeader X-Barracuda-Spam-Flag
IgnoreHeader X-Barracuda-Spam-Report
IgnoreHeader X-Barracuda-Spam-Score
IgnoreHeader X-Barracuda-Spam-Status
IgnoreHeader X-Barracuda-Virus-Scanned
IgnoreHeader X-BeenThere
IgnoreHeader X-Bogosity
IgnoreHeader X-Brightmail-Tracker
IgnoreHeader X-CRM114-CacheID
IgnoreHeader X-CRM114-Status
IgnoreHeader X-CRM114-Version
IgnoreHeader X-CTASD-IP
IgnoreHeader X-CTASD-RefID
IgnoreHeader X-CTASD-Sender
IgnoreHeader X-Cache
IgnoreHeader X-ClamAntiVirus-Scanner
IgnoreHeader X-Comment-To
IgnoreHeader X-Comments
IgnoreHeader X-Complaints
IgnoreHeader X-Complaints-Info
IgnoreHeader X-Complaints-To
IgnoreHeader X-DKIM
IgnoreHeader X-DMCA-Complaints-To
IgnoreHeader X-DMCA-Notifications
IgnoreHeader X-Despammed-Tracer
IgnoreHeader X-ELTE-SpamCheck
IgnoreHeader X-ELTE-SpamCheck-Details
IgnoreHeader X-ELTE-SpamScore
IgnoreHeader X-ELTE-SpamVersion
IgnoreHeader X-ELTE-VirusStatus
IgnoreHeader X-Enigmail-Supports
IgnoreHeader X-Enigmail-Version
IgnoreHeader X-Evolution-Source
IgnoreHeader X-Extra-Info
IgnoreHeader X-FSFE-MailScanner
IgnoreHeader X-FSFE-MailScanner-From
IgnoreHeader X-Face
IgnoreHeader X-Fellowship-MailScanner
IgnoreHeader X-Fellowship-MailScanner-From
IgnoreHeader X-Forwarded
IgnoreHeader X-GMX-Antispam
IgnoreHeader X-GMX-Antivirus
IgnoreHeader X-GPG-Fingerprint
IgnoreHeader X-GPG-Key-ID
IgnoreHeader X-GPS-DegDec
IgnoreHeader X-GPS-MGRS
IgnoreHeader X-GWSPAM
IgnoreHeader X-Gateway
IgnoreHeader X-Greylist
IgnoreHeader X-HTMLM
IgnoreHeader X-HTMLM-Info
IgnoreHeader X-HTMLM-Score
IgnoreHeader X-HTTP-Posting-Host
IgnoreHeader X-HTTP-UserAgent
IgnoreHeader X-HTTP-Via
IgnoreHeader X-Headers-End
IgnoreHeader X-ID
IgnoreHeader X-IMAIL-SPAM-STATISTICS
IgnoreHeader X-IMAIL-SPAM-URL-DBL
IgnoreHeader X-IMAIL-SPAM-VALFROM
IgnoreHeader X-IMAIL-SPAM-VALHELO
IgnoreHeader X-IMAIL-SPAM-VALREVDNS
IgnoreHeader X-Info
IgnoreHeader X-IronPort-Anti-Spam-Filtered
IgnoreHeader X-IronPort-Anti-Spam-Result
IgnoreHeader X-KSV-Antispam
IgnoreHeader X-Kaspersky-Antivirus
IgnoreHeader X-MDAV-Processed
IgnoreHeader X-MDRemoteIP
IgnoreHeader X-MDaemon-Deliver-To
IgnoreHeader X-MIE-MailScanner-SpamCheck
IgnoreHeader X-MIMEOLE
IgnoreHeader X-MIMETrack
IgnoreHeader X-MMS-Spam-Filter-ID
IgnoreHeader X-MS-Exchange-Forest-RulesExecuted
IgnoreHeader X-MS-Exchange-Organization-Antispam-Report
IgnoreHeader X-MS-Exchange-Organization-AuthAs
IgnoreHeader X-MS-Exchange-Organization-AuthDomain
IgnoreHeader X-MS-Exchange-Organization-AuthMechanism
IgnoreHeader X-MS-Exchange-Organization-AuthSource
IgnoreHeader X-MS-Exchange-Organization-Journal-Report
IgnoreHeader X-MS-Exchange-Organization-Original-Scl
IgnoreHeader X-MS-Exchange-Organization-Original-Sender
IgnoreHeader X-MS-Exchange-Organization-OriginalArrivalTime
IgnoreHeader X-MS-Exchange-Organization-OriginalSize
IgnoreHeader X-MS-Exchange-Organization-PCL
IgnoreHeader X-MS-Exchange-Organization-Quarantine
IgnoreHeader X-MS-Exchange-Organization-SCL
IgnoreHeader X-MS-Exchange-Organization-SenderIdResult
IgnoreHeader X-MS-Has-Attach
IgnoreHeader X-MS-TNEF-Correlator
IgnoreHeader X-MSMail-Priority
IgnoreHeader X-MailScanner
IgnoreHeader X-MailScanner-Information
IgnoreHeader X-MailScanner-SpamCheck
IgnoreHeader X-Mailer
IgnoreHeader X-Mailman-Version
IgnoreHeader X-Mlf-Spam-Status
IgnoreHeader X-NAI-Spam-Checker-Version
IgnoreHeader X-NAI-Spam-Flag
IgnoreHeader X-NAI-Spam-Level
IgnoreHeader X-NAI-Spam-Report
IgnoreHeader X-NAI-Spam-Route
IgnoreHeader X-NAI-Spam-Rules
IgnoreHeader X-NAI-Spam-Score
IgnoreHeader X-NAI-Spam-Threshold
IgnoreHeader X-NEWT-spamscore
IgnoreHeader X-NNTP-Posting-Date
IgnoreHeader X-NNTP-Posting-Host
IgnoreHeader X-NetcoreISpam1-ECMScanner
IgnoreHeader X-NetcoreISpam1-ECMScanner-From
IgnoreHeader X-NetcoreISpam1-ECMScanner-Information
IgnoreHeader X-NetcoreISpam1-ECMScanner-SpamCheck
IgnoreHeader X-NetcoreISpam1-ECMScanner-SpamScore
IgnoreHeader X-Newsreader
IgnoreHeader X-Newsserver
IgnoreHeader X-No-Archive
IgnoreHeader X-No-Spam
IgnoreHeader X-OSBF-Lua-Score
IgnoreHeader X-OWM-SpamCheck
IgnoreHeader X-OWM-VirusCheck
IgnoreHeader X-Olypen-Virus
IgnoreHeader X-Orig-Path
IgnoreHeader X-OriginalArrivalTime
IgnoreHeader X-Originating-IP
IgnoreHeader X-PAA-AntiVirus
IgnoreHeader X-PAA-AntiVirus-Message
IgnoreHeader X-PGP-Fingerprint
IgnoreHeader X-PGP-Hash
IgnoreHeader X-PGP-ID
IgnoreHeader X-PGP-Key
IgnoreHeader X-PGP-Key-Fingerprint
IgnoreHeader X-PGP-KeyID
IgnoreHeader X-PGP-Sig
IgnoreHeader X-PIRONET-NDH-MailScanner-SpamCheck
IgnoreHeader X-PIRONET-NDH-MailScanner-SpamScore
IgnoreHeader X-PMX
IgnoreHeader X-PMX-Version
IgnoreHeader X-PN-SPAMFiltered
IgnoreHeader X-Posting-Agent
IgnoreHeader X-Posting-ID
IgnoreHeader X-Posting-IP
IgnoreHeader X-Priority
IgnoreHeader X-Proofpoint-Spam-Details
IgnoreHeader X-Qmail-Scanner-1.25st
IgnoreHeader X-Quarantine-ID
IgnoreHeader X-RAV-AntiVirus
IgnoreHeader X-RITmySpam
IgnoreHeader X-RITmySpam-IP
IgnoreHeader X-RITmySpam-Spam
IgnoreHeader X-Rc-Spam
IgnoreHeader X-Rc-Virus
IgnoreHeader X-Received-Date
IgnoreHeader X-RedHat-Spam-Score
IgnoreHeader X-RedHat-Spam-Warning
IgnoreHeader X-RegEx
IgnoreHeader X-RegEx-Score
IgnoreHeader X-Rocket-Spam
IgnoreHeader X-SA-GROUP
IgnoreHeader X-SA-RECEIPTSTATUS
IgnoreHeader X-STA-NotSpam
IgnoreHeader X-STA-Spam
IgnoreHeader X-Scam-grey
IgnoreHeader X-Scanned-By
IgnoreHeader X-Sender
IgnoreHeader X-SenderID
IgnoreHeader X-Sohu-Antivirus
IgnoreHeader X-Spam
IgnoreHeader X-Spam-ASN
IgnoreHeader X-Spam-Check
IgnoreHeader X-Spam-Checked-By
IgnoreHeader X-Spam-Checker
IgnoreHeader X-Spam-Checker-Version
IgnoreHeader X-Spam-Clean
IgnoreHeader X-Spam-DCC
IgnoreHeader X-Spam-Details
IgnoreHeader X-Spam-Filter
IgnoreHeader X-Spam-Filtered
IgnoreHeader X-Spam-Flag
IgnoreHeader X-Spam-Level
IgnoreHeader X-Spam-OrigSender
IgnoreHeader X-Spam-Pct
IgnoreHeader X-Spam-Prev-Subject
IgnoreHeader X-Spam-Processed
IgnoreHeader X-Spam-Pyzor
IgnoreHeader X-Spam-Rating
IgnoreHeader X-Spam-Report
IgnoreHeader X-Spam-Scanned
IgnoreHeader X-Spam-Score
IgnoreHeader X-Spam-Status
IgnoreHeader X-Spam-Tagged
IgnoreHeader X-Spam-Tests
IgnoreHeader X-Spam-Tests-Failed
IgnoreHeader X-Spam-Virus
IgnoreHeader X-Spam-Warning
IgnoreHeader X-Spam-detection-level
IgnoreHeader X-SpamAssassin-Clean
IgnoreHeader X-SpamAssassin-Warning
IgnoreHeader X-SpamBouncer
IgnoreHeader X-SpamCatcher-Score
IgnoreHeader X-SpamCop-Checked
IgnoreHeader X-SpamCop-Disposition
IgnoreHeader X-SpamCop-Whitelisted
IgnoreHeader X-SpamDetected
IgnoreHeader X-SpamInfo
IgnoreHeader X-SpamPal
IgnoreHeader X-SpamPal-Timeout
IgnoreHeader X-SpamReason
IgnoreHeader X-SpamScore
IgnoreHeader X-SpamTest-Categories
IgnoreHeader X-SpamTest-Info
IgnoreHeader X-SpamTest-Method
IgnoreHeader X-SpamTest-Status
IgnoreHeader X-SpamTest-Version
IgnoreHeader X-Spamadvice
IgnoreHeader X-Spamarrest-noauth
IgnoreHeader X-Spamarrest-speedcode
IgnoreHeader X-Spambayes-Classification
IgnoreHeader X-Spamcount
IgnoreHeader X-Spamsensitivity
IgnoreHeader X-TERRACE-SPAMMARK
IgnoreHeader X-TERRACE-SPAMRATE
IgnoreHeader X-TM-AS-Category-Info
IgnoreHeader X-TM-AS-MatchedID
IgnoreHeader X-TM-AS-Product-Ver
IgnoreHeader X-TM-AS-Result
IgnoreHeader X-TMWD-Spam-Summary
IgnoreHeader X-TNEFEvaluated
IgnoreHeader X-Text-Classification
IgnoreHeader X-Text-Classification-Data
IgnoreHeader X-Trace
IgnoreHeader X-UCD-Spam-Score
IgnoreHeader X-User-Agent
IgnoreHeader X-User-ID
IgnoreHeader X-User-System
IgnoreHeader X-Virus-Check
IgnoreHeader X-Virus-Checked
IgnoreHeader X-Virus-Checker-Version
IgnoreHeader X-Virus-Scan
IgnoreHeader X-Virus-Scanned
IgnoreHeader X-Virus-Scanner
IgnoreHeader X-Virus-Scanner-Result
IgnoreHeader X-Virus-Status
IgnoreHeader X-VirusChecked
IgnoreHeader X-Virusscan
IgnoreHeader X-WSS-ID
IgnoreHeader X-WinProxy-AntiVirus
IgnoreHeader X-WinProxy-AntiVirus-Message
IgnoreHeader X-Yandex-Forward
IgnoreHeader X-Yandex-Front
IgnoreHeader X-Yandex-Spam
IgnoreHeader X-Yandex-TimeMark
IgnoreHeader X-cid
IgnoreHeader X-iHateSpam-Checked
IgnoreHeader X-iHateSpam-Quarantined
IgnoreHeader X-policyd-weight
IgnoreHeader X-purgate
IgnoreHeader X-purgate-Ad
IgnoreHeader X-purgate-ID
IgnoreHeader X-sgxh1
IgnoreHeader X-to-viruscore
IgnoreHeader Xref
IgnoreHeader acceptlanguage
IgnoreHeader thread-index
IgnoreHeader x-uscspam

## EOF

Starting the service

Now we can start dspam(1) with:

sudo service dspam start

You can check that its running with ps(1) (but don’t show all the PostgreSQL connections):

[louisk@mx louisk 101 ]$ ps ax | grep dspam | grep -v postgres
 742 v0- S    0:00.03 /usr/local/bin/dspam --daemon
[louisk@mx louisk 102 ]$

You can also use sockstat(1) to show which sockets dspam is listening on:

[louisk@mx louisk 102 ]$ sockstat -4l | grep dspam
root     dspam      742   6  tcp4   127.0.0.1:10024       *:*
[louisk@mx louisk 103 ]$

Training

Periodically, you will want to clean up the dspam database. Helpfully, there is a purge.sql script in ‘/usr/local/share/examples/dspam/pgsql’. I put a wrapper around it and drove it from cron:

@daily                                  root    /usr/local/bin/dspam-cleanup.sh
cat /usr/local/bin/dspam-cleanup.sh
#!/bin/sh
#
# run sql queries to clean old/unused tokens out of PostgreSQL
/usr/local/bin/psql -U pgsql dspam < /usr/local/etc/dspam-purge.sql
# clean old logs
/usr/local/bin/dspam_logrotate -a 30 -d /var/db/dspam/data

Testing

If you need to download some spam (do you want to do this very often?), you can get some here.

cat /path/to/mail_corpus_file | dspamc --client --user user@dom.tld --class=spam --source=corpus --deliver=summary
X-DSPAM-Result: user@dom.tld; result="Spam"; class="Spam"; probability=1.0000; confidence=1.00; signature=N/A

If you ask dspam to classify this message again, it may still consider it “innocent”, but now it won’t be as confident. Compare the confidence number to the output from before training the message:

cat /path/to/mail_corpus_file | dspam --user user@dom.tld --classify
X-DSPAM-Result: user@dom.tld; result="Innocent"; class="Innocent"; probability=0.7127; confidence=0.75; signature=N/A

If you need to enable debugging (dspam must be built with debug option), you can do

sudo sysrc dspam_debug=YES

Logs are in /var/log/dspam. Debug knobs are in dspam.conf.

Configuring Dovecot

Dovecot is a secure and scalable POP/IMAP server. It supports the relevant internet standards, offers storage via maildir, and is quite responsive under load. Dovecot’s job is to accept mail from Postfix, and cause it to be available when users connect via POP or IMAP. Documentation for Dovecot.

I’m only providing the output from ‘doveconf -n’. You can either put these directives into a dovecot.conf file, or you can put them into the more modern conf.d/*.conf files. The outcome is the same, and ‘doveconf -n’ output should also be the same. Should you wish to compare, you can put the bits into the conf.d/*.conf files, copy them all into a dovecot.conf file, start dovecot, and do a:

diff -u `doveconf -n` dovecot.conf

If things are correct, you should get a prompt back and no differences listed.

Here is the output from ‘doveconf -n’:

# 2.2.27 (c0f36b0): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: FreeBSD 11.0-RELEASE-p2 amd64
auth_socket_path = /var/run/dovecot/auth-userdb
debug_log_path = /var/log/dovecot-debug.log
first_valid_gid = 125
first_valid_uid = 125
hostname = mail.cryptomonkeys.org
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep
last_valid_gid = 125
last_valid_uid = 125
lmtp_save_to_detail_mailbox = yes
mail_gid = 125
mail_home = /usr/local/virtual/%d/%n/
mail_location = maildir:~/Maildir
mail_privileged_group = postfix
mail_uid = 125
mailbox_list_index = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart
namespace inbox {
  inbox = yes
  location =
  mailbox All {
    auto = subscribe
    special_use = \All
  }
  mailbox Archive {
    auto = subscribe
    special_use = \Archive
  }
  mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Flagged {
    auto = subscribe
    special_use = \Flagged
  }
  mailbox Junk {
    auto = create
    autoexpunge = 30 days
    special_use = \Junk
  }
  mailbox Notes {
    auto = subscribe
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix =
}
passdb {
  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  home = /usr/local/virtual/%d/%n/
  mailbox_alias_new = Sent Messages
  mailbox_alias_new2 = Sent Items
  mailbox_alias_new3 = Deleted Items
  mailbox_alias_old = Sent
  mailbox_alias_old2 = Sent
  mailbox_alias_old3 = Trash
  mailbox_alias_old4 = Deleted Messages
  sieve = /usr/local/virtual/%d/%n/dovecot.sieve
  sieve_dir = /usr/local/virtual/%d/%n/
  sieve_global_dir = /usr/local/virtual/sieve
  sieve_global_path = /usr/local/virtual/sieve/globalfilter.sieve
  stats_refresh = 30 secs
  stats_track_cmds = yes
}
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_enable_last = yes
postmaster_address = postmaster@cryptomonkeys.org
protocols = imap pop3 lmtp sieve
quota_full_tempfail = yes
sendmail_path = /usr/local/sbin/sendmail
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener auth-userdb {
    group = postfix
    mode = 0777
    user = postfix
  }
}
service imap-login {
  client_limit = 10
  inet_listener imap {
    port = 0
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_min_avail = 4
  service_count = 1
  vsz_limit = 256 M
}
service lmtp {
  inet_listener lmtp {
    address = 127.0.0.1 ::1
    port = 24
  }
  user = postfix
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
  process_min_avail = 0
  service_count = 1
  vsz_limit = 64 M
}
service managesieve {
  process_limit = 1024
}
service pop3-login {
  client_limit = 10
  inet_listener pop3 {
    port = 0
  }
  inet_listener pop3s {
    port = 995
    ssl = yes
  }
  process_limit = 128
  process_min_avail = 2
  service_count = 1
  vsz_limit = 256 M
}
service stats {
  fifo_listener stats-mail {
    mode = 0600
    user = postfix
  }
}
ssl_ca = </usr/local/etc/ssl/cacert.pem
ssl_cert = </usr/local/etc/ssl/server.crt
ssl_cipher_list = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4
ssl_dh_parameters_length = 2048
ssl_key =  # hidden, use -P to show it
ssl_parameters_regenerate = 1 weeks
ssl_prefer_server_ciphers = yes
ssl_protocols = !SSLv2 !SSLv3
userdb {
  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
protocol lmtp {
  mail_plugins = " sieve"
}
protocol lda {
  mail_plugins = " sieve"
}
protocol imap {
  mail_max_userip_connections = 20
  mail_plugins = " stats expire mail_log notify quota trash imap_quota"
}
protocol pop3 {
  mail_max_userip_connections = 10
}

Tightening things up a little

I’ve tightened security a little from the defaults. I’ve removed the SSLv2, and SSLv3 capabilities completely. I’ve Limited the ciphers to only those on the HIGH list. If you want to know more, check out the openssl docs on ciphers. Additionally, I added a dhparam with a length of 2048 bits. Dovecot generates this key the first time you start it after adding this directive, so you don’t have to worry about having a dhparam that is shared with anybody else. Probability is very low.

# This file is commonly accessed via passdb {} or userdb {} section in
# conf.d/auth-sql.conf.ext

driver = pgsql
connect = host=localhost dbname=postfix user=postfix password=password
default_pass_scheme = MD5
user_query = \
    SELECT '/usr/local/virtual/'||maildir AS home, '*:bytes='||quota AS quota_rule \
    FROM mailbox WHERE username = '%u' AND active = TRUE
password_query = \
    SELECT '/usr/local/virtual/'||maildir AS userdb_home, \
        username AS user, password, '*:bytes='||quota AS userdb_quota_rule \
    FROM mailbox WHERE username = '%u' AND active = TRUE
iterate_query = SELECT username AS user FROM mailbox

Start up dovecot

With all this in place, we should be able to add the

sudo sysrc dovecot_enable=YES

entry to /etc/rc.conf, and then do:

sudo service dovecot start

We can check our success with ps(1):

[root@mx etc 122 ]$ ps ax | grep dove
 3641  -  Is     0:00.03 /usr/local/sbin/dovecot -c /usr/local/etc/dovecot/dovecot.conf
 3642  -  I      0:00.02 dovecot/pop3-login
 3643  -  I      0:00.01 dovecot/imap-login
 3644  -  I      0:00.00 dovecot/anvil
 3645  -  I      0:00.01 dovecot/log
 3647  -  I      0:00.03 dovecot/pop3-login
 3648  -  I      0:00.01 dovecot/imap-login
 3649  -  I      0:00.09 dovecot/imap-login
 3650  -  I      0:00.00 dovecot/imap-login
 3651  -  I      0:00.08 dovecot/config
 3653  -  I      0:00.02 dovecot/auth
 3657  -  I      0:00.00 dovecot/ssl-params
[root@mx etc 123 ]$

and with sockstat(1) (both IPv4 and IPv6, if you don’t want to check IPv6, omit the ‘6’):

[root@mx dovecot 120 ]$ sockstat -46l | grep dove
dovenull imap-login 2733  7  tcp4   *:993                 *:*
dovenull imap-login 2733  8  tcp6   *:993                 *:*
dovenull imap-login 2732  7  tcp4   *:993                 *:*
dovenull imap-login 2732  8  tcp6   *:993                 *:*
dovenull imap-login 2731  7  tcp4   *:993                 *:*
dovenull imap-login 2731  8  tcp6   *:993                 *:*
dovenull pop3-login 2729  7  tcp4   *:995                 *:*
dovenull pop3-login 2729  8  tcp6   *:995                 *:*
dovenull imap-login 2725  7  tcp4   *:993                 *:*
dovenull imap-login 2725  8  tcp6   *:993                 *:*
dovenull pop3-login 2724  7  tcp4   *:995                 *:*
dovenull pop3-login 2724  8  tcp6   *:995                 *:*
root     dovecot    2723  15 tcp4   *:4190                *:*
root     dovecot    2723  16 tcp6   *:4190                *:*
root     dovecot    2723  26 tcp4   *:995                 *:*
root     dovecot    2723  27 tcp6   *:995                 *:*
root     dovecot    2723  40 tcp4   *:993                 *:*
root     dovecot    2723  41 tcp6   *:993                 *:*
[root@mx dovecot 121 ]$

Configuring Postfix

Postfix is a popular and secure SMTP server. Its extensible, flexible, and light on resources. Many people like it because its config file isn’t written in M4. Documentation for Postfix here.

Originally I’d used v2 because it appeared that some of the patches necessary weren’t available for v3, but it turns out those patches are not necessary with this architecture, so yey!

Enable postfix (and disable sendmail) with sysrc.

sudo sysrc postfix_enable=YES
sudo sysrc sendmail_enable=NONE

Disable the sendmail periodic scripts. If /etc/periodic.conf doesn’t exist, create it and add the following contents:

daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_include_submit_mailq="NO"
daily_submit_queuerun="NO"

Most of the config directives will go in the main.cf (/usr/local/etc/postfix). When you dump the config with postconf, those are the settings it looks at.

Here is the output from ‘postconf -n’ (it can be dumped straight into main.cf if you wish):

alias_maps = hash:${config_directory}/aliases
allow_mail_to_commands = alias
body_checks = pcre:${config_directory}/body_checks
broken_sasl_auth_clients = yes
command_directory = /usr/local/sbin
compatibility_level = 2
config_directory = /usr/local/etc/postfix
daemon_directory = /usr/local/libexec/postfix
data_directory = /var/db/postfix
delay_warning_time = 5d
disable_vrfy_command = yes
header_checks = pcre:${config_directory}/header_checks
html_directory = /usr/local/share/doc/postfix
inet_protocols = ipv4, ipv6
lmtp_header_checks = pcre:${config_directory}/lmtp_header_checks
mail_owner = postfix
mailbox_size_limit = 536870912
mailq_path = /usr/local/bin/mailq
manpage_directory = /usr/local/man
maximal_queue_lifetime = 8d
message_size_limit = 33554432
mydestination = $myorigin, localhost.$mydomain, localhost
mydomain = cryptomonkeys.com
myhostname = mail.cryptomonkeys.com
mynetworks = 192.168.0.0/20, 127.0.0.0/8, [::1]/128, [fe80::]/10
myorigin = $myhostname
nested_header_checks =
newaliases_path = /usr/local/bin/newaliases
postscreen_access_list = permit_mynetworks, cidr:$config_directory/postscreen_access.cidr
postscreen_bare_newline_action = ignore
postscreen_bare_newline_enable = yes
postscreen_bare_newline_ttl = 30d
postscreen_blacklist_action = drop
postscreen_cache_cleanup_interval = 12h
postscreen_cache_map = btree:$data_directory/postscreen_cache
postscreen_cache_retention_time = 7d
postscreen_disable_vrfy_command = $disable_vrfy_command
postscreen_dnsbl_action = enforce
postscreen_dnsbl_reply_map = pcre:$config_directory/postscreen_dnsbl_reply_map.pcre
postscreen_dnsbl_sites = zen.spamhaus.org*3 b.barracudacentral.org*2 bl.spameatingmonkey.net*2 bl.spamcop.net dnsbl.sorbs.net psbl.surriel.com bl.mailspike.net swl.spamhaus.org*-4 list.dnswl.org=127.[0..255].[0..255].0*-2 list.dnswl.org=127.[0..255].[0..255].1*-3 list.dnswl.org=127.[0..255].[0..255].[2..255]*-4
postscreen_dnsbl_threshold = 3
postscreen_dnsbl_ttl = 1h
postscreen_dnsbl_whitelist_threshold = -1
postscreen_greet_action = enforce
postscreen_greet_banner = $smtpd_banner
postscreen_greet_ttl = 1d
postscreen_greet_wait = ${stress?2}${stress:6}s
postscreen_helo_required = $smtpd_helo_required
postscreen_non_smtp_command_action = drop
postscreen_non_smtp_command_enable = yes
postscreen_non_smtp_command_ttl = 30d
postscreen_pipelining_enable = yes
postscreen_whitelist_interfaces = static:all
proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $virtual_mailbox_limit_maps
proxy_write_maps = $smtp_sasl_auth_cache_name $lmtp_sasl_auth_cache_name $address_verify_map $postscreen_cache_map
queue_directory = /var/spool/postfix
readme_directory = /usr/local/share/doc/postfix
recipient_delimiter = +
relay_domains = proxy:pgsql:${config_directory}/pgsql_relay_domains_maps.cf
sample_directory = /usr/local/etc/postfix
sendmail_path = /usr/local/sbin/sendmail
setgid_group = maildrop
smtp_destination_concurrency_limit = 20
smtp_header_checks = pcre:${config_directory}/smtp_header_checks
smtp_helo_timeout = 30s
smtp_sasl_auth_enable = no
smtp_tls_loglevel = 1
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3
smtp_tls_note_starttls_offer = yes
smtp_tls_policy_maps = hash:${config_directory}/tls_policy
smtp_tls_protocols = !SSLv2, !SSLv3
smtp_tls_security_level = may
smtp_use_tls = yes
smtpd_banner = $myhostname ESMTP Sendmail 8.10 (Solaris 2.6)
smtpd_client_connection_count_limit = 5
smtpd_client_connection_rate_limit = 10
smtpd_data_restrictions = reject_unauth_pipelining, reject_multi_recipient_bounce, permit
smtpd_end_of_data_restrictions = check_policy_service inet:127.0.0.1:10031
smtpd_error_sleep_time = 0
smtpd_hard_error_limit = 10
smtpd_helo_required = yes
smtpd_recipient_limit = 50
smtpd_recipient_restrictions = permit_dnswl_client list.dnswl.org=127.0.[0..255].[1..3], permit_sasl_authenticated, permit_mynetworks, permit_auth_destination, check_policy_service inet:127.0.0.1:10031, check_client_access hash:${config_directory}/access, check_recipient_access hash:${config_directory}/recipient_access, check_recipient_access hash:${config_directory}/sender_access, warn_if_reject reject_unknown_recipient_domain, warn_if_reject reject_non_fqdn_recipient, warn_if_reject reject_unknown_client, warn_if_reject reject_non_fqdn_sender, warn_if_reject reject_non_fqdn_hostname, reject_unverified_recipient, reject_invalid_hostname, reject_unknown_recipient_domain, reject_unlisted_recipient, reject_unverified_recipient, reject_unknown_hostname, reject_unauth_destination, permit
smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = no
smtpd_sasl_local_domain = $myhostname
smtpd_sasl_path = private/auth
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_sender_restrictions = permit_sasl_authenticated, permit_mynetworks, check_client_access hash:${config_directory}/access, pcre:/usr/local/etc/postfix/rejected_domains, warn_if_reject reject_non_fqdn_sender, reject_unknown_client, reject_unknown_sender_domain, permit
smtpd_soft_error_limit = 5
smtpd_timeout = 30s
smtpd_tls_CAfile = /usr/local/etc/ssl/cacert.pem
smtpd_tls_auth_only = yes
smtpd_tls_cert_file = /usr/local/etc/ssl/server.crt
smtpd_tls_dh1024_param_file = /usr/local/etc/ssl/dh2048.pem
smtpd_tls_dh512_param_file = /usr/local/etc/ssl/dh512.pem
smtpd_tls_eecdh_grade = strong
smtpd_tls_key_file = /usr/local/etc/ssl/server.key
smtpd_tls_loglevel = 1
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
smtpd_tls_protocols = !SSLv2, !SSLv3
smtpd_tls_received_header = yes
smtpd_tls_security_level = may
smtpd_tls_session_cache_timeout = 3600s
smtpd_use_tls = yes
tls_eecdh_strong_curve = prime256v1
tls_eecdh_ultra_curve = secp384r1
tls_random_source = dev:/dev/random
transport_maps = hash:${config_directory}/transport
unknown_local_recipient_reject_code = 550
virtual_alias_maps = proxy:pgsql:${config_directory}/pgsql_virtual_alias_maps.cf
virtual_gid_maps = static:125
virtual_mailbox_base = /usr/local/virtual
virtual_mailbox_domains = proxy:pgsql:${config_directory}/pgsql_virtual_domains_maps.cf
virtual_mailbox_limit = 536870912
virtual_mailbox_limit_maps = proxy:pgsql:${config_directory}/pgsql_virtual_mailbox_limit_maps.cf
virtual_mailbox_maps = proxy:pgsql:${config_directory}/pgsql_virtual_mailbox_maps.cf
virtual_minimum_uid = 125
virtual_transport = lmtp:127.0.0.1:2424
virtual_uid_maps = static:125

I’ve done a number of things, picked up over the years. A few things of note:

  • Again, I’ve disabled SSLv2, and SSLv3 for both SMTP and SMTPD (handled separately).
  • All the ‘proxy:pgsql’ statements are where postfix will do lookups against information in our PostgreSQL database (things like domains, email addresses, aliases, etc.
  • Postscreen is setup to query a number of DNSBL (DNS Black List) and then weigh the answer accordingly. If there are too many hits, the connection is rejected. THis is nice because it keeps a single DNSBL from blocking a connection (sometimes a server is added to a list mistakenly, or because there was a problem, but its been fixed). if you’re not familiar with Postscreen, you can read about it here

The master.cf (also in /usr/local/etc/postfix) controls where and how postfix communicates with other programs. For example, if you want it to listen on a certain port, such as TCP/465, you need to define the ‘smtps’ entry you see below.

# master.cf
smtpd     pass  -       -       n       -       -       smtpd

submission inet n       -       n       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_enforce_tls=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_helo_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject_non_fqdn_recipient,reject_unknown_recipient_domain,reject
  -o milter_macro_daemon_name=ORIGINATING

smtps     inet  n       -       n       -       -       smtpd
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

127.0.0.1:10026 inet  n -       n       -       -        smtpd
    -o content_filter=
    -o receive_override_options=no_address_mappings,no_unknown_recipient_checks,no_header_body_checks
    -o smtpd_helo_restrictions=
    -o smtpd_client_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8

smtp-amavis     unix    -       -       -       -       2       smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
        -o max_use=20

127.0.0.1:10025 inet    n       -       -       -       -       smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_delay_reject=no
        -o smtpd_client_restrictions=permit_mynetworks,reject
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o smtpd_data_restrictions=reject_unauth_pipelining
        -o smtpd_end_of_data_restrictions=
        -o mynetworks=127.0.0.0/8
        -o smtpd_error_sleep_time=0
        -o smtpd_soft_error_limit=1001
        -o smtpd_hard_error_limit=1000
        -o smtpd_client_connection_count_limit=0
        -o smtpd_client_connection_rate_limit=0
        -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

pickup    unix  n       -       n       60      1       pickup
	-o content_filter=
	-o receive_override_options=no_header_body_checks
cleanup   unix  n       -       n       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
trace     unix  -       -       n       -       0       bounce
verify    unix  -       -       n       -       1       verify
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       n       -       -       smtp
relay     unix  -       -       n       -       -       smtp
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
retry     unix  -       -       n       -       -       error
discard   unix  -       -       n       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       n       -       -       lmtp
anvil     unix  -       -       n       -       1       anvil
scache    unix  -       -       n       -       1       scache
smtp      inet  n       -       n       -       1       postscreen
dnsblog   unix  -       -       n       -       0       dnsblog
tlsproxy  unix  -       -       n       -       0       tlsproxy
# dspam retraining
dspam-retrain         unix    -       n       n       -      -     pipe
	flags=Rhq
	user=dspam
	argv=/usr/local/bin/dspam --client --mode=teft --class=$nexthop --source=error --user ${sender}

Of note, I added the last couple lines so that mail could be forwarded to specific addresses and it would be passed directly to dspam for training.

Also, I’ve ensured that people who authenticate aren’t subjected to the DNSBL checks, because mobile users aren’t in control of the network they’re on and shouldn’t be subjected to its reputation (or lack there of).

I then added these entries to the transport file (hash with ‘postmap transport’). This allows anybody who is allowed to send mail through the system to submit (as an attachment) spam or ham to be reclassified by dspam.

spam@spam.spam	dspam-retrain:spam
ham@ham.ham		dspam-retrain:innocent

I want to strip out some headers as things pass through, including some of the authenticated user headers (don’t need to advertise that to the world), as well as the spam/virus scanner headers. It doesn’t matter if somebody else has scanned it or not, it all gets scanned locally.

# We use the header_checks file to remove some headers that we find undesirable.
# Return receipts and software versions are the most significant in this
# situation.

/^Received: from 127.0.0.1/                     IGNORE
/^X-Authenticated-Sender:/                      IGNORE
/^User-Agent:/                                  IGNORE
/^X-Mailer:/                                    IGNORE
/^X-MimeOLE:/                                   IGNORE
/^X-MSMail-Priority:/                           IGNORE
/^X-Originating-IP:/                            IGNORE
/^X-Sanitizer:/                                 IGNORE
/^X-Spam-Status:/                               IGNORE
/^X-Spam-Level:/                                IGNORE
/^X-Virus-Check:/                               IGNORE
/^X-Virus-Checked:/                             IGNORE
/^X-Virus-Scan:/                                IGNORE
/^X-Virus-Scanned:/                             IGNORE
/^X-Virus-Status:/                              IGNORE
/^(X-DSPAM-.*)/                                 IGNORE

These files control how Postfix looks up data in our PostgreSQL database. Each one contains the user, password, database, host, and query to be executed. These queries return a set of information that tells postfix how to behave when its accepting or sending messages (postfix doesn’t deliver messages with this setup, that’s left to dovecot.

# pgsql_relay_domains_maps.cf
user = postfix
password = password
hosts = localhost
dbname = postfix
query = SELECT domain FROM domain WHERE domain='%s' and backupmx = true
# pgsql_virtual_alias_maps.cf
user = postfix
password = password
hosts = localhost
dbname = postfix
query = SELECT goto FROM alias WHERE address='%s' AND active = true
# pgsql_virtual_domains_maps.cf
user = postfix
password = password
hosts = localhost
dbname = postfix
query = SELECT domain FROM domain WHERE domain='%s' and backupmx = false and active = true
# pgsql_virtual_mailbox_limit_maps.cf
user = postfix
password = password
hosts = localhost
dbname = postfix
query = SELECT quota FROM mailbox WHERE username='%s'
# pgsql_virtual_mailbox_maps.cf
user = postfix
password = password
hosts = localhost
dbname = postfix
query = SELECT maildir FROM mailbox WHERE username='%s' AND active = true

Starting up postfix

You should now be able to start postfix with:

service postfix start

and check that its running with:

[louisk@mail postfix 16 ]$ sockstat -46l | grep master
root     master     2985  17 tcp4   *:587                 *:*
root     master     2985  18 tcp6   *:587                 *:*
root     master     2985  21 tcp4   *:465                 *:*
root     master     2985  22 tcp6   *:465                 *:*
root     master     2985  25 tcp4   127.0.0.1:10026       *:*
root     master     2985  31 tcp4   127.0.0.1:10025       *:*
root     master     2985  105 tcp4  *:25                  *:*
root     master     2985  106 tcp6  *:25                  *:*
[louisk@mail postfix 17 ]$

We see postfix listening on all the ports we want.


Footnotes and References