DSPAM_Setup.html

DSPAM Setup

DSPAM is a statistical hybrid filter for SPAM. It is very similar in nature to the junk mail filter that's built into Mail.app, but with one major advantage-the message processing is done on your mail server, not on the mail client.

By processing the messages on the server, you eliminate the need to download hundreds of messages just to throw them into a junk mail folder. Also, keeping statistics about SPAM on the server allows an administrator to build composite information gathered from all users of the mail server-allowing new users to benefit from the SPAM caught by previous users.

The following instructions will help you configure DSPAM on a stock Mac OS X Server installation (version 10.3.x). This configuration utilizes the existing tools that are installed: Postfix, Cyrus, MySQL and procmail.

To perform these steps, you should be feel comfortable using command line tools to download, configure, build and install software.

The latest version of DSPAM is available at the following URL:

<http://www.nuclearelephant.com/projects/dspam/>

These instructions were written for version 3.0.0 and assume that you have reviewed the README file from the DSPAM download. It is also assumed that you've put the DSPAM source code in ~/Temp/dspam-3.0.0/ after downloading.

0. Set up MySQL

MySQL is used as the backend storage for DSPAM. To set it up, run Applications > Server > MySQL Manager from the Finder. First use Install to copy files and then use Start to get the mysqld running.

1. Set up procmail.dspam

Create a copy of /usr/bin/procmail (called /usr/bin/procmail.dspam) so that CGI scripts can run the DSPAM agent (/usr/local/bin/dspam) which can then run procmail as root.

% cd /usr/bin

% cp procmail procmail.dspam

% sudo chmod u+s procmail.dspam

% sudo chgrp mail procmail.dspam

% sudo chmod o-x procmail.dspam

You should now see something like this:

% ls -la procmail*

-rwxr-xr-x 1 root wheel 87208 17 Mar 15:41 procmail

-rwSr-xr-- 1 root mail 87208 7 Jun 16:56 procmail.dspam

2. Patch MySQL Library

Patch libmysqlclient.a because DSPAM build will fail with:

/bin/sh ./libtool --mode=link gcc -g -O2 -Wall -Wmissing-prototypes -Wmissing-declarations -L/usr/lib/mysql -o libdspam.la -rpath /usr/local/lib -version-info 5:0:0 libdspam.lo tbt.lo lht.lo base64.lo buffer.lo util.lo nodetree.lo error.lo decode.lo mysql_drv.lo -lm -lmysqlclient -lm -lz

gcc -dynamiclib -flat_namespace -undefined suppress -o .libs/libdspam.5.0.0.dylib .libs/libdspam.o .libs/tbt.o .libs/lht.o .libs/base64.o .libs/buffer.o .libs/util.o .libs/nodetree.o .libs/error.o .libs/decode.o .libs/mysql_drv.o -all_load /usr/lib/mysql/libmysqlclient.a -L/usr/lib/mysql -lm -lz -install_name /usr/local/lib/libdspam.5.dylib -compatibility_version 6 -current_version 6.0

ld: common symbols not allowed with MH_DYLIB output format with the -multi_module option

/usr/lib/mysql/libmysqlclient.a(my_error.o) definition of common _errbuff (size 512)

/usr/bin/libtool: internal link edit command failed

To patch the library, use:

% cd /usr/lib/mysql

% mv libmysqlclient.a libmysqlclient.a.original

% mkdir /tmp/mysql

% cd /tmp/mysql

% ar x /usr/lib/mysql/libmysqlclient.a.original

% ld -r -d my_error.o

% mv a.out my_error.o

% libtool -o /usr/lib/mysql/libmysqlclient.a *.o

(Ignore "no symbols" warnings)

More information on how this patch works can be found here: <http://curl.haxx.se/mail/lib-2002-06/0202.html>

3. Configure and Build DSPAM

To configure DSPAM to run with procmail as the delivery agent and MySQL as the backend, use:

% cd ~/Temp/dspam-3.0.0

% ./configure \

--with-delivery-agent='/usr/bin/procmail.dspam' \

--with-storage-driver=mysql_drv \

--with-mysql-includes="/usr/include/mysql" \

--with-mysql-libraries="/usr/lib/mysql"

If you want to build the debug version, change debug_text to static in error.h

41c41

< char debug_text[1024];

---

> static char debug_text[1024];

and add the following to the configure command above:

--enable-debug \

Create and install the DSPAM binaries with:

% make

% sudo make install

This will populate:

/usr/local/lib

/usr/local/bin

and create:

/var/dspam

Make dspam setgid:

% chmod g+s /usr/local/bin/dspam

% ll /usr/local/bin/dspam

-r-x--s--- 1 root mail 391500 16 Jun 09:54 /usr/local/bin/dspam

4. Create MySQL database for DSPAM per user statistics

% mysqladmin create dspam

Grant access using instructions at: <http://dev.mysql.com/doc/mysql/en/Adding_users.html>

Create tables used by DSPAM:

% mysql -u root dspam < mysql_objects.sql.speed.optimized

Populate database with a backup of the prebuilt corpus (if you don't have a backup, skip this step):

% mysql -u root dspam < dspam-backup.sql

5. Create configuration file that allows DSPAM to connect to MySQL

Create /var/dspam/mysql.data with contents as:

127.0.0.1

3306

root

dspam

Note the blank line, it's because we have not set a password. The values are db host IP address, port, username, password and database name.

Set owner:group and permissions:

% chmod u+rw g-w o-rw mysql.data

% chown root:mail mysql.data

6. Setup DSPAM CGI

Using the Server Admin application, create a new web Domain in Web Service (DSPAM) enabled on port 8080. After setup, you will be accessing the DSPAM web tools with:

http://mail.yourserver:8080/DSPAM

General:

Set web folder to /Library/WebServer/Documents/DSPAM

Add "dspam.cgi" to default index files (at top of list)

Copy the entire contents of ~/Temp/dspam-3.0.0/cgi folder to /Library/WebServer/Documents/DSPAM

Change the owner of all CGI scripts (*.cgi) to "www", so you have:

-rwxrwxr-- 1 www admin 15825 17 Jun 10:14 admin.cgi

-rwxrwxr-- 1 www admin 3076 16 Jun 17:11 admingraph.cgi

-rwxrwxr-- 1 www admin 26098 17 Jun 11:52 dspam.cgi

-rwxrwxr-- 1 www admin 2660 16 Jun 11:21 graph.cgi

Options:

Enable CGI Execution.

Realms:

Create realm named DSPAM, add users who will be checking & processing mail with browse privileges, add group "admin" with browse and author privileges.

Using the Workgroup Manager, add the "www" user to "mail" group so that Apache can send mail.

Configure dspam.cgi (Perl source code):

## Configuration

$CONFIG{'DSPAM_HOME'} = "/var/dspam";

$CONFIG{'ME'} = "dspam.cgi";

$CONFIG{'DSPAM'} = "/usr/local/bin/dspam";

$CONFIG{'DSPAM_ARGS'} = "--deliver=innocent --mode=teft --class=innocent " .

"--source=error --user $ENV{'REMOTE_USER'} " .

"-t -m USER=$ENV{'REMOTE_USER'} /etc/procmailrc";

$CONFIG{'LARGE_SCALE'} = 0; # --enable-large-scale

$CONFIG{'DOMAIN_SCALE'} = 0; # --enable-domain-scale

$CONFIG{'TEMPLATES'} = "./templates"; # Location of HTML templates

$CONFIG{'HISTORY_SIZE'} = 200; # Number of items in history

$CONFIG{'MAX_COL_LEN'} = 45; # Max chars in list columns

## End Configuration

Configure admin.cgi (Perl source code):

## Configuration

$CONFIG{'DSPAM_HOME'} = "/var/dspam";

$CONFIG{'ME'} = "admin.cgi";

$CONFIG{'TEMPLATES'} = "./templates"; # Location of HTML templates

$CONFIG{'ALL_PROCS'} = "ps -ax";

$CONFIG{'MAIL_QUEUE'} = "mailq | grep '^[0-9,A-F]' | wc -l";

$CONFIG{'PREFIX'} = "/usr/local"; # DSPAM prefix

$CONFIG{'LARGE_SCALE'} = 0;

$CONFIG{'DOMAIN_SCALE'} = 0;

## End Configuration

Setup Graph::GD for graphics in Perl CGI scripts:

Download libpng from <http://www.libpng.org/pub/png/libpng.html> then build and install:

% cp scripts/makefile.macosx Makefile

% make

% sudo make install

(Ignore make errors for pngtest.)

Download libgd from <http://www.boutell.com/gd>, configure build and install:

% ./configure

% make

% sudo make install

% sudo ranlib /usr/local/lib/libpng.a

Get GD::Graph from CPAN:

% sudo cpan

cpan> install GD

installed? /usr/local/lib

support? PNG only

cpan> install GD::Graph::lines3d

cpan> quit

7. Setup Postfix to send messages to DSPAM

Add to end of /etc/postfix/master.cf.defaultserver:

# DSPAM processing before handling message to procmail

dspam unix - n n - - pipe

flags=Rhqu user=cyrus argv=/usr/local/bin/dspam --mode=teft --deliver=innocent --feature=chained,noise --user ${user} -t -m USER=${user} EXTENSION=${extension} /etc/procmailrc

Set flags="hqu" if you want to omit "Return-Path:" from the headers.

Add to /etc/postfix/main.cf

mailbox_transport = dspam

default_privs = cyrus

(Be careful to remove other instances of "mailbox_transport", there's one for Cyrus at end of file!)

After updating the Postfix configuration, shut down the mail service, remove the fscking stupid swap file, and restart the service.

% sudo serveradmin stop mail

% rm /etc/postfix/master.cf.defaultserver.swap

% sudo serveradmin start mail

Create /var/dspam/trusted.users with user ids that will need to run DSPAM:

root

cyrus

www

postfix

DSPAM uses special mail aliases, spam-user and ham-user, that update the per-user statistics. Create these aliases by editing the aliases file (as reported by the "postconf alias_maps" command):

cyrus: root

debug: root

spam-user: "|'/usr/local/bin/dspam' --user 'user' --class=spam --source=error --mode=teft"

ham-user: "|'/usr/local/bin/dspam' --user 'user' --class=innocent --source=error --mode=teft --deliver=innocent -t -m USER=user EXTENSION= /etc/procmailrc"

spam-global: "|'/usr/local/bin/dspam' --user 'global' --class=spam --source=corpus --mode=teft"

ham-global: "|'/usr/local/bin/dspam' --user 'global' --class=innocent --source=corpus --mode=teft"

Replace user with a real user name, there is a spam-user and ham-user for every user who receives mail on the server. The spam-global and ham-global are used to update the corpus used by all users.

The cyrus and debug aliases are used for debugging.

After the aliases are update, deploy them with:

% sudo newaliases

8. Create /etc/procmailrc

This procmailrc file will be processed after DSPAM has processed the message and determined that it is not SPAM. This allows you to do any processing necessary before the message is handed off to Cyrus' deliver command. This happens because the delivery agent was set to /usr/bin/procmail.dspam when running configure for DSPAM.

# to DEBUG the processing done by procmail, uncomment the following line:

#LOGFILE=/var/log/procmail

# the logfile needs to be created with:

# -rw-rw-r-- 1 root mail 2356 23 Jun 09:12 /var/log/procmail

VERBOSE=yes

HOME=/Users/$USER

DROPPRIVS=yes

# set shell so that formail used in autoreply will work correctly

SHELL=/bin/bash

# allow users to create their own recipes

INCLUDERC=$HOME/.procmail

# get debugging info

DEBUG="id='`id`' USER=$USER"

# get subject line from e-mail

#SUBJECT=`formail -xSubject:`

# body from e-mail

#BODY=`formail -I ""`

# convert user name to all lowercase

USER=`echo $USER | tr "[:upper:]" "[:lower:]"`

# debug autoreply

:0 h

* !^FROM_DAEMON

* !^FROM_MAILER

* !^X-Loop: debug_autoreply

* ^TO_debug@Monterey.local

| (formail -rt -A "Precedence: junk (autoreply)" \

-A "X-Loop: debug_autoreply" ; \

echo $DEBUG ; \

echo "EOM" ) | $SENDMAIL -t -i -F "Debug" -f postmaster

# if no matches above, deliver message to the user's inbox

:0w

| /usr/bin/cyrus/bin/deliver -a $USER -m user/$USER

9. Create a global user for DSPAM statistics

Create a "global" user with Workgroup Manager:

Name: Global DSPAM User

Short name: global

Mail: None

Feed message from corpus into the global user:

% cd SA-Corpus

% chmod u+x train.pl

% sudo ./train.pl global

Create /var/dspam/group file with:

groupname:classification:*global

Merge individual user data into global:

% /usr/local/bin/dspam_merge bob carol ted alice -o global

Check that global reporting aliases "ham-global" and "spam-global" are in /etc/aliases. These can be used by other users to train the global user (by forwarding messages to the correct address.)

10. Test

At this point, any mail sent to a user account on the server will pass through DSPAM before being delivered. Any mail that is put into the quarantine (because it is thought to be SPAM) can be viewed with your web browser:

http://mail.yourserver:8080/DSPAM

Of course, nothing ever goes as planned, so if you have problems here are some good places to look:

/var/mail for messages that are not getting delivered correctly

/var/log/mail.log Postfix and Cyrus message processing

/var/log/system.log Postfix and Cyrus errors

/var/dspam/dspam.debug DSPAM processing log (if enabled during configure)

/var/dspam/dspam.messages DSPAM message log (if enabled during configure)

/var/dspam/sql.errors DSPAM errors with MySQL backend

/var/dspam/data/user/user.log A log of messages processed for user (along with status)

/var/dspam/data/user/user.mbox The quarantine for user

/etc/postfix/master.cf.defaultserver Postfix transport configurations

/etc/postfix/main.cf Postfix configuration

/etc/imap.conf Cyrus configuration