Project Titanicarus: Part 9 – Building the Email Servers

Email servers.. the bane of every sysadmins existence. The second something goes wrong with an email server, you’re guaranteed to get 100 phone calls and people dropping by your office to say “My emails aren’t working”. This is one part of your hosting infrastructure you want to get right.

I’ve decided to build my infrastructure on Postfix & Dovecot with a MySQL user database. My previous email setup was built using this howto. One of the major issues I ran into was with Courier’s inability to handle large mailboxes so I’ve decided to use a similar setup only with Dovecot in place of Courier and there are a couple of other major differences:

  1. This is going to be a highly distributed configuration (ie multiple servers in multiple datacentres)
  2. This is going to sit behind load balancers (brings interesting spam filtering and security issues)
  3. This is going to use a clustered MySQL backend

So the goal of todays blog post is to deliver:

  • Multi-server & multi-datacentre replicated mail stores
  • Fault tolerance (pull a server out at any time of the day and mail keeps flowing)
  • POP3 & IMAP user access
  • Authenticated SMTP Submission

Next week we will tackle:

  • Load balancers
  • Spam Filtering behind load balancers
  • Automated Security (POP3/IMAP bruteforce attacks etc)

Postfix

I’m installing Postfix with MySQL configs stored on our MySQL cluster, lets get cracking.

apt-get install postfix postfix-mysql postfix-doc mysql-client dovecot-common dovecot-sieve dovecot-imapd dovecot-mysql dovecot-pop3d libsasl2-2 libsasl2-modules libsasl2-modules-sql sasl2-bin libpam-mysql openssl telnet mailutils

This will install all the basic packages we need to get the basic email server up and running. Postfix Config When you’re prompted to select the type of mail server configuration, select “Internet Site” You’re going to be asked for a fully qualified domain name (FQDN), we will come back and fix that later if its not right so for the moment accept whatever is in there.

User Database

Now we need to create a database for the user configs, so login to your favourite MySQL DB modification tool (PHPMyAdmin/Command Line)

        • Create a database called “mail”
        • Add a user called “mailuser” who has access to the database only from the IP’s of your mail servers
        • Set a password
        • Grant the user “GRANT, INSERT, SELECT, UPDATE, DELETE, CREATE, DROP” privileges on the “mail” database
        • Dump the following SQL into the DB:
          CREATE TABLE `aliases` ( `pkid` smallint(3) NOT NULL auto_increment, `mail` varchar(120) NOT NULL default '', `destination` varchar(120) NOT NULL default '', `enabled` tinyint(1) NOT NULL default '1', PRIMARY KEY (`pkid`), UNIQUE KEY `mail` (`mail`) ) ;
          CREATE TABLE `domains` ( `pkid` smallint(6) NOT NULL auto_increment, `domain` varchar(120) NOT NULL default '', `transport` varchar(120) NOT NULL default 'virtual:', `enabled` tinyint(1) NOT NULL default '1', PRIMARY KEY (`pkid`) ) ;
          CREATE TABLE `users` ( `id` varchar(128) NOT NULL default '', `name` varchar(128) NOT NULL default '', `uid` smallint(5) unsigned NOT NULL default '5000', `gid` smallint(5) unsigned NOT NULL default '5000', `home` varchar(255) NOT NULL default '/var/spool/mail/virtual', `maildir` varchar(255) NOT NULL default 'blah/', `enabled` tinyint(3) unsigned NOT NULL default '1', `change_password` tinyint(3) unsigned NOT NULL default '1', `clear` varchar(128) NOT NULL default 'ChangeMe', `password` varchar(128) NOT NULL default 'sdtrusfX0Jj66', `quota` varchar(255) NOT NULL default '', `procmailrc` varchar(128) NOT NULL default '', `spamassassinrc` varchar(128) NOT NULL default '', PRIMARY KEY (`id`), UNIQUE KEY `id` (`id`) ) ;

Postfix config

        • Jump into /etc/postfix and add the following to main.cf:
# this specifies where the virtual mailbox folders will be located
virtual_mailbox_base = /var/spool/mail/virtual
# this is for the mailbox location for each user
virtual_mailbox_maps = mysql:/etc/postfix/mysql_mailbox.cf
# and this is for aliases
virtual_alias_maps = mysql:/etc/postfix/mysql_alias.cf
# and this is for domain lookups
virtual_mailbox_domains = mysql:/etc/postfix/mysql_domains.cf
# this is how to connect to the domains (all virtual, but the option is there)
# not used yet
# transport_maps = mysql:/etc/postfix/mysql_transport.cf
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000
  • You also want to make sure that “local_recipient_maps” and “mydestination” are empty as we’re going to be using virtual domains.
  • Next you want to symlink /data/mail to /var/spool/mail/virtual now you’ve got your replicating mail folder being looked at by postfix.
    ln -s /data/mail /var/spool/mail/virtual
  • Add “virtual” user & group:
    groupadd --system virtual -g 5000
    useradd --system virtual -u 5000 -g 5000
  • Change the permissions on the mail directory
    chown -R virtual:virtual /var/spool/mail/virtual
  • Create a file called /etc/postfix/mysql_alias.cf and add the following content:
    user=mailuser
    password=XXXXXXX
    dbname=mail
    table=aliases
    select_field=destination
    where_field=mail
    hosts=mysql
    additional_conditions = and enabled = 1
  • Create a file called /etc/postfix/mysql_domains.cf and add the following content:
    user=mailuser
    password=XXXXXX
    dbname=mail
    table=domains
    select_field=domain
    where_field=domain
    hosts=mysql
    additional_conditions = and enabled = 1
  • Create a file called /etc/postfix/mysql_mailbox.cf and add the following content:
    user=mailuser
    password=XXXXXX
    dbname=mail
    table=users
    select_field=maildir
    where_field=id
    hosts=mysql
    additional_conditions = and enabled = 1
  • Fix ownership of these files:
    chmod o= /etc/postfix/mysql*.cf
    chgrp postfix /etc/postfix/mysql*.cf
  • Add some extra stuff to Postfix configs:
    postconf -e 'mynetworks = 127.0.0.0/8'
    postconf -e 'message_size_limit = 30720000'
    postconf -e 'smtpd_sasl_auth_enable = yes'
    postconf -e 'broken_sasl_auth_clients = yes'
    postconf -e 'smtpd_sasl_authenticated_header = yes'
    postconf -e 'reject_unauth_pipelining,permit_mynetworks,permit_sasl_authenticated,reject_sender_login_mismatch,reject_non_fqdn_recipient,reject_unknown_recipient_domain,reject_unauth_destination,reject_unverified_recipient,permit'
    postconf -e 'smtpd_use_tls = yes'
    postconf -e 'smtpd_tls_cert_file = /etc/postfix/smtpd.cert'
    postconf -e 'smtpd_tls_key_file = /etc/postfix/smtpd.key'
    postconf -e virtual_transport=dovecot
    postconf -e dovecot_destination_recipient_limit=1
  • Create an SSL certificate for Postfix
    cd /etc/postfix
    openssl req -new -outform PEM -out smtpd.cert -newkey rsa:2048 -nodes -keyout smtpd.key -keyform PEM -days 365 -x509
  • chmod o= /etc/postfix/smtpd.key

Setup SASLAUTHD

mkdir -p /var/spool/postfix/var/run/saslauthd
cp -a /etc/default/saslauthd /etc/default/saslauthd.bak
  • Stick the following in /etc/default/saslauthd
START=yes
DESC="SASL Authentication Daemon"
NAME="saslauthd"
MECHANISMS="pam"
MECH_OPTIONS=""
THREADS=5
OPTIONS="-c -m /var/run/saslauthd -r"
  • Stick the following into /etc/pam.d/smtp
    auth required pam_mysql.so user=mailuser passwd=xxxxxx host=mysql db=mail table=users usercolumn=id passwdcolumn=password crypt=1
    account sufficient pam_mysql.so user=mailuser passwd=xxxxxx host=mysql db=mail table=users usercolumn=id passwdcolumn=password crypt=1
  • Stick the following into /etc/postfix/sasl/smtpd.conf
    pwcheck_method: saslauthd
    # mech_list: plain login
    mech_list: LOGIN CRAM-MD5 PLAIN
    log_level: 10
    allow_plaintext: false
    auxprop_plugin: sql
    sql_engine: mysql
    sql_hostnames: mysql
    sql_user: mailuser
    sql_passwd: xxxxxx
    sql_database: mail
    sql_select: select password from users where id='%u@%r' and enabled = 1
  • Fix Permissions:
    chmod o= /etc/pam.d/smtp
    chmod o= /etc/postfix/sasl/smtpd.conf
  • Add Postfix user to sasl group:
    adduser postfix sasl
    service postfix restart
    service saslauthd restart

Saslauthd should be working now..

Configure Dovecot

This part of the process is completely new to me as I’ve been running courier previously, lets see how we go. I have borrowed and adapted this process from this howto.

  • Stick the following at the end of /etc/postfix/master.cf
dovecot   unix  -       n       n       -       -       pipe
    flags=DRhu user=virtual:virtual argv=/usr/lib/dovecot/deliver -d ${recipient}
  • Backup the Dovecot config:
    cp -a /etc/dovecot/dovecot.conf /etc/dovecot/dovecot.conf.bak
  • Stick the following into /etc/dovecot/dovecot.confauth_debug = no
    auth_debug_passwords = no
    auth_mechanisms = plain login CRAM-MD5 DIGEST-MD5
    auth_verbose = no
    auth_verbose_passwords = no
    disable_plaintext_auth = yes
    first_valid_uid = 5000
    last_valid_uid = 5000
    protocols = imap pop3
    postmaster_address = postmaster@domain.com
    log_timestamp = “%Y-%m-%d %H:%M:%S ”
    mail_location = maildir:/var/spool/mail/virtual/%n@%0.d
    mail_debug = no
    mail_gid = 5000
    mail_privileged_group = virtual
    mail_uid = 5000
    maildir_copy_with_hardlinks = yesssl_cert = </etc/ssl/certs/dovecot.pem
    ssl_key = </etc/ssl/private/dovecot.pemnamespace {
    type = private
    separator = .
    prefix = INBOX.
    inbox = yes
    }passdb {
    args = /etc/dovecot/dovecot-sql.conf
    driver = sql
    }service auth {unix_listener auth-userdb {
    group = virtual
    mode = 0600
    user = virtual
    }
    }protocol lda {
    auth_socket_path = /var/run/dovecot/auth-userdb
    }
  • Stick the following into /etc/dovecot/dovecot-sql.conf:
    driver = mysql
    connect = host=mysql dbname=mail user=mailuser password=xxxxxx
    default_pass_scheme = CRYPT
    password_query = SELECT id as user, password FROM users WHERE id='%u';
  • Restart dovecot
    service dovecot restart
  • Check the logs to make sure everything is happy
    tail -f -n 100 /var/log/mail.log
  • Fix permissions on configs
    chgrp virtual /etc/dovecot/dovecot.conf
    chmod g+r /etc/dovecot/dovecot.conf
  • Test that dovecot is working
    root@app01:~# telnet localhost 110
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    +OK Dovecot ready.
  • Test that postfix is working
    root@app01:~# telnet localhost 25
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    220 app01.pen01.nsw.hooton.org ESMTP Postfix (Ubuntu)
    ehlo localhost
    250-app01.pen01.nsw.hooton.org
    250-PIPELINING
    250-SIZE 30720000
    250-VRFY
    250-ETRN
    250-STARTTLS
    250-AUTH LOGIN CRAM-MD5 PLAIN
    250-AUTH=LOGIN CRAM-MD5 PLAIN
    250-ENHANCEDSTATUSCODES
    250-8BITMIME
    250 DSN
    quit
    221 2.0.0 Bye
    Connection closed by foreign host.

Create Users & Domains

  • Open up PHPMyAdmin again and paste the following SQL into your mail database to create the basic required config settings.
    # Use phpMyAdmin or command line mysql 
    INSERT INTO domains (domain) VALUES 
    ('localhost'), 
    ('localhost.localdomain');
    INSERT INTO aliases (mail,destination) VALUES
    ('postmaster@localhost','root@localhost'), 
    ('sysadmin@localhost','root@localhost'), 
    ('webmaster@localhost','root@localhost'), 
    ('abuse@localhost','root@localhost'), 
    ('root@localhost','root@localhost'), 
    ('@localhost','root@localhost'), 
    ('@localhost.localdomain','@localhost');
    # Create a root user
    INSERT INTO users (id,name,maildir,crypt) VALUES 
    ('root@localhost','root','root/',encrypt('apassword') );
  • This is the SQL you need to add domains.
    INSERT INTO domains (domain) VALUES
    (‘domain.com’),
    (‘domain.org’),
    (‘mail.domain.org’);
  • This is the SQL you need to add aliases.
    INSERT INTO aliases (mail,destination) VALUES
    (‘@domain.org’,’@domain.com’),
    (‘@mail.domain.org’,’@domain.com’),
    (‘postmaster@domain.com’,’postmaster@localhost’),
    (‘abuse@domain.com’,’abuse@localhost’),
    (‘postmaster@domain.org’,’postmaster@localhost’),
    (‘abuse@domain.org’,’abuse@localhost’);
  • This is the SQL you need to add users.
    INSERT INTO users (id,name,maildir,password) VALUES
    (‘user1@domain.com’,’user1′,’user1@domain.com/’,encrypt(‘insertpasswordhere‘) ),
    (‘user2@domain.com’,’user2′,’user2@domain.com/’, encrypt(‘insertpasswordhere‘) );

Repeat

Repeat the above on all of your nodes (excluding the database stuff).

Done.

Setup a mail client to point at one of the cluster nodes and send yourself an email.. If your /data/mail directory is replicating properly you should be able to send an email via one node and pull it via imap or pop3 from another pretty quickly.

I did find a couple of gotcha’s that you might want to watch out for – SASL & Postfix don’t exactly play nice, I had to turn chroot off for submission and smtp in the /etc/postfix/master.cf for authenticated SMTP to work properly.

Next week we will setup spam filtering, inbound smtp routing and some additional security that will help reduce spam & bruteforce attacks.

Last updated by at .