Scalable Mail System Architecture

In this article, we consider one of the options for implementing the scalable architecture of a large mail system.

On December 6, 2012, Google stopped registering new accounts for the free version of Google Apps.

The clients of our company constantly have a need for e-mail serving their sites.
Previously, we set up Google Apps for them, but after December 6, having studied the solutions offered on the market, we decided that it was time to build our own mail system.

As you know, appetite comes with eating. If you already decided to build something of your own, then it’s worth immediately laying down opportunities for growth.
For the designed mail system, the following requirements were formulated:

• scalability (unlimited number of domains served, the total volume of mailboxes is 100 terabytes or more);
• fault tolerance (all intermediate services should be duplicated);
• extensibility (adding new nodes to the system should be easy and simple).

We started by choosing a repository for letters ...

In fact, the choice was small:

• dovecot / cyrus with storage in the file system via maildir / mailbox;
• dbmail with storage in the database.

After evaluating the number of services provided “out of the box”, as well as examining the nuances of use, we decided to dbmail .
Here is a short list of the goodies that dbmail provides:

• access to mailboxes via IMAP, POP3;
• sieve scripts for sorting mail;
• receiving mail through smtp and lmtp protocols;
• administration through cli and SQL queries.

Postfix was chosen as the MTA. Since we have been working with it for a long time, we confidently believe it to be a very fast, flexible and easily customizable program. Moreover, on the dbmail site there is an instruction for setting up the postfix + dbmail bundle.

And the best proxy of all time is working as a load balancer: nginx.
A web interface to mailboxes can be provided by any of dozens of available programs. For starters, it was decided to use the RoundCube.

As a result, the following project architecture began to emerge:

1. At the forefront, we have nginx running with the Mail module turned on, it accepts SMTP, IMAP, POP3 connections from clients and, based on the username and password, proxies the connection to the desired server.
2. In the depths of defense, there are many postfix and dbmail, each of which serves its own domains.

To the outside world, it all looks like one big mail server.

In theory, scalability has been achieved, now the task is to implement.

Each module of the system will work in its OpenVZ container. As the load increases, we will clone / migrate containers to new physical servers.

In separate containers we put nginx, postfix with various mail filters, dbmail and the message storage in the MySQL database will be placed in one container, since dbmail should not load the processor much, and the volumes of data transferred between dbmail and mysql will be large (duplication of data in the storage Letters will be implemented through the MySQL master-slave replication mechanism, organizing a hotbackup module for each dbmail module).

Also, for convenience, in separate containers you can select web servers with a roundcube and a control panel of the entire system.

All containers communicate with each other over a local network with gray addresses.
It was planned to give out real Internet addresses only to nginx containers.
During the installation and configuration of nginx, it turned out that he could not proxy SMTP sessions without authorization. The solution to this problem slightly changed the initially planned architecture of the project, which led to increased resiliency and extensibility. We refused to proxy SMTP traffic through nginx, and decided that postfix modules will receive it directly.

All postfix-modules of the system were made the same, each with its own real IP-address, and we determine which dbmail-module to send mail through transport_maps.
As a result, any postfix module can receive mail for any domain in the system, apply various filters and deliver it to the desired storage.
They will be balanced through round-robin DNS entries:

nginx1   IN A 1.1.2.1        
nginx2   IN A 1.1.2.2
postfix1 IN A 1.1.1.1
postfix2 IN A 1.1.1.2
postfix3 IN A 1.1.1.3
mx1      IN A  1.1.1.1
         IN A  1.1.1.2
         IN A  1.1.1.3
mx2      IN A  1.1.1.2
         IN A  1.1.1.3
         IN A  1.1.1.1
mx3      IN A  1.1.1.3
         IN A  1.1.1.2
         IN A  1.1.1.1
imap     IN A  1.1.2.1
         IN A  1.1.2.2
pop3     IN A  1.1.2.1
         IN A  1.1.2.2
smtp     IN CNAME mx1
_spf     IN TXT     "v=spf1 ip4:1.1.1.1 ip4:1.1.1.2 ip4:1.1.1.3 ?all"

The necessary entries are added to the client’s domain:

@                        IN MX 10   mx1.dbmail.io.
@                        IN MX 20   mx2.dbmail.io.
@                        IN MX 30   mx3.dbmail.io.
@                        IN TXT     "v=spf1 a mx include:_spf.dbmail.io -all"

Users' email clients are configured to receive mail from pop3.dbmail.io or imap.dbmail.io, and sending mail via smtp.dbmail.io

Only dbmail modules are not duplicated and unique, but they are not accessible from the Internet directly, so the need for there is no dynamic load balancing on them yet.
Each of the dbmail modules stores mail for several domains; if the resources of the module run out,
some of the domains can be moved to a separate module.

And in the future there are plans to switch to storing mail in some dedicated cluster database and unify dbmail modules.

The result is such a scalable fault-tolerant and expandable mail system:

All elements of our mail system communicate with each other according to standard protocols and, as it was naive to imagine, everything should have worked immediately and without the need to build backups.
Unfortunately, the design had to be finished with a file.
Read the history of tuning and debugging in the following articles.