Technical recommendations for mailing lists

“Even if you receive a letter, you will not be able to read it”
(Mark Twain)

We already wrote about how to make mailings correctly, improve their quality and efficiency. They gave metrics on the basis of which the sender's reputation is built. They talked about the Postmaster interface Mail.Ru , with which they can be tracked. Many companies, both those that are at the beginning of their development, and rather large ones, neglect these rules, as a result of which problems arise with the deliverability of letters, litigation with the technical support service, etc. But we hope that you are not one of them.

So, your project is gaining popularity and users like it, you are going to stay in touch with them. You have reviewed the administrative requirements (of which wewrote earlier ) and are going to responsibly and without spam to organize the newsletter for those users who are willing to receive it. Or maybe you're just about to set up corporate email. You pick up the mail server from the distribution kit, write a script, run it and ... 70% of the recipients did not receive the message, 15% got it into the Spam folder, and the rest cannot read what is written in it. I’ll try to talk about what to do to prevent this from happening in this article.

Why can everything not work out of the box and some additional recommendations are required? E-mail is one of the oldest Internet protocols, which has survived to our days practically unchanged, but has gained a huge number of related standards and application practices that are not documented anywhere. The email standards themselves do not have any methods for authenticating the sender or preventing unwanted mailings and spam. Therefore, any mail server performs a number of additional checks to filter out unwanted correspondence (which may be more than 90%).

We’ve put together a list of tips to help you get through these additional tests. Many of the recommendations listed below are not mandatory, but the more of them you follow, the higher the likelihood that there will be no problems with the delivery of letters.

Recommendations for the mail server administrator:

We start with ... IP address. From the provider, host or registrar, you received an IP address (or an entire network). What needs to be done to use the selected address for the mail server, and is it suitable?

1. Make sure the address is real and static.
Most mail servers restrict the reception of mail from dynamic addresses or from networks with address translation. The reason is that in such networks there are end-user computers that do not deliver mail. Accordingly, almost all mail that comes from these networks is spam sent through bots. Therefore, the reputation of these networks is hopelessly damaged.

2. Check whois information with the whois utility or online services (easy to find). Here is a list of what you need to pay attention to:
2.1The network must have a correct description. If the description contains information that it is dynamic or intended for distribution to individual end users (for example, ADSL networks), most likely you will also encounter many problems when trying to use it.
2.2 For the network, abuse-contacts for complaints must be indicated, and these contacts must be live and responsive. Otherwise, the network (and with it your IP) has a great chance of “flying” into blacklists in the presence of spam incidents from any of its hosts.
2.3If the network is European (RIPE registrar), the network status must be ASSIGNED, ASSIGNED PA or ASSIGNED PI - this means that this network is used. Many registrars forget to mark the network as used, and some recipients prohibit receiving mail from unused networks.
Require the registrar to enter the correct information in the network description, but remember that many potential recipients already “remember” the old information. Therefore, it is better to initially use IP addresses from well-administered networks with a good reputation.

3. Check and configure the PTR record for the IP address. The PTR record must point to the real host name. It is desirable that
3.1This name did not resemble the name of a dynamic host, that is, it did not contain the words dynamic, adsl, nat, long sequences of characters or groups of numbers. For example, server.example.com is a good, suitable PTR record.
3.2 If you plan a really large amount of outgoing mail from several servers, then it is advisable to name the servers in accordance with the draft standard " DNS Naming Convention for Outbound Internet Email Servers " and create the zone and A-record mxout provided for by this document.
3.3 The name of the PTR record must be resolved back to the same IP address, that is, an A or CNAME record server.example.com must exist and resolved to the IP address of the server from the example.
If the server will support several domains, there is no need to make PTR records in each domain - just one record on any of the domains is enough. When checking DNS, it is better to use a third-party DNS server to make sure that the records are really visible from the outside.
PTR records are managed by an IP network operator, such as an Internet or hosting provider. Usually this can be done independently through the hosting control panel or the provider's support service.

4.Configure MX records for the domain from which the mailing will be performed. Even if you do not plan to accept any letters for this domain. At a minimum, you should correctly receive and process messages about the impossibility of delivering your letters and the postmaster @ address. Using domains without MX records in the sender’s address or in the SMTP envelope negatively affects message delivery. Some recipients validate the addresses of senders from the SMTP envelope, therefore it is desirable that the server does not give errors when sending a letter to such an address.

5. Configure the SPF record . SPF allows you to specify which servers are allowed to send mail on behalf of a domain. SPF is configured for the address used in envelope-from (SMTP envelope).

6. ConfigureDKIM :
6.1 Generate a DKIM key pair and publish the public key;
6.2 Remember to configure the DKIM signature for all outgoing emails (more on that later).
DKIM allows you to sign letters that are sent on behalf of a specific domain. Currently, DKIM is already a must have technology, it is the technical basis for others - FBL, DMARC, as well as for services such as Postmaster Mail.Ru.

7. Publish the DMARC policy . DMARC indicates exactly how to use SPF and DKIM, and allows you to completely eliminate phishing on behalf of a domain by publishing restrictive policies. And it also gives you the opportunity to receive in the form of structured reports all information about attempts to violate your policy.
DMARC has not yet been adopted as an official standard, but is already supported by the main players in the email market - Mail.Ru, Google, Yahoo and Microsoft. It is very easy to implement and at the same time extremely effective solution.
Now you can try to raise the mail server.

8. In the settings of the Mail Transfer Agent (mail server), configure the hostname of the server, which is transmitted in the HELO command. This name must match the canonical server name from the PTR record (server.example.com). Very often, by default, HELO uses names like localhost.localdomain, and many mail servers refuse to accept mail with this HELO. Configure an SPF record for the name from HELO, this is an RFC 7208 requirement that will allow delivery of service letters with an empty SMTP envelope address.

9.Enable DKIM support in your mail server. In some servers, support is built-in, in some it can be implemented using free programs (for example, OpenDKIM), for Microsoft Exchange / IIS SMTP there are inexpensive and free plug-ins. When setting up, do not confuse DKIM (RFC 4871 / RFC 6376) and DomainKey (RFC 4870). DomainKey is an outdated standard that has never been adopted. Making his support is optional. DKIM can be recognized by the presence of the DKIM-Signature signature in the message headers. It is better to use the relaxed mode to sign the headers and body of the letter.

10. Configure the postmaster box.

11. Avoid configurations that lead to unauthorized relaying of letters, otherwise all work will go down the drain along with the reputation of the server.

12.Test the configuration of your server by sending letters to the main mail services through a standard mail client, for example, Thunderbird. Check the headers of the received emails. Make sure:
12.1 that HELO is correct;
12.2 the availability and passage of SPF-, DKIM- and DMARC-checks on servers that support DMARC;
12.3 inability to unauthorized relaying through your server;
12.4 that letters are sent to postmaster @ and to addresses for DMARC and FBL reports.
12.5 the server correctly generates reports on the inability to deliver letters.

13. Subscribe to FBL (FeedBack Loop). Fblprovided by many major email services. Subscribing will give you the opportunity to receive information about all complaints of spam in the mail from your domain.

Now you can configure mailings / write scripts for sending letters. Getting down.

14.Process messages about the inability to deliver letters. Remember that there are two ways to get an error: in an SMTP session (in this case, a message about delivery failure can form your server) or a letter with a report about delivery failure (NDR) from the recipient server. Both cases must be handled. At the same time, it is imperative to respond and exclude the user from the newsletters in case of repeated or permanent (permanent) delivery errors to his address. The presence of a large number of invalid recipients can lead to a decrease in reputation and / or triggering of grayling.

fifteen.Set the SMTP address MAIL FROM: (envelope sender) - the sender address that is used at the SMTP protocol level. It may not match the sender address from the From: message header. For normal operation of DMARC, it is desirable that the From: address and envelope sender address are from the same domain. And, of course, the letter must have a valid DKIM signature for this domain and pass SPF checks. An incorrect sender address in an envelope is the most common and serious error in mailings from various websites.

16. Do not use public mail addresses in From: and MAIL FROM addresses, since such mailings will not pass SPF / DKIM authentication under DMARC. Use the addresses of your own domain.

17.In the Content-Type headers for all text parts of the letter, specify the correct encodings. Make sure that the actual encoding of the text matches the one indicated in the header. It is advisable to use one encoding in all headers and parts of the letter. The currently recommended, most widely supported text encoding is UTF-8. If the encoding is not specified (or incorrectly specified), the text may be displayed differently by various services and email programs.

18.Configure the From :, Message-ID: and Date: headers directly in the message sending script. Make sure that these headers, especially Date :, are formatted correctly. By standards, these headers are formed along with the text of the letter. If they are missing or incorrectly formed, headers may be added by one of the transit servers, which may lead to a violation of the integrity of the DKIM signature.

19.Make sure that in all the service and other headers, including the subject line (Subject), the names of the attached files (Content-Type and Content-Disposition), non-ASCII characters, in particular Cyrillic, are encoded in quoted-printable or base64. According to standards, unencrypted eight-bit characters cannot be found in message headers; some servers do not accept such letters. If you are guided by foreign subscribers, it is desirable that the letter does not contain any uncoded 8-bit characters.

20. Limit the string length and use the correct string terminators.
20.1The maximum allowed line length in email is 998 8-bit characters. When using UTF-8, one displayed character may take several octets, so try to make the text in the lines shorter or encode the text in base64.
20.2 The correct terminator for email strings is CRLF (characters 13 and 10). On Unix, the standard line terminator is LF. If the used script or MTA does not replace the line terminators, this can lead to problems - incorrect display of the letter or cutting off part of the text. Errors can also be caused by a double replacement of terminators. Check with the documentation or test to see if you want to recode the ends of the lines in your configuration.
20.3Line terminators should not break UTF-8 characters so that there is no situation in which a terminator is added to a long line, for example, between two octets of the same Cyrillic character. Therefore, the breakdown of the text should be done before the formation of the letter.
The encoding of text parts in base64 increases the message size by about a third, but it saves from all the problems associated with line terminators in text parts.

21. Try to adhere to a fairly simple writing structure, avoid deep nesting of multipart parts (that is, the inclusion of one component in another). If the letter contains more than one multipart-parts of each type (one multipart / mixed one multipart / alternative and one multipart / related), most likely, the letter is not optimally formed.

22. Correctly put the Content-Disposition of each part as an attachment (attachment shown separately from the letter) or inline (an object displayed as part of the letter, for example, a picture in the text). It often happens that the document attached to the letter is marked as inline, although it is intended for download by the user, or the logo in the message text is marked as attachment, although it should not be visible in the list of attached files.

23. Form the plaintext part of the letter as an alternative to the HTML part. Do not forget that the user can read the letter, for example, from a cell phone. Correctly and beautifully converting HTML content into text is not always possible. By adding the text part to the letter, you can be sure that the text will be shown exactly as you expect it to.

24.Test your script on test boxes from different email services. Do not forget to download the original letter that arrived in the mailbox and check:
24.1 correctness of envelope sender;
24.2 passing SPF-, DKIM- and DMARC checks;
24.3 the presence and validity of text encodings in the Content-Type headers for all text parts (including HTML);
24.4 in HTML parts - correspondence to the validity of the encodings indicated in the meta tags (if any);
24.5 display of various characters, including the Cyrillic alphabet in the text, HTML part and file names (if you plan to send files);
24.6the presence of the correct Content-Disposition and Content-Transfer-Encoding for each part of the letter (except multipart);
24.7 display of pictures and attachments;
24.8 correctness of line terminators;
24.9 correct pagination and display of long texts in HTML and plain text parts;
24.10 Make sure that the From: Date: and Message-ID headers are generated specifically by your server, signed with a DKIM signature and valid.

Now you can proceed to the layout of letters. Recommendations for the layout designer:

25.Be easier. Do not forget that web-mails are forced to filter some of the tags and attributes in order to ensure security, and they all do it differently. Avoid using classes, minimize the use of styles, especially for positioning. The good old layout on the tables is the most portable.

26. Do not try to impose a specific mail server under the web interface. Users often set up redirects, mail collectors, or read emails in email programs.

27. Think about the user. He can open your letter both on a huge retina display and on an old phone on top of Everest, trying to grab a GPRS connection. Accordingly, the letter should load quickly, but look beautiful even when the pictures are turned off.

28.No small prints. They make you nervous and make you suspicious. Feel free to show the user that you know and respect his rights and his duties.

29. Choose the most suitable way to embed images. There are the following options, each with its pros and cons:
29.1External pictures: in terms of sending speed, this option is the most preferable. External pictures do not weigh anything and do not complicate the structure of the letter. However, many applications and mail services require permission from the user to display such images. In addition, the server on which they are located must be reliable enough so that when you open the letter it does not lead to “freezes” due to the inaccessibility of the picture. Use them when having pictures is optional.
29.2 Inline Pictures. Sent along with the contents of the letter. They complicate the structure of the letter and increase its weight, but can be quite large and are usually displayed by default.
29.3Data URI The content of the image is inserted directly into the tag. They do not complicate the structure of the letter, they are almost always shown in webmail, sometimes even with pictures turned off. As a rule, they have a size limit, suitable for small icons.
29.4 Font pictures. Images of Unicode characters from standard fonts or using embedded custom fonts. Unique property - can be scaled along with the text. Virtually nothing weighs. May not be displayed correctly, depending on the version of the operating system and installed updates, device, mail program or service.

thirty.Be sure to include the protocol in all URIs (http: //, https: // or mailto :), using a URI without a protocol, for example, "//example.com/", is unacceptable and can lead to message freezing in some mail applications due to attempts contact the network, the use of protocols other than those listed is not recommended, because can be stabbed with filters.

31. Test the display of each new layout of the letter on various devices in different web mails before sending it to "live" users.

What to do if, despite following all the recommendations, the letters still do not reach the users? Contact the support services of the recipients of letters - as a rule, such problems are resolved quickly. You can write about delivery problems to addresses served by Mail.Ru servers at abuse@corp.mail.ru. Do not forget to describe the problem as fully as possible - error messages, texts about delivery failure messages, server logs and other technical information will greatly help.

Unfortunately, the format of the article does not allow detailed descriptions of each of the recommendations or give detailed instructions for customization, but if you have technical questions, I will be happy to answer them in the comments.