Trigger Mailings

    Recently, in email marketing, automatic mailings to certain consumer groups are increasingly used. Typical Tasks:
    • happy birthday
    • call the site if the consumer has not visited it for a long time
    • make a personalized offer (divide consumers into segments and send each segment a letter)

    In this article, we will describe how we solved this problem - from writing each separate mailing list by a developer from scratch 3 years ago, to setting up mailing lists by a manager via the web interface at present. The story may be of interest not only to those involved in Email marketing, but also to everyone who has to periodically perform complex operations on certain consumer samples (I know that sounds very abstract, but in the end we had to solve such an abstract problem).

    First implementations

    3 years ago, such tasks arose extremely rarely and every time we implemented them from scratch. The same questions arose:
    1. How to tag consumers to whom we have already sent this email?
    2. How to process all consumers as quickly as possible and at the same time not slow down the work of sites (which access the same records in the database)?

    The answer to the first question was obvious to us: in our system information is stored about all significant actions performed by the consumer (entering the site, changing personal data) or over it (prize drawing, sending a notification). In addition, we use actions for a variety of technical notes by consumers. So when sending an automatic newsletter, we also decided to give the consumer a special action marker , as a mark, that this automatic newsletter was already sent to him. In order not to resend the newsletter, the condition “the consumer has no token action” is always added to the newsletter condition.

    On the second question, we got a lot of cones related to locking in the database, and as a result we came to the following pattern:
    1. Sending newsletters comes from the windows-service, which periodically checks to see if there are new consumers that match the conditions.
    2. In the service, the first step is one query to the database with the Read Uncommitted isolation level . This request pulls the Id of all consumers to whom the letter should be sent. Due to the low level of isolation, such a request does not impose locks on entries in the database and, as a result, extremely weakly affects the operation of the site. However, it does not guarantee the purity of the data and must be re-checked with a higher level of isolation.
    3. After we pulled the Id of consumers, for each consumer we perform a separate transaction with a isolation level of Serializable. In this transaction, we re-check whether the consumer is suitable for the conditions and if so, send him a letter and issue a marker action. Since we process each consumer in a separate transaction, locks are imposed only on the data of one consumer and are not affected by the work of other consumers. Since such a transaction is very short, the consumer to whom the letter is sent will also not have much problems if he visits the site at this time. The transaction isolation level should be exactly Serializable, in order not to accidentally send one letter twice, or not send a letter to a consumer who suddenly ceased to fit the conditions. Although, if we guarantee that sending the same mailing can go only from one stream and from one server, we’ll also hammer it at a small chanceRead committed transaction.

    Of course, after implementing several mailings on this template, we decided to make out the template code. For this, the BatchMailing class was created , and for each new mailing we created and registered its heir in a special registry. In the heir, it was necessary to overload the following properties and methods:
    • marker action template (we used to call the template an action type: I think this is a more understandable term for developers) that is issued when sending an email
    • sending method
    • a method that performs additional actions (for example, together with sending a birthday greetings, we can issue points to the consumer)
    • method that forms expressionion>verifying that the consumer is eligible

    The property and the first two methods never caused any problems, but compiling Expression was quite difficult . This Expression was used twice - first in the Read Uncommitted request to pull out the Id of consumers, and then in the Serializable transaction to re-check whether the consumer fits the condition. It had to be written so that Linq to SQLwas able to translate it into T-SQL. The conditions could be quite complex and they always had problems. Not a single newsletter could be made without writing a bunch of tests on it. In addition, to send SMS and email, we got different intermediate heirs from BatchMailing. When we had to send both email and SMS, we had to copy-paste. I had ideas on how to fix this, but since clients didn’t ask for automatic mailings so often, this was a low priority task.

    Replacing inheritance with a composition

    2 years ago, when developing the next advertising campaign, the client asked him to make 8 different automatic mailings at once. Moreover, partly the conditions in the newsletters were repeated. There was no longer any doubt that it was no longer possible to live like that, and I set about rewriting our architecture. In order to cope with all the problems described above, it was enough to apply our favorite technique: replacing inheritance with a composition. This technique has helped us so many times that I advise you to use composition instead of inheritance wherever possible (well, or at least consider this option). If you create a basic abstract class with the thought “for each specific task I will have an heir overloading methods and properties”, immediately ask yourself “why don’t I register an instance of the class for each task, passing him different settings. ” And only if you are sure that the composition is not suitable here, use inheritance. If this and that is appropriate, always lean toward composition - this gives you a much more flexible and understandable architecture.

    In our situation:
    • instead of overloading a property that returns a marker action template, this property is affixed to the class instance
    • instead of overloading the methods sending letters / sms and performing additional logic, an arbitrary operation is put down on the class instance, which must be performed on the consumer. Moreover, the operation may be a combination of other operations
    • instead of overloading the method that forms Expression, the class instance is given a condition. In this case, the conditions can be combined through AND / OR

    Since, in addition to sending mailings, this entity can now perform any arbitrary operations on the consumer, it is incorrect to call it a mailing list. In fact, this is a class that does some abstract work on a given sample of consumers. Without inventing anything better, we began to call it triggers (in marketing they are called something like that, so the name is not bad). To be honest, it scared me a little that I introduced an extremely abstract entity into the system, which can be called DoSomeWorkOnSomeCustomers . But there was no sense in the specialization of triggers, so I decided not to bother with this, and, in principle, there are no big problems with understanding what a trigger is for clients.

    Registering a trigger looked something like this:
    Add(new Trigger(“Приглашение на сайт для пришедших через канал one-to-one”)
    	MarkerActionTemplateSystemName = “InvitationMarker”,
    	TriggerAction = new TriggerActionCombination(
    		new GeneratePasswordForCustomerTriggerAction(),
    		new SendEmailTriggerAction(“InvitationMailing”)),
    	TriggerCondition = new AndTriggerConditionSet(
    		new CustomerHasSubscripionCondition(),
    		new CustomerHasEmailTriggerCondition(),
    		new CustomerHadFirstActionOverChannelCondition(“OneToOne”)),

    TriggerAction's interface is extremely simple:
    public interface ITriggerAction
    	void Execute(
    		ModelContext modelContext, // класс для работы с БД 
    		Customer customer);

    The base class for trigger conditions is as follows:
    public class TriggerCondition
    	private readonly Func>> triggerExpressionBuilder;
    	public TriggerCondition(Func>> triggerExpressionBuilder)
    		if (triggerExpressionBuilder == null)
    			throw new ArgumentNullException("triggerExpressionBuilder");
    		this.triggerExpressionBuilder = triggerExpressionBuilder;
    	public Expression> GetExpression(ModelContext modelContext)
    		return triggerExpressionBuilder(modelContext, brand);
    	// Используется в Read Uncommitted транзакции для получения спиcка Id потребителей, подходящих под условие
    	public IQueryable ChooseCustomers(ModelContext modelContext, IQueryable customers)
    		if (modelContext == null)
    			throw new ArgumentNullException("modelContext");
    		if (customers == null)
    			throw new ArgumentNullException("customers");
    		var expression = GetExpression(modelContext);
    		return customers.Where(expression).ExpandExpressions();
    	// Используется в Serializable транзакции, для проверки, что потребитель все еще подходит под условие
    	public bool ShouldTrigger(ModelContext modelContext, Customer customer)
    		if (modelContext == null)
    			throw new ArgumentNullException("modelContext");
    		if (customer == null)
    			throw new ArgumentNullException("customer");
    		var expression = GetExpression(modelContext);
    		// Можно бы было просто вызывать expression.Evaluate(customer),
    		// но тогда для сложных условий выполнилось бы несколько запросов в БД вместо одного
    		return modelContext.Repositories.Get().Items
    			.Where(aCustomer => aCustomer == customer)
    			.Where(aCustomer => expression.Evaluate(aCustomer))
    For commonly used conditions, we created heirs from TriggerCondition , in which a specific Expression was built depending on the parameters passed to the constructor.

    All tired, start your triggers yourself

    Using the architecture described above, we set up a trigger in less than half an hour, by combining the conditions already written and TriggerActions. However, this was not enough for us. The next step, we wanted to completely exclude developers from the process of creating triggers. And how to do this in general terms, I realized a couple of months after the implementation of the previous version of the architecture. The trigger conditions were one-on-one similar to filtersthat we use in the admin area. Our filter system allows you to describe complex conditions, including queries to related entities, and also allows you to combine them through AND / OR. The filter forms Expression, with the help of which it is already possible to filter entities in the database. And for all this, UI and serialization have already been written. It only remained to add a couple of filters, which are often needed for triggers, but they did not make sense during normal work with the list of consumers (for example: “N days have passed since the action”). For TriggerActions, it was necessary to write a UI and a structure for storing them in the database, but here, in general, everything was clear. However, there were still small questions that had to be broken:
    • By this time, we began to register the sending of any letter as an action, and the action marker became redundant - we could already determine to whom we sent the letter, and in general we would like to get rid of issuing unnecessary actions wherever possible
    • in addition to simple triggers that performed a specific set of operations once on each consumer, we now have periodic triggers . It was necessary to figure out how to transfer all this to the database and at the same time allow the use of arbitrary markers
    • marketers come up with triggers not separately from each other, but as chains in which there are both triggers and operations performed by the consumer on the site (a letter asking to go to the site and do something → the consumer performs several operations on the site → bonus points are awarded and a letter is sent about this). If I didn’t immediately realize this, I would like to leave the groundwork for the future so that it would not be difficult to describe the dependencies between triggers and operations
    All three of these problems are related to how we determine whether a trigger is executed on a consumer or not. If you set up your own marker for each trigger and operation on the site, the task is greatly simplified, but I really did not want to produce unnecessary actions in the system. There was even an idea to force managers to create a filter in such a way that it was fully responsible for whether the action on the consumer can now be performed (and, accordingly, the trigger repetition rate would be described by the condition in the filter), but this approach is too error prone. After much painful reflection, I nevertheless came up with the idea of ​​how to track the execution of triggers without additional entities and without complicating the work of the manager.

    Need More Expressions

    Since the trigger performs an abstract operation step (formerly TriggerAction) on the consumer, and almost always this operation step is unique (for example, a certain letter is sent or a certain prize is issued only from this trigger), then logic can be checked out to check whether it was executed. Since the trigger may have several steps of the operation, the manager will need to choose which one is a marker (it makes no sense to check the execution of each step). However, it’s simple to implement in the operation step a method that returns Expression>It’s impossible, since in each step of the operation it would be necessary to form one Expression for one-time triggers, the other for periodic ones. Here we are saved by the fact that almost any operation on a user in our system gives him action. Accordingly, the operation step can filter out the actions that were issued to them. Most steps of the operation produce a specific action, and for them, the method that forms Expression to filter actions looks like this:
    public sealed override Expression> GetIsMarkerExpression(ModelContext modelContext)
    	return action => action.ActionTemplateId == ActionTemplateId;

    But, for example, at the step issuing the prize, it looks as follows:
    public override Expression> GetIsMarkerExpression(ModelContext modelContext)
    	IQueryablehabracut customerPrizes = modelContext.Repositories.Get().GetByPrizes(Prize);
    	// отфильтровываем действия, связанные с выдачей заданного приза
    	return action => customerPrizes.Any(prize => prize.CustomerActionId == action.Id);

    Also, I again applied my favorite replacement for inheritance with a composition, and instead of individual heirs for periodic and one-time triggers, I made a strategy that checks whether the trigger should be repeated on the current consumer. This strategy takes Expression>from the marker step of the trigger and using it forms Expression>, for additional verification of whether it is necessary to execute a trigger on a consumer. Here is the implementation for a one-time trigger:
    public override Expression> BuildShouldRepeatExpression(ModelContext modelContext, 
    Expression> isMarkerExpression)
    	var markerActions = modelContext.Repositories.Get().Items
    	return customer => !markerActions.Any(action => action.Customer == customer);

    But for the periodic:
    public override Expression> BuildShouldRepeatExpression(
    	ModelContext modelContext, Expression> isMarkerExpression)
    	var isInPeriodExpression = PeriodType.BuildIsInPeriodExpression(modelContext, PeriodValue);
    	var markerActions = modelContext.Repositories.Get().Items
    	var markerActionsInPeriod = markerActions.Where(isInPeriodExpression.ExpandExpressions());
    	if (MaxRepeatCount == null)
    		return customer => !markerActionsInPeriod.Any(action => action.Customer == customer);
    		return customer => 
    			!markerActionsInPeriod.Any(action => action.Customer == customer) &&
    			markerActions.Count() < MaxRepeatCount.Value;
    Here, it is supported not only once every N days, but also once a calendar month / year, therefore Expression, which checks whether the action is in a given period, is moved to the special PeriodType class. It also supports limiting the number of repetitions.

    The storage scheme for all this stuff in the database looks something like this:

    The essence of an OperationStepGroup with one field looks pretty strange, but it allows different entities (triggers, operations on the site, etc.) to refer to a group of records in a relational database. Moreover, later additional fields appeared in this entity, so everything is not so scary.

    Besides the fact that we got rid of unnecessary marker action templates, we can use IsMarkerExpression obtained from the marker steptrigger, in order to display statistics on the number of trigger operations. We can also add chains of triggers and operations (operations also use steps, one of which is marked as marker).

    As a result, the manager can start the trigger directly in the admin panel without the participation of the developer, although they often have to prompt them: the establishment of a new trigger is not an easy task, but such a price for the flexibility of this solution. A simpler solution would be less flexible, although we, of course, will have to work a lot to simplify the UI without losing the current flexibility of our architecture (for example, you can make Wizards for creating simple triggers).

    How it all looks in the UI, you can see here .

    Also popular now: