How we did the Sportmaster club program
If you visit our stores more often than once a year for sporting goods or clothing, most likely you have our club card (blue, silver or gold). My name is Maxim, I’m the deputy director of the department of software development, implementation and maintenance, and in this post we will talk with colleagues about the establishment of the Sportmaster’s club program, about the collection of rakes we collected in the process and how our club program differs from the usual discount cards of others trading networks.
The year 2004 was in the yard. What happened was Sportmaster’s club program and 27 rubles each. What was not - normal Internet in the field and stable communication channels at stores.
In those years, we ourselves wrote a loyalty system that could normally keep track of each user's bonus points. But since we already had a lot of stores then, in contrast to the data processing facilities, our entire bonus database was located in one file, which was simply sent to stores and serviced locally, and the changes for the day were returned. By the way, this was the primary reason that bonuses can be spent only the day after the purchase, and not the requirements of the business and the provision of customer returns - during the day all this simply did not have time to be updated and recounted properly.
In other words, at that time the bonuses could start to work and in general the fifth day - until this table with the newly calculated bonuses reaches all stores.
The cards themselves worked quite simply - each card simply accumulated a certain number of bonuses that its owner could spend joyfully and with feeling. That is, for the base it looked simple, roughly speaking, one card - one digit. With gold cards it was a little more complicated - there, besides the number with bonuses, there were also service points. This is when you bought a bike, and after six months you wanted to tighten the chain, check the brakes, the bell began to warm your soul and scare away passers-by and so on (or sharpen skates for the new season and fix the snowboard, for example).
At the same time, bonuses were calculated using the power of a solution based on natural intelligence - we had a special employee who wrote selections and made selections, then added bonuses in a special plate, and the system pulled this plate up. And yes, if this person suddenly decided to cheer a little or what else - some customers could be left without bonuses in these difficult days.
Of course, this situation was quite expensive for business, and indeed it was dangerous and unpredictable, and the business wanted to transfer the system to normal (automatic) rails.
First we decided to study the market. We realized that there are several excellent systems with great features and price tags that are superior to these features. Moreover, with such a licensing system, according to which it was supposed to collect bribes not just for the use of capacities or for the software itself, but for each active user in such a bonus system. What could not but rejoice the authors of the system. But our business was sad.
The second problem was also the fact that with the growth of the customer base and the number of stores, there was a situation where an increasingly hefty base had to be sent to more and more stores. And one fine day, this base simply stopped crawling through existing communication channels. It was also reviewed at all stores in a short period of time with a big creak.
The bases didn’t have time to shop, sometimes because of this the stores forgot to update them, and they got frank porridge - in the store A person bought something useful and received a bonus, and store B after a couple of days is still not aware that a person could and deduct bonuses from a new purchase.
And then Alexander Afanasyev (now the IT director of another company)figured out how to do all this yourself without buying third-party software. I collected from the business a number of requirements for this system on their part and registered new opportunities. At first it’s just in the form of pleasant features - for example, now bonuses are not just one number, but a whole complex system. You can give bonuses to a person for a birthday, you can separately give a bonus only for skis and related product items, you can offer bonuses only for Columbia brand products and at certain periods of time - and all this, combine, combine, combine.
Fortunately, the development of the network has reached the point where the Internet has already appeared almost everywhere, and it was possible to bring the solution online. That is, the work scheme has become like this - there is a store with a stable network channel, it contacts the database for the rest (and now the database is all the freshest and most tasty), takes the rest of the bonuses from the database and works with it already. And all this, while the buyer happily communicates with a friendly seller.
The third problem was the performance of bonuses burning operations. We have the same thing - you can earn points all year fervently, but on March 1 they always treacherously burn out if you did not manage to spend them.
The first version of the system (called CARDS) could normally take them into account, but when it switched to the bonus incinerator plant mode, problems started. After all, burning bonuses is a full passage through the entire base with changes. Given the size of the base, this could take 3-4 days. Moreover, in the process she terribly slowed down and stupid, because of which sometimes the burning of bonuses was interrupted, and it turned out that in some store, Comrade Petrov, who had come for new ping-pong balls, still had bonuses, and Sidorov, who had gone for unfortunately, no new great one.
We made a prototype somewhere in 3-4 days, then a couple of days we tested it on live checks. It turned out that the system is quite functional for itself, and you can use it to generate different bonus conditions and generate communication texts.
By the way, about communications - from the very beginning we made it so that the loyalty system itself at the right time formed the texts of communications with customers, extracting bonus points from the database, and sending them to customers ourselves. We had a lot of customers, so we used third-party providers at the time to send SMS.
Interaction with them went something like this:
So, about the prototype. In principle, the whole system was decided to be heavily reengineered, because initially it was sharpened only for bonuses, and not for online work, so it expectedly ceased to cope with the return. Moreover, it fell, of course, during moments of high load. That is, at the most delicious time for the store - New Year, March 8, February 23 and other pleasant dates.
The system falls -> the mood of the business falls -> the mood of everyone falls.
Together with a colleague, we rewrote the system according to the following principle.
Component 1. Preprocessing that gives the answer to stores as quickly as possible.
Component 2. Processing, the same magic box, difficult and cleverly crediting bonuses on commodity checks.
Component 3.Marketing, bringing it all together and forming communication texts.
Plus, we solved the problem of burning bonuses. The new system simply did not burn them. After all, if you do not force the system to burn bonuses - you have no problems with burning bonuses.
In the new version, the system simply stores the bonuses of each client in the database, but at some point in time it ceases to consider them active. That is, now there are always bonuses, but each with its own period of activity. Which, incidentally, allowed the introduction of more accurate and more urgent promotions and campaigns.
The old system actually just kept records of cards and bonuses on these cards. The new system does not prioritize a card, but a person’s account. We can identify it by phone number (this thing works for us from the very start, we were one of the first to implement authorization by phone number).
An additional feature of the new system is the so-called product bonuses, it works like this:
We showed this prototype in business work. Business gave the go-ahead.
We started writing the system on March 1, put it into operation on October 27, 2013 (we wrote together, yes). In fact, the planned delivery date was September 1, but the main counterparty of the system did not have time - retail stores. Stores did not have time for a number of reasons, plus not everyone updated the cash register software (and updating the cash register software on a rather large network is still a pain). Therefore, they postponed it, waited for them, and started on October 27.
They laid the main idea - neither the store, nor the cash register software anymore work with the logic of bonuses. The store now just sends the customer’s basket to the Center, the Center processes the whole thing, gives the store a calculation of bonuses.
Now bonuses are spread like this:
In general, it was possible to clog up all the old problems, and instead of new problems add new features.
Instead of backlog, we had this notebook of Alexander.
Since the new system was different from the old one not only technically, but also ideologically, we could not roll it somehow partially, into half of the stores or somehow. We just had to turn off the old one and turn on the new one instead.
It sounds good, but in fact it comes down to a couple of limitations.
Firstly, due to the large number of stores (1200+), we had to manage to do everything in 3 hours. While one store closes at midnight in one time zone, another has a completely different time, and then there is also a convenience store. In general, to convert all the data from the old system, feed the new one, start on three servers at once - 3 hours.
Pitfalls were like this:
We trained for a long time, hung scripts on the most indulge. After long tests and attempts to do everything as quickly as possible, we achieved the best result at 9 o’clock.
Which was slightly different from the planned figure of 3 hours.
Then we decided to first do the preprocessing, which kept the remains in ourselves. Raised the main server, he contacted the shops. At the same time, he did not know that the whole system had not risen yet, and at that time we were bravely rolling everything else.
But still, such a volume of data on standard machines could not be done in a timely manner.
And here it should be noted Oracle Exadata. The guys from Oracle made a special piece of hardware that works great with its own database, and even on flash drives. In general, a strong-willed decision was made to use Exadata. With its help on the tests, we mastered doing everything you need in 2 hours instead of 9 and realized - we need to take it.
Since we are meticulous guys, in the process of setting up and working, we scooped up a bunch of bugs and failed the support of Oracle with a margin. For example, there was one interesting bug - due to an error in the internal processing of the request, Oracle began to intensively consume TEMP. We noticed this in time, and threw him TEMP'ovyh files, it was very interesting when he got drunk. But since the piece of iron turned out to be very sensible and knowledgeable, she used 3 Tb TEMP with feeling for 10 minutes, realized that she was no more, and went to bed. I had to come up with workarounds.
On the one hand, it was cool that everything in terms of conversion we did in 2 hours. On the other hand, in the whole process of clean conversion, 2 hours, and we also planned:
At the same time, in each database there was a bunch of useful service information such as the same indexes that could help during the rebuild, but we scored on this and decided to rebuild everything again already on the battle servers.
We prepared with might and main. We were sleeping at work. We hung ourselves not only with scripts, but also with many metrics.
At 23.00 every day we started the process and monitored the metrics. We made the necessary changes, as a result, we set everything up so that nothing could go wrong at all.
Of course, on the launch day something went wrong.
To our honor, the cant was not on our side. Somewhere the network blinked stupidly. That is, you sit to yourself like that, you set everything up so that the mosquito not only doesn’t undermine its nose, but doesn’t even have time to think about it - and somewhere, someone just pulls the wrong cable.
Anyway, we managed to start the first server on time. The general deadline was 5 in the morning, by this time all servers should be cheerful and cheerful, because the first stores in the Far East open at 10 in their time.
So the last server started at 11 in the morning. But since we built the system in such a way that everything was isolated, everything worked fine.
Currently, 14 developers and 8 analysts are working on the club system. Considering all the goodies that we hung around with, this is no longer just a card that gives you a certain number of bonuses available for spending in stores.
We began to fully combine bonuses. The main criterion for a successful combination for the system is the maximum benefit for the buyer. There may be many utilities and promotions, for example:
We wrote an algorithm that receives a check from a store, runs over the product items in the check, applies all possible promotions and discounts to it on a given date and in a given city, and gives the result that is most beneficial for the user. Then he returns it all to the store. And there are still directions of development:
A grateful client can at this moment recall that he also wanted to buy socks, and asks to add them to the check. Of course, adding socks (or whatever) requires a complete recount.
But we will also deal with this. And in one of the future posts we will tell you the story of the creation of the site of the Sportmaster.
Then
The year 2004 was in the yard. What happened was Sportmaster’s club program and 27 rubles each. What was not - normal Internet in the field and stable communication channels at stores.
In those years, we ourselves wrote a loyalty system that could normally keep track of each user's bonus points. But since we already had a lot of stores then, in contrast to the data processing facilities, our entire bonus database was located in one file, which was simply sent to stores and serviced locally, and the changes for the day were returned. By the way, this was the primary reason that bonuses can be spent only the day after the purchase, and not the requirements of the business and the provision of customer returns - during the day all this simply did not have time to be updated and recounted properly.
In other words, at that time the bonuses could start to work and in general the fifth day - until this table with the newly calculated bonuses reaches all stores.
The cards themselves worked quite simply - each card simply accumulated a certain number of bonuses that its owner could spend joyfully and with feeling. That is, for the base it looked simple, roughly speaking, one card - one digit. With gold cards it was a little more complicated - there, besides the number with bonuses, there were also service points. This is when you bought a bike, and after six months you wanted to tighten the chain, check the brakes, the bell began to warm your soul and scare away passers-by and so on (or sharpen skates for the new season and fix the snowboard, for example).
At the same time, bonuses were calculated using the power of a solution based on natural intelligence - we had a special employee who wrote selections and made selections, then added bonuses in a special plate, and the system pulled this plate up. And yes, if this person suddenly decided to cheer a little or what else - some customers could be left without bonuses in these difficult days.
Of course, this situation was quite expensive for business, and indeed it was dangerous and unpredictable, and the business wanted to transfer the system to normal (automatic) rails.
First we decided to study the market. We realized that there are several excellent systems with great features and price tags that are superior to these features. Moreover, with such a licensing system, according to which it was supposed to collect bribes not just for the use of capacities or for the software itself, but for each active user in such a bonus system. What could not but rejoice the authors of the system. But our business was sad.
The second problem was also the fact that with the growth of the customer base and the number of stores, there was a situation where an increasingly hefty base had to be sent to more and more stores. And one fine day, this base simply stopped crawling through existing communication channels. It was also reviewed at all stores in a short period of time with a big creak.
The bases didn’t have time to shop, sometimes because of this the stores forgot to update them, and they got frank porridge - in the store A person bought something useful and received a bonus, and store B after a couple of days is still not aware that a person could and deduct bonuses from a new purchase.
And then Alexander Afanasyev (now the IT director of another company)figured out how to do all this yourself without buying third-party software. I collected from the business a number of requirements for this system on their part and registered new opportunities. At first it’s just in the form of pleasant features - for example, now bonuses are not just one number, but a whole complex system. You can give bonuses to a person for a birthday, you can separately give a bonus only for skis and related product items, you can offer bonuses only for Columbia brand products and at certain periods of time - and all this, combine, combine, combine.
Fortunately, the development of the network has reached the point where the Internet has already appeared almost everywhere, and it was possible to bring the solution online. That is, the work scheme has become like this - there is a store with a stable network channel, it contacts the database for the rest (and now the database is all the freshest and most tasty), takes the rest of the bonuses from the database and works with it already. And all this, while the buyer happily communicates with a friendly seller.
The third problem was the performance of bonuses burning operations. We have the same thing - you can earn points all year fervently, but on March 1 they always treacherously burn out if you did not manage to spend them.
The first version of the system (called CARDS) could normally take them into account, but when it switched to the bonus incinerator plant mode, problems started. After all, burning bonuses is a full passage through the entire base with changes. Given the size of the base, this could take 3-4 days. Moreover, in the process she terribly slowed down and stupid, because of which sometimes the burning of bonuses was interrupted, and it turned out that in some store, Comrade Petrov, who had come for new ping-pong balls, still had bonuses, and Sidorov, who had gone for unfortunately, no new great one.
New version of the system
We made a prototype somewhere in 3-4 days, then a couple of days we tested it on live checks. It turned out that the system is quite functional for itself, and you can use it to generate different bonus conditions and generate communication texts.
By the way, about communications - from the very beginning we made it so that the loyalty system itself at the right time formed the texts of communications with customers, extracting bonus points from the database, and sending them to customers ourselves. We had a lot of customers, so we used third-party providers at the time to send SMS.
Interaction with them went something like this:
- the provider understood that he was a major client, and began to rejoice
- a major client in the form of us specified whether the provider would actually handle such mailings
- the provider said that it will master, of course
- the provider received the task of sending a huge amount of SMS in a short period of time and decided to lie down to rest
So, about the prototype. In principle, the whole system was decided to be heavily reengineered, because initially it was sharpened only for bonuses, and not for online work, so it expectedly ceased to cope with the return. Moreover, it fell, of course, during moments of high load. That is, at the most delicious time for the store - New Year, March 8, February 23 and other pleasant dates.
The system falls -> the mood of the business falls -> the mood of everyone falls.
Together with a colleague, we rewrote the system according to the following principle.
Component 1. Preprocessing that gives the answer to stores as quickly as possible.
Component 2. Processing, the same magic box, difficult and cleverly crediting bonuses on commodity checks.
Component 3.Marketing, bringing it all together and forming communication texts.
Plus, we solved the problem of burning bonuses. The new system simply did not burn them. After all, if you do not force the system to burn bonuses - you have no problems with burning bonuses.
In the new version, the system simply stores the bonuses of each client in the database, but at some point in time it ceases to consider them active. That is, now there are always bonuses, but each with its own period of activity. Which, incidentally, allowed the introduction of more accurate and more urgent promotions and campaigns.
The old system actually just kept records of cards and bonuses on these cards. The new system does not prioritize a card, but a person’s account. We can identify it by phone number (this thing works for us from the very start, we were one of the first to implement authorization by phone number).
An additional feature of the new system is the so-called product bonuses, it works like this:
- each product has attributes (name, product category, size, color, sport, other, other, other).
- the system combines these attributes, forming a logical condition for accruing bonuses.
- when a check arrives, this condition is always checked.
We showed this prototype in business work. Business gave the go-ahead.
We started writing the system on March 1, put it into operation on October 27, 2013 (we wrote together, yes). In fact, the planned delivery date was September 1, but the main counterparty of the system did not have time - retail stores. Stores did not have time for a number of reasons, plus not everyone updated the cash register software (and updating the cash register software on a rather large network is still a pain). Therefore, they postponed it, waited for them, and started on October 27.
System ideology
They laid the main idea - neither the store, nor the cash register software anymore work with the logic of bonuses. The store now just sends the customer’s basket to the Center, the Center processes the whole thing, gives the store a calculation of bonuses.
Now bonuses are spread like this:
- First of all, bonuses are spread out all over the check evenly, for all commodity items. This is useful for analytics, and helps in case of return of goods.
- We introduced the concept of priority bonuses. Bonuses are commodity, there are bonuses for birthdays, which have a short validity period, there are regular ones (the most tenacious). Therefore, we first write off specific bonuses. That is, a person came for skiing - we will primarily write off the bonuses that he has on skis. And it turns out that he came for skiing, we wrote off regular bonuses. A week later he will come for a jacket, and we will give him a man, you only have bonuses for skiing. Do you want to ski? The same thing with purchases during birthday periods, first write off them, and then regular ones.
- We carry back office operations and the front. Now, stores that come with requests do not affect the work and performance of the service that calculates bonuses, and vice versa
In general, it was possible to clog up all the old problems, and instead of new problems add new features.
Instead of backlog, we had this notebook of Alexander.
Launching a new version of the system
Since the new system was different from the old one not only technically, but also ideologically, we could not roll it somehow partially, into half of the stores or somehow. We just had to turn off the old one and turn on the new one instead.
It sounds good, but in fact it comes down to a couple of limitations.
Firstly, due to the large number of stores (1200+), we had to manage to do everything in 3 hours. While one store closes at midnight in one time zone, another has a completely different time, and then there is also a convenience store. In general, to convert all the data from the old system, feed the new one, start on three servers at once - 3 hours.
Pitfalls were like this:
- The system was cut immediately on the entire network. If everything is good everywhere - everything works all over the network. If something falls, yes, it falls all over the network.
- When you turn on the new system, it must contain all the data that was in the old system at the time the stores closed and the latest bonus was issued. We launched into 4 countries at once. The database was more than a terabyte and stored hundreds of billions of records.
- At 23.00 we had to turn off the system. Convert everything. Pour into the new system. Enable all. In this case, everything should work.
We trained for a long time, hung scripts on the most indulge. After long tests and attempts to do everything as quickly as possible, we achieved the best result at 9 o’clock.
Which was slightly different from the planned figure of 3 hours.
Then we decided to first do the preprocessing, which kept the remains in ourselves. Raised the main server, he contacted the shops. At the same time, he did not know that the whole system had not risen yet, and at that time we were bravely rolling everything else.
But still, such a volume of data on standard machines could not be done in a timely manner.
And here it should be noted Oracle Exadata. The guys from Oracle made a special piece of hardware that works great with its own database, and even on flash drives. In general, a strong-willed decision was made to use Exadata. With its help on the tests, we mastered doing everything you need in 2 hours instead of 9 and realized - we need to take it.
Since we are meticulous guys, in the process of setting up and working, we scooped up a bunch of bugs and failed the support of Oracle with a margin. For example, there was one interesting bug - due to an error in the internal processing of the request, Oracle began to intensively consume TEMP. We noticed this in time, and threw him TEMP'ovyh files, it was very interesting when he got drunk. But since the piece of iron turned out to be very sensible and knowledgeable, she used 3 Tb TEMP with feeling for 10 minutes, realized that she was no more, and went to bed. I had to come up with workarounds.
On the one hand, it was cool that everything in terms of conversion we did in 2 hours. On the other hand, in the whole process of clean conversion, 2 hours, and we also planned:
- reload all the data from the servers of the old system to exadata, because it counts everything wildly fast.
- Convert data from old structures to new.
- Pour all this converted good into three different servers.
At the same time, in each database there was a bunch of useful service information such as the same indexes that could help during the rebuild, but we scored on this and decided to rebuild everything again already on the battle servers.
Training
We prepared with might and main. We were sleeping at work. We hung ourselves not only with scripts, but also with many metrics.
At 23.00 every day we started the process and monitored the metrics. We made the necessary changes, as a result, we set everything up so that nothing could go wrong at all.
Of course, on the launch day something went wrong.
To our honor, the cant was not on our side. Somewhere the network blinked stupidly. That is, you sit to yourself like that, you set everything up so that the mosquito not only doesn’t undermine its nose, but doesn’t even have time to think about it - and somewhere, someone just pulls the wrong cable.
Anyway, we managed to start the first server on time. The general deadline was 5 in the morning, by this time all servers should be cheerful and cheerful, because the first stores in the Far East open at 10 in their time.
So the last server started at 11 in the morning. But since we built the system in such a way that everything was isolated, everything worked fine.
Now
Currently, 14 developers and 8 analysts are working on the club system. Considering all the goodies that we hung around with, this is no longer just a card that gives you a certain number of bonuses available for spending in stores.
We began to fully combine bonuses. The main criterion for a successful combination for the system is the maximum benefit for the buyer. There may be many utilities and promotions, for example:
- the user has accumulated a good number of regular bonuses;
- plus now the period when there is an action on a particular brand;
- plus now discounts also on a specific product group and subgroup;
- and there may also be a discount in a particular city or in a particular store.
We wrote an algorithm that receives a check from a store, runs over the product items in the check, applies all possible promotions and discounts to it on a given date and in a given city, and gives the result that is most beneficial for the user. Then he returns it all to the store. And there are still directions of development:
- Development of a mechanism for launching complex multi-way marketing campaigns, including mailing, providing bonuses, discounts and personalizing offers for a client
- Connection of new communication channels, such as instant messengers, social networks, etc.
A grateful client can at this moment recall that he also wanted to buy socks, and asks to add them to the check. Of course, adding socks (or whatever) requires a complete recount.
But we will also deal with this. And in one of the future posts we will tell you the story of the creation of the site of the Sportmaster.