Lamoda Inside: Why Online Store 300 Engineers
Hi, Habr! My name is Valentine, I am CTO in Lamoda, where I have been working almost since the company was founded. All these years, we ran forward so fast with the whole team that it was not possible to stop a bit and tell about ourselves. I think the time has come.
It may seem that Lamoda is one of the pioneers of the Russian Internet, but we are only seven years old. Since our founding in 2011, our company has grown from 11 employees to more than five thousand. Every month, more than 10 million people visit our website. In fact, we were a start-up newcomer to well-established Russian IT, and in the end, in such a short time, we were able to catch up and outperform many well-deserved guys.
I hope that we will gradually tell you about our most useful and interesting achievements, failures, experience and what tasks our team faces every day. We will consider this post our acquaintance.
In 2011, we independently developed only content preparation and purchase of stock for sale, everything else was outsourced. Now we do everything ourselves. We supervise the operation of our own five-level warehouse the size of a football field, three contact centers and delivery to customers. Despite the fact that Russia has a very high IT culture, large companies are only beginning to dive into how to build such an infrastructure, and Lamoda has been successfully following this path for many years.
So, what is the technical backstage of Lamoda? In fact, these are five large divisions:
This story is about the guys who develop an e-commerce platform, and in their free time they drive on mots and take excursions to craft beer bars.
It is in the development department of e-commerce platforms that create everything that Lamoda customers use every day: the desktop and mobile version of the site and applications for iOS and Android. The complexity of the work is not in the implementation of functionality for the user; the team needs as quickly as possible, but without loss of quality and stability, roll out new features and support projects in four countries: Russia, Kazakhstan, Ukraine and Belarus. Today, the staff of this department is able to start the service every week, if necessary, and work under the motto: “Everything is in the GO”.
The teams are mainly divided by platform (desktop, mobile site, mobile apps), there are also four commands for backend services. Each team includes front and back-end developers, analysts, testers and a product manager.
In order to deliver changes as often as possible, we practice this approach to development iterations: at the beginning of the sprint we put forward a hypothesis, and releasing new features in production, test it, thereby understanding whether the innovation benefits the product and whether life of Lamoda users becomes better. For example, not so long ago we learned how to show how much of a product viewed by a person was purchased lately. To do this, we took the product challenge, found the MVP in it and determined the fastest implementation strategy. Before rolling out everything in production, we determined that the criterion of success would be an increase in conversion. According to the results of the A / B test, the conversion is higher in the group where the feature was implemented.
Some time ago, a massive and rapid transition to microservices began in Lamoda. What does this give us? The first is a low entry threshold for a new developer or specialist from another team. The second is the easy support and change of microservice systems so that the workflow is filled with interesting things, not just pain. But a small number of monoliths (for example, the system that is responsible for the distribution of orders) still lives in our country, and it is difficult and inconvenient to get rid of them at the moment.
Everyone knows that there is no work without fayl. Each time, solving the problem and running the code, we hope that here it is, happiness, and then not - again experience. Speaking of experience, it is worth noting that employees of the e-commerce platform development department, as true fighters, are able to literally assemble Lamoda from scratch. It happened that due to incorrect network settings, our cluster decided that it was no longer a cluster, and refused to exist. We were lucky that it was at night, and in four hours we managed to breathe life into Lamoda. We have other stories.
Timur Nurutdinov, Head of E-commerce Development Platform:
Before starting work on the new functionality, we, as usual, sat down to assess what resources we need. Four teams were involved in the project. We took into account the priorities of other tasks, the schedule of colleagues’s vacations and labor costs. As a result, we got 32 weeks.
Eight months to implement one feature. That sounds crazy. With the help of simple changes, we managed to reduce time to market to 4 weeks, and this is what we did.
The trick of platform teams is that they are able to do both the front and the back. So work and in our department. But on the back there is a need to make changes to many integrated systems, and the competencies of the platform team do not allow this. We started the Sizes project concerning the provision of the client with the most detailed dimensional grids, and did not want to wait. To begin with, we had to figure out which systems need to be changed, then we put together a small team with suitable competencies. So we removed the block on waiting for resources from other platform commands and received a grocery command. As for the tasks, we acted with our proven method - we broke the big tasks into smaller ones, made them, rolled them out and tested our hypotheses on users to understand whether we were moving in the right direction.
Automation guys don't have time to be bored, and the puzzles here are not trivial. For example, how to fill the site with millions of products with equally high quality content (photo studio automation), how to process and take into account all orders from the site, taking into account hundreds of marketplace partners and four CIS countries, how to assemble an order in a five-hour warehouse, how to realize delivery to the client the next day at the 15-minute interval chosen by him in 600 cities in Russia only. And for dessert, here they serve the sale of this whole farm to B2B partners and the Marketplace-direction.
The work is carried out mainly in PHP, for warehouse automation we use Java plus Docker / Kubernetes, Atlassian stack, PostgreSQL, RabbitMQ.
We have a batch system in our department for planning sprints: 60% is a project batch, 20% is technical debt, priority bugs are given 10% of the sprint, and 10% that something will definitely come from outside. Among other things, backlog grooming, online planning poker, stand-up, retro, code review 360, collecting and analyzing basic metrics, conducting monitoring (Prometheus, Grafana, Icinga, Kibana), in general, everything is like in the besthouses of Paris development teams.
Here are a couple of funny stories from Pavel Savelyev, head of business process automation.
It is impossible to test and take everything into account, because in every business process people are involved in one way or another. And people, as you know, are intelligent beings and always try to come up with cunning pieces that will make their lives easier for them. But when these very notions go against the described business process, funny stories happen.
Once we noticed that the system responsible for the distribution of goods in stock ,gets a hundred times more scans in a minute than usual. It turned out that the warehouse staff found the hack system and decided to make their work easier. Leaving for lunch, they clamped the button on the scanner so that they would not be thrown out of the user session. And this their work hack would work further, but in one of the totes (a special box for goods in the warehouse) there were many small items. The scanner, like Maxim's machine gun, handled goods from the ill-fated totah, which led to a sharp jump in load, system glitches and the detection of an obvious bug from the developers. Bug we, of course, fixed, but I think that the warehouse staff will not let us get bored and come up with something new.
The second case also occurred in stock. This story is called "43 funny t-shirts." It is not always immediately possible to recognize the complexity of the algorithm, especially when you solve the backpack problem and you need to optimally place in a certain amount of N items (the task of three-dimensional packaging). It turned out that if 43 perfectly identical T-shirts come to our warehouse, the system responsible for packing goods generates so many distribution combinations for a given case that it simply does not have enough memory. We revised the algorithm and we are no longer afraid of the same T-shirts - but what if the packaging gets hundreds of pairs of socks that manufacturers decide to sell one at a time? This is worth thinking about ...
Changes regarding analytics at Lamoda are long overdue, and this year we started to unite disparate analytics departments with our analytical infrastructure and habits into one big department. What for? The main reason is that employees who are in separate teams of analysts often do the same work, but in different ways, and then it is not quite clear what data to focus on. Different data is generally normal, because the teams come from different tasks and prerequisites, but you need to spend a lot of time to understand them.
Department employees are true evangelists that all decisions in a company should be made on the basis of data, so every day they enthusiastically study business phenomena in data, analyze and extract value from data, assessing how they can be applied. The main tool is SQL, as well as Spark, Hadoop, Python for data analysis, Excel, SAP BusinessObjects for reporting and Tableau for visualization.
One of the important tasks of the team is to personalize customer experience: we create a solution where each user will be shown the most relevant list of products and offers for him, and all our services will individually adjust to each individual client, rather than to the group, as is done now.
Sergey Gilev, Head of Data and Analytics:
The department of analytics currently faces two major tasks: the first is to consolidate a heterogeneous economy, which we had before the merger. For further effective work, we need common metrics, analytical infrastructure and processes. The second goal is a project to create analytical dashboards that describe the "health" of a specific process or the entire company. Thus, we strive to significantly improve the availability of data for decision makers and instill in all of our approach to work: in any incomprehensible situation, be guided by data.
The gradual abandonment of outsourcing led us to the fact that it was time to put into operation our own warehouse. In addition to all the training on automating operational processes in the new warehouse, we set ourselves the goal of not reducing the range and in no way stopping the sale. Our specialists have developed a solution based on the creation of additional “virtual” warehouses. Thus, throughout the relocation, we had three types of warehouses: old, new and “on the road”. Since the transportation of goods was carried out gradually by groups of trucks, the stock, which turned out to be in the next batch, moved to a “virtual” warehouse. We had a schedule of loading and unloading, so we knew exactly how long the stock would be on the way and oriented customers to the correct date of delivery of the order.
We also invented and implemented a clever algorithm that allowed us to balance the speed of the move and organize the receipt of all the goods of the order with one delivery: when a person ordered several goods that could physically be in different warehouses, we tried to organize a complete assembly of the order in any one warehouse, and the advantage was given to the partner's warehouse, since the assembly process there was already debugged.
The work on launching our own warehouse was in full swing for three months, during which not a single client was hurt.
Several years ago we were afraid of Black Friday as a scary monster. We had no idea how our systems would react to such a flow of orders. But the constant work on refactoring and infrastructure development has stabilized our systems and made them as predictable as possible. The last “Black Friday” was the most boring day of the year. Specialists of key systems and DevOps just sat at the table, played video games or watched movies and watched the state of our infrastructure with just a glance. But the preparation for this day goes a little differently.
We make a schedule of load tests based on business forecasts, then the developers and administrators set up systems for passing tests. Several months of testing, crashes, corrections, and only after that we believe that we are ready for a wave of users and orders.
War room - our operational center on Black Friday. The
decisions we make to survive on Black Friday depend on the bottlenecks we face during testing. A few years ago, we physically replaced the network switches to eliminate the bandwidth problem. Another action that we usually perform in order to reduce the load is to turn off subsystems that are not critical.
All these years we have been trying to continuously improve and simplify our workflow, collecting feedback from the company's employees, establishing bottom-up channels and fully supporting fresh ideas and concepts.
Someone will say that Lamoda is an absolute zoo of technologies and systems. We ourselves often joke about this topic and say: "It is better to ask what we do not use." But in this matter, the essential fact is that we have a constant evolution of the stack and technology, and at the same time there is no thoughtless choice of them. In this, we are helped by the Architecture review of each new service and project, a reference to the existing expertise of employees, as well as the maintenance of Technology Radar , the details and arguments for which we will be happy to tell in the next post. And also we are happy to talk about this topic.
I hope that we will gradually tell you about our most useful and interesting achievements, failures, experience and what tasks our team faces every day. We will consider this post our acquaintance.
In 2011, we independently developed only content preparation and purchase of stock for sale, everything else was outsourced. Now we do everything ourselves. We supervise the operation of our own five-level warehouse the size of a football field, three contact centers and delivery to customers. Despite the fact that Russia has a very high IT culture, large companies are only beginning to dive into how to build such an infrastructure, and Lamoda has been successfully following this path for many years.
So, what is the technical backstage of Lamoda? In fact, these are five large divisions:
- development department (business process automation department and online store development department)
- IT support department (infrastructure and security)
- ERP system implementation and support department
- service support department
- data and analytics department
All in GO
This story is about the guys who develop an e-commerce platform, and in their free time they drive on mots and take excursions to craft beer bars.
It is in the development department of e-commerce platforms that create everything that Lamoda customers use every day: the desktop and mobile version of the site and applications for iOS and Android. The complexity of the work is not in the implementation of functionality for the user; the team needs as quickly as possible, but without loss of quality and stability, roll out new features and support projects in four countries: Russia, Kazakhstan, Ukraine and Belarus. Today, the staff of this department is able to start the service every week, if necessary, and work under the motto: “Everything is in the GO”.
The teams are mainly divided by platform (desktop, mobile site, mobile apps), there are also four commands for backend services. Each team includes front and back-end developers, analysts, testers and a product manager.
In order to deliver changes as often as possible, we practice this approach to development iterations: at the beginning of the sprint we put forward a hypothesis, and releasing new features in production, test it, thereby understanding whether the innovation benefits the product and whether life of Lamoda users becomes better. For example, not so long ago we learned how to show how much of a product viewed by a person was purchased lately. To do this, we took the product challenge, found the MVP in it and determined the fastest implementation strategy. Before rolling out everything in production, we determined that the criterion of success would be an increase in conversion. According to the results of the A / B test, the conversion is higher in the group where the feature was implemented.
Some time ago, a massive and rapid transition to microservices began in Lamoda. What does this give us? The first is a low entry threshold for a new developer or specialist from another team. The second is the easy support and change of microservice systems so that the workflow is filled with interesting things, not just pain. But a small number of monoliths (for example, the system that is responsible for the distribution of orders) still lives in our country, and it is difficult and inconvenient to get rid of them at the moment.
We collect Lamoda from scratch without SMS and registration
Everyone knows that there is no work without fayl. Each time, solving the problem and running the code, we hope that here it is, happiness, and then not - again experience. Speaking of experience, it is worth noting that employees of the e-commerce platform development department, as true fighters, are able to literally assemble Lamoda from scratch. It happened that due to incorrect network settings, our cluster decided that it was no longer a cluster, and refused to exist. We were lucky that it was at night, and in four hours we managed to breathe life into Lamoda. We have other stories.
Timur Nurutdinov, Head of E-commerce Development Platform:
Before starting work on the new functionality, we, as usual, sat down to assess what resources we need. Four teams were involved in the project. We took into account the priorities of other tasks, the schedule of colleagues’s vacations and labor costs. As a result, we got 32 weeks.
Eight months to implement one feature. That sounds crazy. With the help of simple changes, we managed to reduce time to market to 4 weeks, and this is what we did.
The trick of platform teams is that they are able to do both the front and the back. So work and in our department. But on the back there is a need to make changes to many integrated systems, and the competencies of the platform team do not allow this. We started the Sizes project concerning the provision of the client with the most detailed dimensional grids, and did not want to wait. To begin with, we had to figure out which systems need to be changed, then we put together a small team with suitable competencies. So we removed the block on waiting for resources from other platform commands and received a grocery command. As for the tasks, we acted with our proven method - we broke the big tasks into smaller ones, made them, rolled them out and tested our hypotheses on users to understand whether we were moving in the right direction.
Warehouse automation and 15-minute delivery intervals
Automation guys don't have time to be bored, and the puzzles here are not trivial. For example, how to fill the site with millions of products with equally high quality content (photo studio automation), how to process and take into account all orders from the site, taking into account hundreds of marketplace partners and four CIS countries, how to assemble an order in a five-hour warehouse, how to realize delivery to the client the next day at the 15-minute interval chosen by him in 600 cities in Russia only. And for dessert, here they serve the sale of this whole farm to B2B partners and the Marketplace-direction.
The work is carried out mainly in PHP, for warehouse automation we use Java plus Docker / Kubernetes, Atlassian stack, PostgreSQL, RabbitMQ.
We have a batch system in our department for planning sprints: 60% is a project batch, 20% is technical debt, priority bugs are given 10% of the sprint, and 10% that something will definitely come from outside. Among other things, backlog grooming, online planning poker, stand-up, retro, code review 360, collecting and analyzing basic metrics, conducting monitoring (Prometheus, Grafana, Icinga, Kibana), in general, everything is like in the best
Here are a couple of funny stories from Pavel Savelyev, head of business process automation.It is impossible to test and take everything into account, because in every business process people are involved in one way or another. And people, as you know, are intelligent beings and always try to come up with cunning pieces that will make their lives easier for them. But when these very notions go against the described business process, funny stories happen.
Once we noticed that the system responsible for the distribution of goods in stock ,gets a hundred times more scans in a minute than usual. It turned out that the warehouse staff found the hack system and decided to make their work easier. Leaving for lunch, they clamped the button on the scanner so that they would not be thrown out of the user session. And this their work hack would work further, but in one of the totes (a special box for goods in the warehouse) there were many small items. The scanner, like Maxim's machine gun, handled goods from the ill-fated totah, which led to a sharp jump in load, system glitches and the detection of an obvious bug from the developers. Bug we, of course, fixed, but I think that the warehouse staff will not let us get bored and come up with something new.
The second case also occurred in stock. This story is called "43 funny t-shirts." It is not always immediately possible to recognize the complexity of the algorithm, especially when you solve the backpack problem and you need to optimally place in a certain amount of N items (the task of three-dimensional packaging). It turned out that if 43 perfectly identical T-shirts come to our warehouse, the system responsible for packing goods generates so many distribution combinations for a given case that it simply does not have enough memory. We revised the algorithm and we are no longer afraid of the same T-shirts - but what if the packaging gets hundreds of pairs of socks that manufacturers decide to sell one at a time? This is worth thinking about ...
In any incomprehensible situation, be guided by data.
Changes regarding analytics at Lamoda are long overdue, and this year we started to unite disparate analytics departments with our analytical infrastructure and habits into one big department. What for? The main reason is that employees who are in separate teams of analysts often do the same work, but in different ways, and then it is not quite clear what data to focus on. Different data is generally normal, because the teams come from different tasks and prerequisites, but you need to spend a lot of time to understand them.
Department employees are true evangelists that all decisions in a company should be made on the basis of data, so every day they enthusiastically study business phenomena in data, analyze and extract value from data, assessing how they can be applied. The main tool is SQL, as well as Spark, Hadoop, Python for data analysis, Excel, SAP BusinessObjects for reporting and Tableau for visualization.
One of the important tasks of the team is to personalize customer experience: we create a solution where each user will be shown the most relevant list of products and offers for him, and all our services will individually adjust to each individual client, rather than to the group, as is done now.
Sergey Gilev, Head of Data and Analytics:The department of analytics currently faces two major tasks: the first is to consolidate a heterogeneous economy, which we had before the merger. For further effective work, we need common metrics, analytical infrastructure and processes. The second goal is a project to create analytical dashboards that describe the "health" of a specific process or the entire company. Thus, we strive to significantly improve the availability of data for decision makers and instill in all of our approach to work: in any incomprehensible situation, be guided by data.
The story of a big move: how we launched our own warehouse imperceptible to customers
The gradual abandonment of outsourcing led us to the fact that it was time to put into operation our own warehouse. In addition to all the training on automating operational processes in the new warehouse, we set ourselves the goal of not reducing the range and in no way stopping the sale. Our specialists have developed a solution based on the creation of additional “virtual” warehouses. Thus, throughout the relocation, we had three types of warehouses: old, new and “on the road”. Since the transportation of goods was carried out gradually by groups of trucks, the stock, which turned out to be in the next batch, moved to a “virtual” warehouse. We had a schedule of loading and unloading, so we knew exactly how long the stock would be on the way and oriented customers to the correct date of delivery of the order.
We also invented and implemented a clever algorithm that allowed us to balance the speed of the move and organize the receipt of all the goods of the order with one delivery: when a person ordered several goods that could physically be in different warehouses, we tried to organize a complete assembly of the order in any one warehouse, and the advantage was given to the partner's warehouse, since the assembly process there was already debugged.
The work on launching our own warehouse was in full swing for three months, during which not a single client was hurt.
Preparing for Black Friday or how we survive shopping
Several years ago we were afraid of Black Friday as a scary monster. We had no idea how our systems would react to such a flow of orders. But the constant work on refactoring and infrastructure development has stabilized our systems and made them as predictable as possible. The last “Black Friday” was the most boring day of the year. Specialists of key systems and DevOps just sat at the table, played video games or watched movies and watched the state of our infrastructure with just a glance. But the preparation for this day goes a little differently.
We make a schedule of load tests based on business forecasts, then the developers and administrators set up systems for passing tests. Several months of testing, crashes, corrections, and only after that we believe that we are ready for a wave of users and orders.
War room - our operational center on Black Friday. The
decisions we make to survive on Black Friday depend on the bottlenecks we face during testing. A few years ago, we physically replaced the network switches to eliminate the bandwidth problem. Another action that we usually perform in order to reduce the load is to turn off subsystems that are not critical.
Results
All these years we have been trying to continuously improve and simplify our workflow, collecting feedback from the company's employees, establishing bottom-up channels and fully supporting fresh ideas and concepts.
Someone will say that Lamoda is an absolute zoo of technologies and systems. We ourselves often joke about this topic and say: "It is better to ask what we do not use." But in this matter, the essential fact is that we have a constant evolution of the stack and technology, and at the same time there is no thoughtless choice of them. In this, we are helped by the Architecture review of each new service and project, a reference to the existing expertise of employees, as well as the maintenance of Technology Radar , the details and arguments for which we will be happy to tell in the next post. And also we are happy to talk about this topic.