June 13, 2019 at 10:03Why do we at Leroy Merlin need our own Russian development department for 200 people
Hello! I am Valery Laptev, Head of LM Development in Russia. For two years I needed to raise a huge department, and it was a rather interesting experience.
The fact is that Leroy Merlin is in many countries. The parent company in France is called ADEO. They write code for France, Italy, Spain and Russia. Our business models are different: if we keep the lowest prices on the Russian market (below all competitors in monitoring), then in Europe everything is different. In fact, the sea is different - from the characteristics of the locale to other legislation. There are features of the infrastructure of Russia (the same very large delays to Khabarovsk) and another life cycle for placing an order. All this gives rise to such hellish code consisting of huge IF blocks:
Two years ago, we had 60 stores and many, many Wishlist features. They rolled in about six months and not always correctly. The last straw after a bunch of features rejected by low priority was a request to enter a row field in the order, so that we would parse it later. This was necessary for the features of delivery in Russia, since the country is larger than other countries of presence of LM. They refused us this, or rather, they said that it would be somewhere in seven to eight months.
The half-year cycle of the leisurely parent company did not suit us. Naturally, we suggested writing our own code, submitting it to the review and waiting for implementation ... True, nothing good came of this.
The story is very simple: despite our dozens of stores, we are not the first in the world. There are the needs of France (where the parent organization is), there are the needs of other countries. They are prioritized according to possible profit or cost reduction for a group of companies and are executed in this order. That is, our features are made almost never, except very lucky and in some kind of sprint someone will finish their part earlier and go to disassemble the bottom of the backlog.
The second feature is that when you roll any feature into the master branch (here in this Legacy IF-IF-IF code), you need to check how it behaves in other countries. Let me quote 441869 from Bash:
In general, when we introduce something for the Russian box office, somewhere in Brazil, someone may rebound.
This means very large test coverage, long pre-release procedures, and generally reluctance to maintain code that is growing rapidly. Therefore, the less features - the better. But you hold on there.
The position as a whole is very understandable, and I would do just that on the site of the parent company. Because it is rational.
The first approach was on the forehead: we suggested opening a code for us to do pull requests there. It was supposed that there would be about 10 developers sitting there who would quickly and quickly code business critical functions, give them to the French, and they would test according to their own procedure, and everything would be fine.
Actually these people already were: we were engaged in cutting out the extra business logic that we got from other countries, such as different cumulative discounts and unusual stocks (which are simply impossible with our business model), and put dummies in this place.
The French at ADEO wanted a release cycle for rolling and installing new versions through themselves. Accepted our offer about one branch for experiments.
Gave access, began to take the code for the review. It turned out to be slow anyway. Rollouts for several months - this is not the case. Well won a few weeks, but still the process did not fit.
Then, for six months, a feature important to us in the content part (managing product data that the client sees) did not come out. We sat for six months without a release. Either their feature didn’t go, or the tests didn’t pass, then they didn’t realize exactly what we needed. As a result, we sat on a key system for a long time without updates. There were NodeJS + PostgreSQL + Couchbase + Elasticsearch + Angular at the front. In the code, due to a number of archaeological layers, things have been encountered, such as the misuse of the SQL database and non-relational database. In one of the places, a huge piece of master data was taken, inserted into one field of the SQL database, and then split into entities in the NoSQL database. On one page of the site with the display of goods there were tens and hundreds of such requests. With further reading of this legacy, hair on different parts of the body began to move. We understood
The first idea was to do features together. On the spot, that is, in France. The four of us went to France and began to sit next to our colleagues to understand everything together and do as we needed.
It didn’t work. All the same, everything was very slow.
The second idea was to fork all that already exists and gradually finish it. We sat down by the architectural committee, evaluated the prospects, calculated the approximate development plans and realized that we need to choose a different method. Specifically, yes, to fork, but not to develop the existing code, but to break the monolith into parts and put microservices where it is needed.
That is, we planned to rewrite entire blocks of logic in the form of our code. And for this they began to switch part of the services to what could be done with us. We started to recruit developers. Then we completely followed the stack of the parent company - Java, Spring and everything is nearby, instead of Couchbase was Mongo (this is a similar NoSQL-base).
As the project developed, we realized that we needed to do a lot of things in our own way (because it’s faster and easier so as not to support legacy that we don’t need, in particular), and began to expand to other technologies. Then there were Java 7, Wildfly (formerly JBoss) and SVN. We asked why they are not migrating to Java 8, GIT, and Tomcat. It turned out they would not mind, but after a couple of years. In the meantime, dear ones, write on the old stack. So we word for word and decided to separate completely. The business sorted out the question, what are the pros and cons, and fully supported it.
Almost immediately threw about 30 percent of the code, wrote a lot of their microservices around. From the fact that they did not touch, it is almost the entire core of the transaction of business processes for money.
Naturally, the first thing we thought about was how to distinguish between areas of development. I would also talk about this separately, but in general terms the scheme is as follows:
Horizontal is a business area. For example, everything related to customer relations (the first green cell) is one area, and there is a set of applications from one department. This separation of the objective function creates several duplicates of the code in different places and requires a good corporate bus (again, a separate story), but it allows you to clearly find the ends and solve the problem until the result. Looking at this after almost three years, I can say that the architecture was chosen correctly, but if I knew what I know now, I would make a couple of changes.
Now we have come to the conclusion that in the general structure there are many feature teams that make up large product teams. At the same time, the product team includes not only the developers themselves, but also designers and business representatives. Since the ultimate goal of any IT department of a company is either to increase the speed of business development, or to reduce costs due to automation, we need business representatives within these teams. Retail is all about IT. There is not a single process that could be called "unapproachable."
A feature team is a small group of people (usually an analyst, tester, back-end developer and front-end developer, ops). The analyst usually plays the role of the owner of the product or a person from the business comes to this place. The owner has a backlog, priority, tasks. He dictates their development. The developer chooses the implementation option himself, discussing in the team what will be done and how. We have no team lead for task assessment, coding, and decision making. Everyone does everything. Usually more experienced team members give opinions that many are willing to rely on. But anyone can make a decision. Of course, there are conflicts when two developers cannot agree on the implementation of a feature. If the conflict turned to personality - you need an escalation to the head. But most of it is the choice of a solution for implementation. Then everyone stands and draws near the board until they find an option that suits everyone. Usually it turns out quickly, but there were cases when they argued all day. When the argument comes to a standstill, the referee is often called - a person from another team whose opinion everyone trusts. They explain the essence of the problem, he solves. Even a june or a trainee can participate in theory in this dispute, but I only know a couple of such cases - usually the joons have a mentor whom they listen to.
An example of a feature team: we have a service platform that allows you to buy a service with a contractor along with goods. For example, I bought a door - you can immediately put in the basket and installation, so that an independent master comes and does according to our standard. We will check and pay, give a guarantee. So, this team makes a product, for this it writes IT solutions and make business decisions, change processes in stores. Agree with contractors. There - the owner of the product from the business, the clerk from the stores, the architect, developers, analysts and the tester. They immediately use our corporate API platform to deliver all the data back and forth and write a microservice. Such an approach - immediately operational and business people in conjunction with developers and a small team - allows you to quickly create a product. But I think they’d better tell you later
Initially, we did not want to do separate testers: there was a trend that the developer should either follow the TDD methodology or test his code to the end himself. In fact, it was not very effective. At first everything went well: wrote - you answer. You need tests in order for your application to work correctly. But then, the more tasks and applications became, the more difficult it was. Some applications died out, some went to the prod and did not change for months. It became hard to write tests, maintain them, and so on. Team analysts have changed. Relatively recently, they agreed that they were wrong: testers are needed. But developers do not stop writing tests and - sometimes - doing TDD. We realized for ourselves that the tests that check the functionality (the application works correctly), and the tests that check, that the application works in problematic situations - this should be done by different people. And for the second, testers are needed, because they cover possible strange cases not only with unit tests.
Now there are 60 pure developers - back, front, full stacks. There are also analysts, testers and support. And plus an additional seven more devops. But still, 200 planned people from the title are not recruited, so we are now looking for new people, because the field of work is huge. There are vacancies in My circle , if that. That is, from development we now have 74 out of about 200.
Given that we have many independent teams only in Russia, and there are still teams that saw something similar in many other countries, we are moving in the direction of innsource . This is very similar to open source within a limited group of people.
Within the ADEO group of all country divisions, all the code lies in the cloud github. The project is drawn up according to uniform design rules. There are no restrictions on style, technology and tools. When you have open source code and clear design rules, any developer on the stack can contribute. To copy you into the code, you need to read the dock, clone the repository, make an edit, send a pull request.
Currently, Brazil, Russia and France are very actively using this common base.
We are now trying to transfer the entire code of contractors (and we have a lot of them in different directions) to InnerSource. In the analysis, we created a map on two axes: the technical complexity of the transfer to the mode of the inner circle on one axis and the second axis, how much the transfer to the inner circle is generally useful. If the code is unique and needed only in one place - perhaps you don’t even have to touch it. But if many teams use this site - it is definitely necessary.
All of this generally enhances the development culture. And it speeds up the speed of several teams, because you can write a merge request for any feature you need. It will be appropriated by the owner team, and tested by the one who is the contributor. One of the important conditions is the availability of autotests and descriptions of any code in the repository.
When they started, there was nothing at all. In parallel with the developers, they recruited devops. Now devobservices are either provided as a service (for those who need it), or the product team has its own ops, which already determines what and how.
The assembly is done in Jenkins, the code is run in Sonar (more precisely, already SonarQube). Sonar failed - no release. Testers write autotests, code owners - functional tests. The database is given as a service by infrastructure engineers. Full-fledged load testing is done in rare cases, since the structure of the test base and the main one are different: on the preprod, we have chaotic data, and not anonymized real data (this is one of the steps in the future), so you need to roll it gently and not all at once.
Soon we should go to Kubernetes (and some of the new products likethe marketplace is already there initially), as soon as we figure out the transition plan and agree on all the details in the infrastructure.
My colleagues and I will continue to talk about how what part of the work is arranged, because almost everywhere you can go on infinitely. Well, you can follow our reality show (because everything is just being built and changing fast) and bet whether we screw it up or not.
The fact is that Leroy Merlin is in many countries. The parent company in France is called ADEO. They write code for France, Italy, Spain and Russia. Our business models are different: if we keep the lowest prices on the Russian market (below all competitors in monitoring), then in Europe everything is different. In fact, the sea is different - from the characteristics of the locale to other legislation. There are features of the infrastructure of Russia (the same very large delays to Khabarovsk) and another life cycle for placing an order. All this gives rise to such hellish code consisting of huge IF blocks:
Two years ago, we had 60 stores and many, many Wishlist features. They rolled in about six months and not always correctly. The last straw after a bunch of features rejected by low priority was a request to enter a row field in the order, so that we would parse it later. This was necessary for the features of delivery in Russia, since the country is larger than other countries of presence of LM. They refused us this, or rather, they said that it would be somewhere in seven to eight months.
The half-year cycle of the leisurely parent company did not suit us. Naturally, we suggested writing our own code, submitting it to the review and waiting for implementation ... True, nothing good came of this.
Why features were implemented slowly?
The story is very simple: despite our dozens of stores, we are not the first in the world. There are the needs of France (where the parent organization is), there are the needs of other countries. They are prioritized according to possible profit or cost reduction for a group of companies and are executed in this order. That is, our features are made almost never, except very lucky and in some kind of sprint someone will finish their part earlier and go to disassemble the bottom of the backlog.
The second feature is that when you roll any feature into the master branch (here in this Legacy IF-IF-IF code), you need to check how it behaves in other countries. Let me quote 441869 from Bash:
Programmer: Well, imagine that you are a writer and support the project "War and Peace." You have TK - write a chapter on how Natasha Rostova walked in the park in the rain. You write - “it was raining”, save, the error message flies out: “Natasha Rostova has died, continuation is impossible.” Why is she dead? You begin to understand. It turns out that Pierre Bezukhov’s slippery shoes, he fell, his gun hit the ground and shot at a pole, and the bullet from the pillar ricocheted to Natasha. What to do? Charge the gun idle? Change shoes? We decided to remove the pillar. We get the message: "Lieutenant Rzhevsky has died." It turns out that in the next chapter he leans on a pillar that is no longer ...
In general, when we introduce something for the Russian box office, somewhere in Brazil, someone may rebound.
This means very large test coverage, long pre-release procedures, and generally reluctance to maintain code that is growing rapidly. Therefore, the less features - the better. But you hold on there.
The position as a whole is very understandable, and I would do just that on the site of the parent company. Because it is rational.
Solution
The first approach was on the forehead: we suggested opening a code for us to do pull requests there. It was supposed that there would be about 10 developers sitting there who would quickly and quickly code business critical functions, give them to the French, and they would test according to their own procedure, and everything would be fine.
Actually these people already were: we were engaged in cutting out the extra business logic that we got from other countries, such as different cumulative discounts and unusual stocks (which are simply impossible with our business model), and put dummies in this place.
The French at ADEO wanted a release cycle for rolling and installing new versions through themselves. Accepted our offer about one branch for experiments.
Gave access, began to take the code for the review. It turned out to be slow anyway. Rollouts for several months - this is not the case. Well won a few weeks, but still the process did not fit.
Then, for six months, a feature important to us in the content part (managing product data that the client sees) did not come out. We sat for six months without a release. Either their feature didn’t go, or the tests didn’t pass, then they didn’t realize exactly what we needed. As a result, we sat on a key system for a long time without updates. There were NodeJS + PostgreSQL + Couchbase + Elasticsearch + Angular at the front. In the code, due to a number of archaeological layers, things have been encountered, such as the misuse of the SQL database and non-relational database. In one of the places, a huge piece of master data was taken, inserted into one field of the SQL database, and then split into entities in the NoSQL database. On one page of the site with the display of goods there were tens and hundreds of such requests. With further reading of this legacy, hair on different parts of the body began to move. We understood
Own development
The first idea was to do features together. On the spot, that is, in France. The four of us went to France and began to sit next to our colleagues to understand everything together and do as we needed.
It didn’t work. All the same, everything was very slow.
The second idea was to fork all that already exists and gradually finish it. We sat down by the architectural committee, evaluated the prospects, calculated the approximate development plans and realized that we need to choose a different method. Specifically, yes, to fork, but not to develop the existing code, but to break the monolith into parts and put microservices where it is needed.
That is, we planned to rewrite entire blocks of logic in the form of our code. And for this they began to switch part of the services to what could be done with us. We started to recruit developers. Then we completely followed the stack of the parent company - Java, Spring and everything is nearby, instead of Couchbase was Mongo (this is a similar NoSQL-base).
As the project developed, we realized that we needed to do a lot of things in our own way (because it’s faster and easier so as not to support legacy that we don’t need, in particular), and began to expand to other technologies. Then there were Java 7, Wildfly (formerly JBoss) and SVN. We asked why they are not migrating to Java 8, GIT, and Tomcat. It turned out they would not mind, but after a couple of years. In the meantime, dear ones, write on the old stack. So we word for word and decided to separate completely. The business sorted out the question, what are the pros and cons, and fully supported it.
Almost immediately threw about 30 percent of the code, wrote a lot of their microservices around. From the fact that they did not touch, it is almost the entire core of the transaction of business processes for money.
Naturally, the first thing we thought about was how to distinguish between areas of development. I would also talk about this separately, but in general terms the scheme is as follows:
Horizontal is a business area. For example, everything related to customer relations (the first green cell) is one area, and there is a set of applications from one department. This separation of the objective function creates several duplicates of the code in different places and requires a good corporate bus (again, a separate story), but it allows you to clearly find the ends and solve the problem until the result. Looking at this after almost three years, I can say that the architecture was chosen correctly, but if I knew what I know now, I would make a couple of changes.
Now we have come to the conclusion that in the general structure there are many feature teams that make up large product teams. At the same time, the product team includes not only the developers themselves, but also designers and business representatives. Since the ultimate goal of any IT department of a company is either to increase the speed of business development, or to reduce costs due to automation, we need business representatives within these teams. Retail is all about IT. There is not a single process that could be called "unapproachable."
A feature team is a small group of people (usually an analyst, tester, back-end developer and front-end developer, ops). The analyst usually plays the role of the owner of the product or a person from the business comes to this place. The owner has a backlog, priority, tasks. He dictates their development. The developer chooses the implementation option himself, discussing in the team what will be done and how. We have no team lead for task assessment, coding, and decision making. Everyone does everything. Usually more experienced team members give opinions that many are willing to rely on. But anyone can make a decision. Of course, there are conflicts when two developers cannot agree on the implementation of a feature. If the conflict turned to personality - you need an escalation to the head. But most of it is the choice of a solution for implementation. Then everyone stands and draws near the board until they find an option that suits everyone. Usually it turns out quickly, but there were cases when they argued all day. When the argument comes to a standstill, the referee is often called - a person from another team whose opinion everyone trusts. They explain the essence of the problem, he solves. Even a june or a trainee can participate in theory in this dispute, but I only know a couple of such cases - usually the joons have a mentor whom they listen to.
An example of a feature team: we have a service platform that allows you to buy a service with a contractor along with goods. For example, I bought a door - you can immediately put in the basket and installation, so that an independent master comes and does according to our standard. We will check and pay, give a guarantee. So, this team makes a product, for this it writes IT solutions and make business decisions, change processes in stores. Agree with contractors. There - the owner of the product from the business, the clerk from the stores, the architect, developers, analysts and the tester. They immediately use our corporate API platform to deliver all the data back and forth and write a microservice. Such an approach - immediately operational and business people in conjunction with developers and a small team - allows you to quickly create a product. But I think they’d better tell you later
Initially, we did not want to do separate testers: there was a trend that the developer should either follow the TDD methodology or test his code to the end himself. In fact, it was not very effective. At first everything went well: wrote - you answer. You need tests in order for your application to work correctly. But then, the more tasks and applications became, the more difficult it was. Some applications died out, some went to the prod and did not change for months. It became hard to write tests, maintain them, and so on. Team analysts have changed. Relatively recently, they agreed that they were wrong: testers are needed. But developers do not stop writing tests and - sometimes - doing TDD. We realized for ourselves that the tests that check the functionality (the application works correctly), and the tests that check, that the application works in problematic situations - this should be done by different people. And for the second, testers are needed, because they cover possible strange cases not only with unit tests.
Now there are 60 pure developers - back, front, full stacks. There are also analysts, testers and support. And plus an additional seven more devops. But still, 200 planned people from the title are not recruited, so we are now looking for new people, because the field of work is huge. There are vacancies in My circle , if that. That is, from development we now have 74 out of about 200.
InnerSource
Given that we have many independent teams only in Russia, and there are still teams that saw something similar in many other countries, we are moving in the direction of innsource . This is very similar to open source within a limited group of people.
Within the ADEO group of all country divisions, all the code lies in the cloud github. The project is drawn up according to uniform design rules. There are no restrictions on style, technology and tools. When you have open source code and clear design rules, any developer on the stack can contribute. To copy you into the code, you need to read the dock, clone the repository, make an edit, send a pull request.
Currently, Brazil, Russia and France are very actively using this common base.
We are now trying to transfer the entire code of contractors (and we have a lot of them in different directions) to InnerSource. In the analysis, we created a map on two axes: the technical complexity of the transfer to the mode of the inner circle on one axis and the second axis, how much the transfer to the inner circle is generally useful. If the code is unique and needed only in one place - perhaps you don’t even have to touch it. But if many teams use this site - it is definitely necessary.
All of this generally enhances the development culture. And it speeds up the speed of several teams, because you can write a merge request for any feature you need. It will be appropriated by the owner team, and tested by the one who is the contributor. One of the important conditions is the availability of autotests and descriptions of any code in the repository.
Some more nuances
When they started, there was nothing at all. In parallel with the developers, they recruited devops. Now devobservices are either provided as a service (for those who need it), or the product team has its own ops, which already determines what and how.
The assembly is done in Jenkins, the code is run in Sonar (more precisely, already SonarQube). Sonar failed - no release. Testers write autotests, code owners - functional tests. The database is given as a service by infrastructure engineers. Full-fledged load testing is done in rare cases, since the structure of the test base and the main one are different: on the preprod, we have chaotic data, and not anonymized real data (this is one of the steps in the future), so you need to roll it gently and not all at once.
Soon we should go to Kubernetes (and some of the new products likethe marketplace is already there initially), as soon as we figure out the transition plan and agree on all the details in the infrastructure.
My colleagues and I will continue to talk about how what part of the work is arranged, because almost everywhere you can go on infinitely. Well, you can follow our reality show (because everything is just being built and changing fast) and bet whether we screw it up or not.