OZON inside: feels like a startup
Ozon.ru is almost the same age as the Runet, in our 20 years we are older than many of our clients. From the online bookstore, the company has grown into an e-commerce platform that brings together infrastructure fulfillment centers and logistics, web and mobile applications, and withstands the raids of tens of millions of users during sales, and attacks from Internet fraudsters.
In this post, we’ll tell you a little about ourselves: how to rebuild and develop the platform, while simultaneously serving 1.2 million users daily. And at the same time we will show the office where the OZON IT laboratory works, and of a couple of gorgeous views from its windows.
Over the past year, the company's technology division has grown eight times - we now have more than 600 people. In July, OZON moved to a new headquarters - it is located in one of the towers of Moscow City. We have occupied eight floors (and while you are reading this, we are capturing the ninth).
The longer the company exists and the more customers it has, the more diverse the zoo of IT systems and the more difficult it is to maintain and develop them. Over the years, OZON has built one of the best logistics and management systems for fulfillment factories; at the same time, the e-commerce platform has remained a massive MS SQL with stored procedures and logic in the database.
Living with a closed Microsoft Windows stack, IIS is hard - these technologies are relatively cheap at the entrance, but at some point become unbearably expensive to support.
In addition to the MS stack, the IT team inherited a lot of systems written almost 20 years ago in Delphi — they need to be rewritten simply because they do not meet the requirements of modern realities.
As a result, about a year ago, a global IT restructuring began at OZON. We started actively hiring developers, switching to open-source and microservices.
The primary task is to cut the logic that is in the bases to make it scalable. Some of this work has already been done - and thanks to this, OZON can grow by 80-90% per year.
Now, many systems are written in GO - this is a reasonable compromise between performance, complexity of development and speed of development. The language is quite low-level to write productive services, and at the same time, quite simple - it can be quickly taught. In Moscow (and in Russia) there are not so many GO-Schnick that we need, so we hire specialists who are ready to master a new stack and teach. The approach works like a win-win - specialists acquire knowledge, and we can finally abandon proprietary solutions, replacing them with our own. And for their tasks, the team is likely to write a more optimal solution than those available on the market.
We have already mentioned above the new office, and now we are gradually settling into it: the designers paint the walls, the developers disassemble the outdated hard drives into magnets and bring scooters and Pepsi cans collections to work.
We do not have a complex hierarchy and subordination - and the CEO does not have a cabinet. He can write to the email - and get an answer on the same day, or in Slack - there it is available almost constantly.
This approach allows you to solve problems quickly and in small teams. Key indicators of projects are displayed in real-time on screens with Grafana, which stand for each of the teams.
On the screen in the office CTO anatolixThere are a dozen of such schedules. For example, it is clear that there are several tens of thousands of packages that have not yet been collected in the warehouse. This is normal - we deliver more than 100,000 orders per day, and several tens of thousands of packages are backlog less than per shift. But if backlog is more than a day, this is already a problem. Thanks to the graphs, it is immediately obvious whether some problem is “accumulating” and in which of the systems something went wrong.
The OZON sustainability test takes place several times a year during sales. So, this November on “Black Friday” the number of orders increased 2.5 times - at that time it was a record for the entire service life, but the December sales broke this record.
In the system of fulfillment and logistics, where goods are sorted and assembled into parcels - their complex IT, which are not visible to the user, and serious workload. And it, too, could choke in moments of peak loads.
At the same time, the number of products is growing, the search algorithm is changing (we recently moved to Elasticsearch) and recommendations (now based on Machine Learning), new features appear. Over the past six months, we have almost completely rewritten the site (from ASP.NET Web Forms to node.JS), updated our mobile applications and are now working on reducing technical debt.
IT is now experiencing explosive growth - we have people and technologies that OZON has never seen before. We try a lot, and if something doesn’t go away, we refuse and take another solution. So, we built Kubernetes cluster, and like in any cuber cluster, there is traffic routing there - we decided to use trendy Linkerd, which is used, for example, by Twitter. But there were constant problems with him: he then fell, then he exhausted all the CPU, then he behaved inadequately. We spent a lot of time on the search, tried to patch something up - and then threw it out and made it easier on Ingress and nginx.
We are actively developing data engineering - there is a huge list of technologies. We try different data storages, algorithms and in general all the technologies that are in machine learning.
We are constantly between the desire to make the most correct and the pace of business growth. The teams do not have half a year to think about the best decisions, because to stay in place, you need to run very, very quickly. We work in the mode: features-features - ensuring sustainability.
In the new year, OZON plans to increase sales in some categories by 10 times, and the area of fulfillment factories - by 2.5 times, launch new areas and develop the newly created ones.
In 1998, we were an online bookstore, then added-added-added categories and became an online hypermarket. But today we and his outgrown. OZON has become an e-commerce platform, within which not only product categories and their supporting logistics infrastructure are developed, but also financial services and about 20 new projects.
In 2018, for example, we launched the marketplace in beta mode - despite the fact that we are testing it, hundreds of sellers are already on it.
The next task within this specific project is to create an ecosystem, providing a showcase, fulfilment and logistics, analytics and monitoring supply chain tools for sellers. Ideally, on the marketplace partners will have access to all the tools that a commerce team has at OZON, and even more.
In general, our IT department has a lot of work to do. Although we also love to relax. By the way, come to visit - we have parties on Fridays.
In this post, we’ll tell you a little about ourselves: how to rebuild and develop the platform, while simultaneously serving 1.2 million users daily. And at the same time we will show the office where the OZON IT laboratory works, and of a couple of gorgeous views from its windows.
Over the past year, the company's technology division has grown eight times - we now have more than 600 people. In July, OZON moved to a new headquarters - it is located in one of the towers of Moscow City. We have occupied eight floors (and while you are reading this, we are capturing the ninth).
Stack replacement
The longer the company exists and the more customers it has, the more diverse the zoo of IT systems and the more difficult it is to maintain and develop them. Over the years, OZON has built one of the best logistics and management systems for fulfillment factories; at the same time, the e-commerce platform has remained a massive MS SQL with stored procedures and logic in the database.
Living with a closed Microsoft Windows stack, IIS is hard - these technologies are relatively cheap at the entrance, but at some point become unbearably expensive to support.
In addition to the MS stack, the IT team inherited a lot of systems written almost 20 years ago in Delphi — they need to be rewritten simply because they do not meet the requirements of modern realities.
As a result, about a year ago, a global IT restructuring began at OZON. We started actively hiring developers, switching to open-source and microservices.
The primary task is to cut the logic that is in the bases to make it scalable. Some of this work has already been done - and thanks to this, OZON can grow by 80-90% per year.
Now, many systems are written in GO - this is a reasonable compromise between performance, complexity of development and speed of development. The language is quite low-level to write productive services, and at the same time, quite simple - it can be quickly taught. In Moscow (and in Russia) there are not so many GO-Schnick that we need, so we hire specialists who are ready to master a new stack and teach. The approach works like a win-win - specialists acquire knowledge, and we can finally abandon proprietary solutions, replacing them with our own. And for their tasks, the team is likely to write a more optimal solution than those available on the market.
Become obese
We have already mentioned above the new office, and now we are gradually settling into it: the designers paint the walls, the developers disassemble the outdated hard drives into magnets and bring scooters and Pepsi cans collections to work.
We do not have a complex hierarchy and subordination - and the CEO does not have a cabinet. He can write to the email - and get an answer on the same day, or in Slack - there it is available almost constantly.
This approach allows you to solve problems quickly and in small teams. Key indicators of projects are displayed in real-time on screens with Grafana, which stand for each of the teams.
On the screen in the office CTO anatolixThere are a dozen of such schedules. For example, it is clear that there are several tens of thousands of packages that have not yet been collected in the warehouse. This is normal - we deliver more than 100,000 orders per day, and several tens of thousands of packages are backlog less than per shift. But if backlog is more than a day, this is already a problem. Thanks to the graphs, it is immediately obvious whether some problem is “accumulating” and in which of the systems something went wrong.
The OZON sustainability test takes place several times a year during sales. So, this November on “Black Friday” the number of orders increased 2.5 times - at that time it was a record for the entire service life, but the December sales broke this record.
In the system of fulfillment and logistics, where goods are sorted and assembled into parcels - their complex IT, which are not visible to the user, and serious workload. And it, too, could choke in moments of peak loads.
At the same time, the number of products is growing, the search algorithm is changing (we recently moved to Elasticsearch) and recommendations (now based on Machine Learning), new features appear. Over the past six months, we have almost completely rewritten the site (from ASP.NET Web Forms to node.JS), updated our mobile applications and are now working on reducing technical debt.
IT is now experiencing explosive growth - we have people and technologies that OZON has never seen before. We try a lot, and if something doesn’t go away, we refuse and take another solution. So, we built Kubernetes cluster, and like in any cuber cluster, there is traffic routing there - we decided to use trendy Linkerd, which is used, for example, by Twitter. But there were constant problems with him: he then fell, then he exhausted all the CPU, then he behaved inadequately. We spent a lot of time on the search, tried to patch something up - and then threw it out and made it easier on Ingress and nginx.
We are actively developing data engineering - there is a huge list of technologies. We try different data storages, algorithms and in general all the technologies that are in machine learning.
We are constantly between the desire to make the most correct and the pace of business growth. The teams do not have half a year to think about the best decisions, because to stay in place, you need to run very, very quickly. We work in the mode: features-features - ensuring sustainability.
In the new year, OZON plans to increase sales in some categories by 10 times, and the area of fulfillment factories - by 2.5 times, launch new areas and develop the newly created ones.
In 1998, we were an online bookstore, then added-added-added categories and became an online hypermarket. But today we and his outgrown. OZON has become an e-commerce platform, within which not only product categories and their supporting logistics infrastructure are developed, but also financial services and about 20 new projects.
In 2018, for example, we launched the marketplace in beta mode - despite the fact that we are testing it, hundreds of sellers are already on it.
The next task within this specific project is to create an ecosystem, providing a showcase, fulfilment and logistics, analytics and monitoring supply chain tools for sellers. Ideally, on the marketplace partners will have access to all the tools that a commerce team has at OZON, and even more.
In general, our IT department has a lot of work to do. Although we also love to relax. By the way, come to visit - we have parties on Fridays.