January 20, 2014 at 16:26Wildberries online store: what is happening on the other side of the monitor
Today we decided to introduce you to the backstage of a completely different direction of human activity and consider the ins and outs of such a complex system, the servers of which are tested in thousands of transactions per second and the price of a software or hardware failure can be too high. So, today we are visiting one of the largest Russian online clothing stores Wildberries.
How does the online store work? To make it clearer, we propose to break down the process into main stages, consider them and draw up a general picture of what is happening.
On the photo: Wildberries online store warehouse
So, first you need to decide which product will replenish the assortment of the store, then ask the supplier for the necessary information on the selected product (composition, equipment, price, etc.). Next, primary price monitoring is carried out, and all the necessary information about the product is poured into the system. After coordinating the order with the supplier and receiving from him all the necessary documents, the manager gives the go-ahead for the shipment of the goods.
After the supplier delivers the goods to the warehouse, the second stage begins. Now the warehouse workers are checking the compliance of the new batch with those positions that were established in the system, checking the goods for factory defects, repacking the goods and sticking the internal barcode. In case of defect detection, discrepancy of the received goods with information in the system or requirements for deliveries, the goods are not accepted into the warehouse. In this way, the store tries to protect the customer from buying low-quality goods and reduce the number of so-called “wrong investments”. After that, the newly made warehouse unit is measured (do not forget that we are talking about a clothing store), the photographer takes pictures on models, and the diligent copywriter writes the accompanying text on the site.
We will not delve into the mechanisms according to which the client gets to the product page. It is clear that to one degree or another he comes to the online store prepared: maybe he is looking for a specific thing or just became interested in an advertising link. It is much more interesting to see how the system responds to client actions, in a word, what happens when a user has already arrived at the site. We know that one of the most important factors that determines how much a client lingers on a site is the speed of processing his search survey. If, according to the specified filters (blue dress, size S, knee-length), the result is 10-15 seconds, a bored client will close the tab and automatically stop being a client. By the way, on average in Russia, the response time of a site to user actions is 4 seconds or more, but Wildberries managed to reduce this time to two seconds. To achieve this, the system uses proxying using nginx: behind one external IP address there are several nginx servers based on the VMWare platform, which is installed on the Dell PowerEdge R620 server. Behind them, in turn, are located directly the web server. In addition, the most frequently requested data is stored in the cache of servers and issued almost instantly. Fault tolerance of web servers is ensured by clustering: at least two servers completely duplicate each other and work in master mode. They are connected to Dell EqualLogic PS Shared Storage, which works in hybrid mode with SSD and SAS drives. This approach allowed us to achieve high response results, despite the enormous load on the server: the number of processed requests, past "external perimeter" - outcash, - within one day varies from 750 thousand to 1.5 million. Yes, this is such a complicated hardware substrate with a simple and intuitive interface that helps you find this or that product in a matter of seconds.
The next stage begins when the customer determines the choice and confirms his order. In certain cases, confirmation occurs via SMS, but most often the operator calls the customer back. By the way, an important point: the system provides for automatic reservation of goods even at the moment when the client has formed an order, but has not yet received confirmation. And all because with large volumes of sales the goods can be taken from under the nose even at that moment, while the operator calls the customer.
For store employees, this stage is not so fast and easy, because the work of processing and placing an order is not so simple as it might seem from the outside. Initially, customer requests go to the warehouse, on their basis a so-called assembly sheet is formed, according to which the warehouse worker collects the necessary goods. The warehouse on which Wildberries goods are stored is divided into zones: for example, there are shoes in one place, things hanging on hangers in another, clothes in the third, etc. The assembly sheet is not tied to a specific order, but is formed in such a way as to minimize the employee’s movement along the rather large warehouse premises and save his time. This algorithm is automatically generated by the system. By the way, it takes into account not only the optimal route, but also the number of things,
The collected things are transported to the place of the formation of orders, on which there are several dozen shelving with numbered cells (they are jokingly called breakwaters). The employee scans the barcodes of the goods, and the system gives him the number of the rack and the cell in which this or that order is collected. They are taken away after the entire rack is completely filled: the goods are sorted into machines that transport them to different cities. Then the courier picks up the order and takes it to the buyer at the agreed time.
Thus, a single IT system of the online store combines the frontend implemented on Linux and the backend on Windows. A large staff of developers and system administrators is working here to support and develop the system, and the computer park of the project is very diverse. In places that do not require heavy workload (for example, to perform some logistics tasks or in a call center), Wyse terminals are installed. Where goods are received and orders are collected — and these are actions that require speed and responsiveness of the interface — personal computers are installed. Employees who process photos of products for the site have a PC with discrete graphics and a large amount of RAM.
The entire information base of the project - a warehouse, call center, store and ERP system (and this is about 20 TB of constantly changing data), is managed using M620 blade servers. They are installed in the M1000e chassis, of which, to date, four have already been involved.
The use of virtualization avoids the risk of data loss during the failure or failure of one or even several blade servers: the current load is evenly distributed between the other servers until the failed element is replaced. All critical nodes are duplicated, and when designing, a certain power margin is initially laid. By the way, Wildberries rents racks at once in two data centers, this allows the site to remain accessible even in the event of an accident in one of the data centers, an accident at an electrical substation or other technological disaster. A direct network connection with the largest Internet providers and traffic exchange points (MSK-IX and others) ensures that the majority of user requests go through the minimum number of network nodes before they reach the company's servers.
The choice in favor of Dell was not accidental. The equipment of many vendors was tested by Wildberries employees using synthetic tests and on non-critical services, and the decisions from the finalists were in “combat” mode right on the store’s site, and the choice was made in our direction according to the combination of technical characteristics, as well as the level of technical support and support. . It is worth adding that the performance of the equipment is monitored in real time using the advanced functionality of Zabbix. All parameters of critical nodes are displayed on monitors (and there are six, by the way!) Of the specialist on duty. Current operating parameters are recorded absolutely from all devices of the enterprise: servers, disk racks, uninterruptible power supplies, network devices and communication channels, not only in the central office, but also in the regions. Besides,
However, Wildberries secretly admitted that the main and most valuable resource is not the server, but a close-knit team of professionals, rooting for their cause. Nice to hear this!
PS If you want to learn more about the capabilities of Dell equipment for organizations working in the field of e-commerce, feel free to write to Marat Rakayev . Our experts will help solve almost any problem.
How does the online store work? To make it clearer, we propose to break down the process into main stages, consider them and draw up a general picture of what is happening.
On the photo: Wildberries online store warehouse
So, first you need to decide which product will replenish the assortment of the store, then ask the supplier for the necessary information on the selected product (composition, equipment, price, etc.). Next, primary price monitoring is carried out, and all the necessary information about the product is poured into the system. After coordinating the order with the supplier and receiving from him all the necessary documents, the manager gives the go-ahead for the shipment of the goods.
After the supplier delivers the goods to the warehouse, the second stage begins. Now the warehouse workers are checking the compliance of the new batch with those positions that were established in the system, checking the goods for factory defects, repacking the goods and sticking the internal barcode. In case of defect detection, discrepancy of the received goods with information in the system or requirements for deliveries, the goods are not accepted into the warehouse. In this way, the store tries to protect the customer from buying low-quality goods and reduce the number of so-called “wrong investments”. After that, the newly made warehouse unit is measured (do not forget that we are talking about a clothing store), the photographer takes pictures on models, and the diligent copywriter writes the accompanying text on the site.
We will not delve into the mechanisms according to which the client gets to the product page. It is clear that to one degree or another he comes to the online store prepared: maybe he is looking for a specific thing or just became interested in an advertising link. It is much more interesting to see how the system responds to client actions, in a word, what happens when a user has already arrived at the site. We know that one of the most important factors that determines how much a client lingers on a site is the speed of processing his search survey. If, according to the specified filters (blue dress, size S, knee-length), the result is 10-15 seconds, a bored client will close the tab and automatically stop being a client. By the way, on average in Russia, the response time of a site to user actions is 4 seconds or more, but Wildberries managed to reduce this time to two seconds. To achieve this, the system uses proxying using nginx: behind one external IP address there are several nginx servers based on the VMWare platform, which is installed on the Dell PowerEdge R620 server. Behind them, in turn, are located directly the web server. In addition, the most frequently requested data is stored in the cache of servers and issued almost instantly. Fault tolerance of web servers is ensured by clustering: at least two servers completely duplicate each other and work in master mode. They are connected to Dell EqualLogic PS Shared Storage, which works in hybrid mode with SSD and SAS drives. This approach allowed us to achieve high response results, despite the enormous load on the server: the number of processed requests, past "external perimeter" - outcash, - within one day varies from 750 thousand to 1.5 million. Yes, this is such a complicated hardware substrate with a simple and intuitive interface that helps you find this or that product in a matter of seconds.
The next stage begins when the customer determines the choice and confirms his order. In certain cases, confirmation occurs via SMS, but most often the operator calls the customer back. By the way, an important point: the system provides for automatic reservation of goods even at the moment when the client has formed an order, but has not yet received confirmation. And all because with large volumes of sales the goods can be taken from under the nose even at that moment, while the operator calls the customer.
For store employees, this stage is not so fast and easy, because the work of processing and placing an order is not so simple as it might seem from the outside. Initially, customer requests go to the warehouse, on their basis a so-called assembly sheet is formed, according to which the warehouse worker collects the necessary goods. The warehouse on which Wildberries goods are stored is divided into zones: for example, there are shoes in one place, things hanging on hangers in another, clothes in the third, etc. The assembly sheet is not tied to a specific order, but is formed in such a way as to minimize the employee’s movement along the rather large warehouse premises and save his time. This algorithm is automatically generated by the system. By the way, it takes into account not only the optimal route, but also the number of things,
The collected things are transported to the place of the formation of orders, on which there are several dozen shelving with numbered cells (they are jokingly called breakwaters). The employee scans the barcodes of the goods, and the system gives him the number of the rack and the cell in which this or that order is collected. They are taken away after the entire rack is completely filled: the goods are sorted into machines that transport them to different cities. Then the courier picks up the order and takes it to the buyer at the agreed time.
Thus, a single IT system of the online store combines the frontend implemented on Linux and the backend on Windows. A large staff of developers and system administrators is working here to support and develop the system, and the computer park of the project is very diverse. In places that do not require heavy workload (for example, to perform some logistics tasks or in a call center), Wyse terminals are installed. Where goods are received and orders are collected — and these are actions that require speed and responsiveness of the interface — personal computers are installed. Employees who process photos of products for the site have a PC with discrete graphics and a large amount of RAM.
The entire information base of the project - a warehouse, call center, store and ERP system (and this is about 20 TB of constantly changing data), is managed using M620 blade servers. They are installed in the M1000e chassis, of which, to date, four have already been involved.
The use of virtualization avoids the risk of data loss during the failure or failure of one or even several blade servers: the current load is evenly distributed between the other servers until the failed element is replaced. All critical nodes are duplicated, and when designing, a certain power margin is initially laid. By the way, Wildberries rents racks at once in two data centers, this allows the site to remain accessible even in the event of an accident in one of the data centers, an accident at an electrical substation or other technological disaster. A direct network connection with the largest Internet providers and traffic exchange points (MSK-IX and others) ensures that the majority of user requests go through the minimum number of network nodes before they reach the company's servers.
The choice in favor of Dell was not accidental. The equipment of many vendors was tested by Wildberries employees using synthetic tests and on non-critical services, and the decisions from the finalists were in “combat” mode right on the store’s site, and the choice was made in our direction according to the combination of technical characteristics, as well as the level of technical support and support. . It is worth adding that the performance of the equipment is monitored in real time using the advanced functionality of Zabbix. All parameters of critical nodes are displayed on monitors (and there are six, by the way!) Of the specialist on duty. Current operating parameters are recorded absolutely from all devices of the enterprise: servers, disk racks, uninterruptible power supplies, network devices and communication channels, not only in the central office, but also in the regions. Besides,
However, Wildberries secretly admitted that the main and most valuable resource is not the server, but a close-knit team of professionals, rooting for their cause. Nice to hear this!
PS If you want to learn more about the capabilities of Dell equipment for organizations working in the field of e-commerce, feel free to write to Marat Rakayev . Our experts will help solve almost any problem.
