Warehouse management system using CQRS and Event Sourcing. Design

    image
    So, after setting the requirements described in part 1, you can proceed to the design of the system.

    Our main task in designing, as the name of the article implies, is to achieve separation of interfaces into Query and Command, in order to subsequently divide business scenarios into those that will read data (Query interfaces) and those that will change data (Command interfaces). And also provide the minimum latency for updating the data available through Query, after we changed the data through Command.

    Why CQRS?


    Recently, the separation of interface methods ( CQS ), and subsequently the interfaces themselves on Query / Command, has become a popular trend in the development of application architecture. But Magento does not use CQRS because of popularity, but because sometimes it’s the only way for us to build a flexible extensible (adaptation to specific needs) solution with the ability to independently scale the Read and Write operations.

    In fact, we came to CQRS as a result of the comprehension and widespread use of SOLID in coding. CQRS is the implementation of The Single Responsibility Principle and The Interface Segregation Principle and a departure from the classic CRUD.

    We got CQRS elements a long time ago when we wondered how to scale the EAV (entity-attribute-value) data model for read operations and introduced index aggregation tables for this.

    More details about CQRS, and what it means to us, can be heard in the presentation that I made at MageCONF 2016 ( video , slides ).

    Domain Entity Description


    In the Inventory subject area, we have six main domain models:

    • Source - an entity responsible for representing any physical warehouse where goods may be located (store, warehouse).

    • Source Item - entity-bunch, represents the amount of a specific product ( SKU ) on a specific physical store. It also stores the status of whether the product is available for sale at the moment.

    • Stock is a virtual entity responsible for representing stocks in several physical warehouses (Source) at the same time. It is an aggregation of physical warehouses. The connection between warehouses and Stock (which warehouses are included in the aggregation) is set by the administrator in the admin panel or through an API call.

    • Stock Item - an index entity, is built on the basis of the relationship between Source and Stock, and represents the amount of a specific product ( SKU ) available on a particular Stock, i.e. virtual aggregation.

    • Sales Channel - a sales channel through which sales to the final customer are carried out. In the case of Magento, this can be (Website, Store, Store View), but the sales channel can be determined by the seller independently, so for some sellers it can be a Country, or a large Wholesale client can act as an independent sales channel. In fact, the sales channel is the context within which the sale takes place, which helps us to clearly determine the Stock that should be used during the execution of the business operation.

    • Reservation - an object of reservation, created in order to have the current level of goods that we have for sale between the events of order creation and the reduction in the number of goods in specific physical warehouses. In fact, the reservation object helps us to get rid of the need to carry out checks, locks and write-offs of goods from the physical warehouse inventory (Source Item) during the operation of placing an order, while the value of goods available for sale within a certain Stock changes immediately.

    This diagram represents the interaction of the entities described above:



    Thus, we get the separation of interfaces into Query and Command:



    Theory of operation


    One of our main tasks is to unload the process of placing an order as much as possible. And it is ideal to make this operation such that it will not change the state of the system (State Modification). This will eliminate unnecessary locks and improve checkout scaling.

    The diagrams above show that such operations as rendering a category page will be performed using only the Query API (StockItem), where we need to display the number of products that we can sell in a specific context (SalesChannel).

    The synchronization operation with an external ERP or PIM system will use the Command API (SourceItem) and update the quantity of goods in specific warehouses, after which the reindexing that will rebuild StockItem will allow these updates to be visible at the front.

    In the case of an order placement operation, everything becomes more interesting.

    Placing an Order in Steps


    The operation of placing an order is divided into two operations: directly placing an order , in which the buyer takes part and which ends with payment and confirmation of the acceptance of the order for execution; and order processing , which takes place after the fact (with a certain delay from a few milliseconds to minutes and hours), the main task of which is to determine from which warehouse ( or warehouses ) we must deliver to our customer and calibrate the quantity of goods in these warehouses after the order is completed.

    Actually, we consider these operations in more detail as an example.
    Suppose we have 3 physical warehouses: Source A, Source B, Source C. On which SKU-1 goods are stored in such quantities:
    • SourceItem A - 20
    • SourceItem B - 25
    • SourceItem C - 10
    We have only one Sales Channel, the role of which is performed by the Website.
    For this sales channel, we have created a virtual aggregation of Stock A, with which all current physical warehouses are connected (Source A, Source B, Source C). We get StockItem A for SKU-1 has the quantity 20 + 25 + 10 = 55

    Accordingly, when the buyer visits the Website, the system determines exactly the Stock that should be used to determine the quantity of goods and uses the Stock Items within this stock for all products (SKU) in the category, in our case SKU-1.

    Order placement


    Let the buyer decide to buy 30 units of SKU-1.

    1. We decide whether we can make a sale (do we have enough goods for sale), the number of StockItem A for SKU-1 = 55 minus the number of all reservations for the SKU-1 product on Stock A, in our case 0 (since so far no reservation has been created), 55 - 0> 30, then we decide that we can sell 30 units of goods SKU-1.

    2. At the time of placing an order, we are agnostic to what particular physical warehouses will be written off as a result, therefore we do not work with Source Item entities. * This topic will be discussed in a separate article, which will be fully devoted to the algorithm for choosing warehouses for delivery as part of the order.

    3. At the same time, we cannot reduce the number of SKU-1s on StockItem, since this is a Read projection created for reading only. Therefore, we create a Reservation for SKU-1 on Stock A in the amount of 30 units. Creating a reservation is an Append Only operation that is added without any checks.

    4. The order team itself can then be queued for further processing.

    The status of the system that we have at the moment:
    Quantity of SKU-1 goods in warehouses:
    • SourceItem A - 20
    • SourceItem B - 25
    • SourceItem C - 10
    The quantity of SKU-1 for StockItem A is 55 (unchanged).
    Reservation for SKU-1 on Stock A in the amount of (-30).

    Until we managed to process this order, for example, due to the high latency, another buyer comes to our site who also wants to purchase SKU-1 goods in the amount of 10 units.

    The system begins to perform the same steps as described above.
    Check if we can sell 10 units of SKU-1. The check is carried out as follows: the number of StockItem A for SKU-1 = 55 minus the number of all reservations for the SKU-1 product at Stock A, in our case (-30), 55 - 30 = 25> 10, we decide that we sell 10 units of goods SKU-1 can.

    In fact, the state in Event Sourcing is defined as the projection of aggregated data (in our case, StockItem), with the addition of all the events that were received for the delta time from the moment this projection was formed (in our case, it is Reservation).

    Order Processing


    At this stage, we must determine which physical warehouses will take part in the delivery, and for these warehouses, reduce the value of the quantity of goods shipped.
    For this part, the responsible algorithm that will decide on the choice of warehouses (will be described in the next article). For example, the algorithm decided that the cheapest thing for the seller would be to send 30 goods from the warehouses Source C and Source B.

    Accordingly, the quantity of SKU-1 goods in warehouses should change:
    • SourceItem A: 20
    • SourceItem B: 25 - 20 = 5
    • SourceItem C: 10 - 10 = 0

    Then we create another Reservation object on SKU-1 within Stock A in the amount of (+30) units in order to “zero” the last created reserve object (-30 + 30 = 0). After that, a team should be created to update the StockItem index (which is built as an aggregation of the availability of goods in physical warehouses), which can also be asynchronous.

    Magento MSI (Multi Source Inventory)


    This article is the second article in the series “Warehouse Management System Using CQRS and Event Sourcing”, which will cover the collection of requirements, design and development of a warehouse management system using the example of Magento 2.

    An open project where development is being carried out and where community engineers are involved , as well as where you can get acquainted with the current state of the project and the documentation, is available here .

    Also popular now: