Special thanks to Pavel Kovtun, CEO of Mycroft Assistant. Pavel has been
specializing in the field of business automation and supply chain management for more than 12 years. He is the author of unique methods of analysis and decision making at SCM, the creator of the first innovative expert-level inventory management system.

The task of any business is the efficient use of available resources. For a business working with goods, one of the main problems is the effective investment of money in a negotiable product - the stock of goods should always be exactly as much as needed. If it is more than necessary ( overstock ), then the money is invested inefficiently, if less ( out-of-stock ) - that is, the likelihood of missed sales, poor quality of service and lost customers. Result - the wrong solution to the problem leads to lost money.

The management industry called Supply Chain Management deals with this problem.. The mechanisms that are used in solving problems are associated with the processing of large amounts of data, mathematical modeling, the use of statistical methods, solving optimization problems and machine learning.

Most companies solve supply problems simply by using Excel calculations, manually collecting data, trivial formulas, and an error that is insignificant at small revs of the company. And up to a certain size and turnover of the company, this option is suitable - it is possible to scale the calculations by hiring new managers. But with the growth of the company, the accumulated error and the increasing amount of data does not allow to carry out all the calculations "in manual mode" - the efficiency decreases so much that as a result of a constant mass of incorrect decisions, the inefficiency of working capital can reach 30%.

What to do business

To solve these problems, there is a special class of systems called Expert Supply Chain Management (expert-SCM) . Such decisions independently monitor data on current sales and the available balances of each product in all outlets (warehouses). And they do this not only with you, but also with other participants in the supply chain. And, having this data, taking into account the individual characteristics of the sales of goods, they online process the entire gigantic data array, making decisions about which goods to invest in and at which point of sale this goods should be delivered now so that there are no problems in the future.



Our solution is Mycroft Assistant (mycroftbs.ru ) is one of the leading cloud representatives of Expert-SCM solutions for SMB . We have been engaged in SCM solutions for more than 5 years, and our customers confirm the economic efficiency of using our solutions in business.

Two years ago, we decided to collect our best practices and accumulated experience in this field, creating a product that best solves the problems of customers. At the same time, we set two tasks for SCM Mycroft Assistant:

1. give maximum business efficiency (usability, speed of integration and deployment, the result of work, the ability to unite all participants in the supply chain),
   
2. have an optimal modern technical implementation (architecture, calculation speed, methods and approaches used).
   

An important point was to enable the wide use of our solution to a large number of customers and the possibility of their interaction with each other within a single supply chain. The result is a solution for cost and convenience available to most SMB customers because it was necessary to take into account the high load on the system. As a result, we got tomorrow’s enterprise application.

How did we design MycroftAssistant?

We initially decided that this would be a cloud solution implemented on a proven Microsoft technology stack (Azure, ASP.NET, MS SQL) that has proven itself and is familiar to customers, and will be able to work as flexibly as possible:

• Onpremise - in a private cloud for those who want to manage their data on their own
  
• Multi-Tenant - a cloud-based SaaS solution for most customers on a shared cloud,
  

Although we started developing on our server in collocation in the data center, it was originally planned to use the Azure cloud for production both for business reasons and for technical reasons:

1. Architecture Azure offers a ready-made pool of solutions and modules, in the logic of which our architecture is implemented. So we did not have to create a complex of replicated VMs. At the same time, we got optimal reliability and scalability , which will allow us to easily scale the business without worrying about technical problems and without coming up with “crutches” to solve those problems that have already been solved by the capabilities of Azure
   
2. Despite the reduced performance relative to its server collocation, the economic effect of a high level of fault tolerance covers these “estimated” costs. So we do not worry about the stability of our own servers, and give a high level of customer service.
   
3. Azure allows you to place server access points closer to clients. This is relevant for us, as our customers are located in Russia, America and Europe. The ability to reduce latency of access to the server by moving it closer to the client is a great advantage, which is not a shame to mention for the client.
   
4. The cost of using and maintaining Azure is quite low, and the ability to easily scale resources according to actual needs helps to predict costs for each client.
   
5. Implementation allows you to combine customer objects within a single database. It is convenient when there is a bundle on the principle of Supplier - Buyer.
   

The implementation of cloud architecture was laid in the design of Mycroft Assistant as a SaaS web-based solution.

What stack was used for implementation?

On the server side, we use the following technologies:

• EntityFramework 6,
  
• SQL Server
  
• ASP.NET MVC 5,
  
• ASP.NET Web API 2,
  
• ASP.NET Identity,
  
• OWIN,
  
• SignalR,
  
• Hangfire
  

From the client side are used:

jQuery, jQuery UI, knockoutjs, flot, Globalize, DataTables, Bootstrap, fancytree

We chose our architecture for a long time and painfully at the start of implementation, but in the end we came to the most optimal one, which can be represented in the form of the following scheme:



SCM Mycroft Assistant consists, conditionally, of three parts: Core, WebAccess , Workers.

Core - a set of core libraries for the overall functioning of the system: common classes, basic interfaces, kernel components, etc.

WebAccess - ASP.NET MVC-based web access. View and manage data, configuration, administration. It consists of several subsystems, while it is possible to install any other subsystem without rebuilding the WebAccess module. Registration of subsystems occurs automatically at application startup.

Workers - Azure Worker Role, within which all calculations are performed.

The system implements the mechanism of "plugins". For example, to load data into the system, a plug-in can be written that can receive data from an external system via API, which are then converted into internal business logic objects and stored in the database. Through plugins can also be expanded: notifications, reports, data calculation modules.

After several iterations and refinement of the architecture, we built the following order of interaction of parts of the system, and their features:

• The user through the website ( Web Role ) looks at the data from his database ( SQL Azure , which is located in the Elastic Database Pool ).
  
• Each client has its own database. The load is not constant, which is why Elastic Pool, Autoscale for Web Role and Worker Role are used . This allows you to scale performance in proportion to the number of customers.
  
• Through the website, the user initiates the download of data from external systems, downloads files, sends a task to recalculate recommendations, planning, etc.
  
• Azure Storage Blob s is used to store files . When downloading a file, Web Role saves it, and Worker Role should receive this file and load data from it into the client database.
  
• The Web site sends a message with job parameters to Storage Queue . Worker Role takes the task, launches the Task to execute it.
  
• The progress of the task is sent to the user’s page through SignalR.
  
• Worker Role is configured for auto scaling, which will allow you to quickly launch new instances if necessary.
  
• Each of the Worker Role works the same way, there is no primary-secondary separation - thus, the Multi-Tenancy approach is implemented .
  

Not bad, but what about the load?

After deploying to Azure, we got the opportunity to conduct comprehensive comprehensive load testing using Visual Studio Online. It revealed a bottle-neck that did not appear when working with current customers. It looked something like this:



For example, if a customer operates 50,000 different products at 60 outlets for a total of only 1 day, if the count all movements between various points, the volume of records is operated 50 000 * 60 * 60 = 180 million , and for the history of data reduction for the year (which should occur in the operating mode), the number of records more than 6.5 * 10 ^ 10 . Big cache does not cope with such a volume .

The problem was the data cache architecture - the whole cache wasin-memory . This made it possible to significantly reduce the response expectations on the client side, but at the same time increased the memory consumption. If you cache only the information to which the most frequent access occurs, then this is not as problematic as caching all raw data.

Resolving Identified Issues

Fortunately, we initially planned to switch to a distributed cache, but for this we had to translate filtering and data aggregation from LINQ to SQL .

Not even in LINQ to SQL, but in manual query construction, since we use a versatile slice for various data (goods, warehouses, managers, etc., and even for years, months and the most “interesting” - for weeks) , and Entity Framework cannot call functions from C # code, unfortunately (Expression didn’t suit us).

Aggregated data, firstly, takes up less memory space, and secondly, we generally abandoned caching by shifting the load on the database. At the cost of an insignificant decrease in the speed of loading information, we defeated memory leaks.

What is the result?

Based on the result of deploying the solution in Azure and gaining the ability to use all the modern “goodies” that this platform provides, we came to the conclusion that our calculation of obtaining technical advantages from using the Microsoft technology stack and Azure cloud technologies fully paid off.

We also see that customers using cloud technologies, such as our SCM Mycroft Assistant solution, are getting better services today than using solutions built on on-premise technology. Such as the:

1. Solution deployment speed. The client can start work in just 1 day. There is no dependence on the accounting system used.
   Accessibility of the solution from various places and devices. No need for long-term customer installation or IT staff involvement in solution integration
   
2. High speed: both on the client side and on the server side. A very large amount of data can be recounted within minutes. In this case, there is no need to spend money on an expensive server. There is also no dependence on the speed of client machines.
   
3. There are no problems with scalability and reliability of the solution. When a company grows, it may not think about the need to increase maintenance costs of an increasing amount of data. Also, the company can be sure of a high level of accessibility of the solution, since it is located on the proven Microsoft facilities, the technical capabilities of which are more than the capabilities of any private company.
   
4. The cost of using and maintaining a turnkey solution is orders of magnitude less. At the same time, costs are transferred from CAPEX to OPEX , which pleases the financial department.
   
5. It is possible to integrate all participants in the supply chain into a single chain with automatic data exchange in the cloud, which does not provide any other solution.
   

You can make sure of all this by reading our demo version , which is available to everyone. And if you are interested in SCM Mycroft Assistant as a client, then you can easily connect to the solution and, having received a free test period, independently verify the effectiveness of its work.

---

about the author

Pavel Kovtun - CEO of Mycroft Assistant

More than 12 years in the field of business automation, more than 8 years of entrepreneurial experience. In the field of supply chain management - 5 years.

The author of several unique methods of analysis and decision making in SCM, with publications in electronic and print publications.

Creator of the first innovative expert-level inventory management system for SMEs.